[PATCH] Allow removing words from the personal dictionary, that weren't previously added

Wed Nov 16 06:55:52 UTC 2022

Am 25.10.2022 um 09:19 schrieb Jürgen Spitzmüller <jspitzm at gmail.com>:
> 
> I'd like this nice feature not to be forgotten.
> 
> Stephan, do you think the problems on Mac can be resolved? Could we add
> the patch with Mac support disabled for the time being?
> 
> Thanks,
> Jürgen

I’ve made two patches on top of Isaacs patch to make it work again on Mac. 
See attached patches. 1st one is the patch from Isaac, 2nd and 3rd are mine.

1. There are code paths with SpellChecker::WORD_OK with WordLangTuple not being assigned anywhere. 
IMO a check for it is required to avoid a crash when using the lang pointer later.

2. Spell checkers in paragraph check mode (Apple) aren’t using the exclude word list at all.
IMO it's the right thing to skip the option to add words to the exclude word list then.

Please test and give feedback for it…

I’ve noticed two additional problems while testing:

1. The new feature with exclude word lists doesn’t work for me with Hunspell. 
But this may be a defect of my private hunspell framework.

2. There is no context menu option to revert the LFUN_FONT_NO_SPELLCHECK. 
One may use the font options dialog - but opening it resets the NO_SPELLCHECK from word immediately.

BR, Stephan

> Am Montag, dem 30.05.2022 um 06:39 +0000 schrieb Isaac Oscar Gariano:
>> Dear Stephen,
>> Sorry for the delay, I had other pressing work to do.
>> Thank you so much for looking at my patch!
>> 
>> What exactly was the problem on MacOSX? Where you getting compilation
>> errors?
>> 
>> I just noticed the following line in src/AppleSpellChecker.cpp:
>>> status == SPELL_CHECK_LEARNED ? LEARNED_WORD : WORD_OK ;
>> Which I should've changed to:
>>           WORD_OK ;
>> 
>> If you can try and compile it and test it for me that would be great!
>> Although I'll try getting myself a Mac VM so I can do so myself.
>> 
>> The way the text is passed in chunks should hopefully not be a
>> problem,
>> the main problem I see is what happens when you call this
>> method https://developer.apple.com/documentation/appkit/nsspellchecke
>> r/1525147-unlearnword  when you haven't previously
>> called https://developer.apple.com/documentation/appkit/nsspellchecke
>> r/1534837-learnword
>> 
>> The documentation dosen't explain sadly.
>> If my code doesn't work, I could always write a post-processing loop
>> to check for unlearned words, (which will be slower).
>> In fact that's pretty much what I made the Hunspell/Aspell backends
>> do.
>> 
>> I should probably do some performance testing, but I've been using
>> this patch for a while now and haven't noticed any extra lag in spell
>> checking.
>> 
>> 
>> 
>> — Isaac Oscar Gariano
>> From: Stephan Witt <st.witt at gmx.net>
>> Sent: Monday, 16 May 2022 2:07 AM
>> To: Isaac Oscar Gariano <IsaacOscar at live.com.au>
>> Cc: lyx-devel at lists.lyx.org <lyx-devel at lists.lyx.org>
>> Subject: Re: [PATCH] Allow removing words from the personal
>> dictionary, that weren't previously added
>>  
>> Am 26.04.2022 um 09:27 schrieb Isaac Oscar Gariano
>> <IsaacOscar at live.com.au>:
>>> 
>>> I've made the "Remove from personal dictionary" function/context
>>> menu item work for words that haven't been previously added, but
>>> instead are in the system-wide dictionary.
>>> I've modified the behaviour so that it will cause the word to be
>>> marked as incorrectly spelt, regardless of whether it is in the
>>> user's personal dictionary, or the system-wide dictionary.
>>> I've attached two patches, one that can be applied on the latest
>>> release branch 2.3.6.1, and the other works on the master branch)
>>> 
>>> Previously, this only worked for words not in the system-wide
>>> dictionary, e.g.:
>>>         • write "ello", it will be red squiggly underlined
>>>         • right click and go "Add to personal dictionary"
>>>         • now the underline will disappear
>>>         • right click it and go "Remove from personal dictionary"
>>>         • it will now be red underlined again
>>> Now you can also do:
>>>         • write "hello"
>>>         • right click on it, and click "Remove from personal
>>> dictionary"
>>>         • there should now be a red squiggly underline under
>>> "hello", i.e. it is no longer considered a word
>>> (The above also works with the corresponding lyx-functions
>>> spelling-add and spelling-remove)
>>> It might be better to change the text of the context menu button,
>>> e.g., "Add to personal bad words list" or something, but then it'd
>>> need to be translated
>>> 
>>> 
>>> The main use cases I have for this feature are:
>>>         • removing rare words that are common misspellings, e.g.,
>>> if you often write "whet" instead of "wet", you can mark the former
>>> as invalid.
>>>         • ensure you consistently use the same variant of a word ,
>>> e.g., make "spelled" an error if you prefer "spelt" (the latter
>>> being a word in the en_GB dictionary)
>>> This change is backwards compatible: any words you had previously
>>> added to the personal dictionary will still be recognised.
>>> 
>>> I have fully tested this on Linux (specifically OpenSUSE
>>> Tumbleweed, with Aspell v0.60.8, Enchant v2.2.15, and, Hunspell
>>> 1.7.0), and Windows 11 (using the included Hunspell v1.6.2), I
>>> don't have a Mac so I can't test the AppleSpeller/Native backend.
>>> 
>>> How it works:
>>> 
>>>         • Enchant already supports this feature out of the box, so
>>> I didn't need to change the backend at all, all I needed to do was
>>> to make the "Remove from personal dictionary" context menu button
>>> show up for all correctly spelt words (and not just those in the
>>> personal dictionary).
>>> Specifically, enchant uses two files ~/.config./enchant/<lang>.dic
>>> and ~/.config/enchant/<lang>.exc to store the personal dictionary.
>>> The former contains all words that you have clicked "Add to
>>> personal dictionary", and the latter uses all that you have clicked
>>> "Remove from personal dictionary" for.
>>> Note that words are added to .dic and .exc even if unnecessary
>>> (because the word is in the system-wide dictionary, or not in it,
>>> respectively).
>>> The ".exc" file appears to take precedence over .dic, so if a word
>>> is in both, it is considered misspelt.
>>>         • Aspell and Hunspell currently uses a file
>>> $LYX_USERDIR/pwl_<lang>.dict to store words you have clicked "Add
>>> to personal dictionary" for. LyX now also uses
>>> the$LYX_USERDIR/pwl_<lang>.excl file for words you have "Remove
>>> from personal dictionary". For consistency, my code treats these
>>> files like the enchant .dic and .exc files above, specifically
>>> the .excl file takes precedence of the .dict file, and words may
>>> be added to these files even if redundant/unnecessary.
>>> The Hunspell backend already natively supports removing words from
>>> the dictionary at runtime, however Aspell does not, so after spell
>>> checking a word, my code manually checks for it's presence in the
>>> .excl list, which I have not optimised at all, and so it is an
>>> O(n) operation, where n is the number of words in the .excl file.
>>> AppleSpeller/Native: this may work out of the box like Enchant, or
>>> not. I have no idea, as I can't test it and the documentation is
>>> unhelpful
>>> (e.g.https://developer.apple.com/documentation/appkit/nsspellchecke
>>> r/1525147-unlearnword)
>>> Note: I have deleted LEARNED_WORD from the SpellChecker::Result
>>> enum, as the code no longer distinguishes between words in the
>>> personal dictionary and the base dictionary. I also removed the
>>> ROOT_FOUND, COMPOUND_WORD, and IGNORED_WORD variants as they
>>> were never used.
>>> 
>>> For the master branch, I haven't modified the "Remove from document
>>> dictionary" option to also work with words not in said dictionary;
>>> I'd have to modify the LyX file format to support a
>>> \spellchecker_reject or something like, but I can probably work
>>> out how to do that if you're happy with the idea.
>> 
>> Hi Isaac,
>> 
>> thank you for your work on LyX.
>> 
>> I’ve tried your patch and unfortunately it doesn’t work on Mac.
>> 
>> Despite the fact that it relies on the existence of the LEARNED_WORD
>> in the SpellChecker::Result enum there is IMO a problem with the
>> general idea of the solution with the Apple speller. Here the whole
>> paragraph is passed as a big chunk to the speller engine to get a
>> reasonable performance.
>> 
>> Best regards,
>> Stephan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Allow-removing-words-from-the-personal-dictionary-th.patch
Type: application/octet-stream
Size: 12839 bytes
Desc: not available
URL: <http://lists.lyx.org/pipermail/lyx-devel/attachments/20221116/d9c7f8d3/attachment-0003.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-Additional-check-for-empty-word-to-avoid-potential-c.patch
Type: application/octet-stream
Size: 895 bytes
Desc: not available
URL: <http://lists.lyx.org/pipermail/lyx-devel/attachments/20221116/d9c7f8d3/attachment-0004.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0003-Fix-broken-Apple-speller-interface.patch
Type: application/octet-stream
Size: 1991 bytes
Desc: not available
URL: <http://lists.lyx.org/pipermail/lyx-devel/attachments/20221116/d9c7f8d3/attachment-0005.obj>