[MacPorts] #74298: hunspell-en: contractions such as "didn't" are reported as misspellings (WORDCHARS omits the apostrophe)

[MacPorts]

#74298: hunspell-en: contractions such as "didn't" are reported as misspellings
(WORDCHARS omits the apostrophe)
----------------------------+-------------------------
 Reporter:  diekhans        |      Owner:  (none)
     Type:  defect          |     Status:  new
 Priority:  Normal          |  Milestone:
Component:  ports           |    Version:
 Keywords:  hunspell scowl  |       Port:  hunspell-en
----------------------------+-------------------------
 Analysis and ticket text by Claude Code

 = Description =

 The affix files installed by every `hunspell-en_*` subport declare

 {{{
 WORDCHARS 0123456789
 }}}

 with no apostrophe. Hunspell derives its word tokenizer from that line, so
 it
 splits at the apostrophe and checks the fragments separately. Contractions
 are
 therefore never matched against the dictionary, even though they are
 present in
 it verbatim: `en_US.dic` line 20172 is literally `didn't`, and it can
 never be
 reached.

 Where the leading fragment is not itself a word, the user sees a bogus
 misspelling: `didn`, `doesn`, `isn`, `wasn`, `hasn`, `couldn`, and so on.
 Where
 it happens to be a word (`don't` -> `don` + `t`), the error is silent but
 the
 dictionary entry is still unused.

 `en_US.dic` contains 412 entries with an apostrophe. Checking that list
 against
 the installed dictionary produces bogus misspellings for 56 of them; the
 other
 356 pass only by accident.

 = Steps to reproduce =

 {{{
 $ printf "didn't\nisn't\ndoesn't\n" | hunspell -d en_US -a
 @(#) International Ispell Version 3.2.06 (but really Hunspell 1.7.3)
 & didn 4 0: did, din, dido, did n
 *

 & isn 9 0: sin, ins, ism, is, in, inn, ion, isl, is n
 *

 & doesn 2 0: does, does n
 *
 }}}

 Two result lines per input word show the split. Expected output is a
 single `*`
 for each.

 = Cause =

 The line comes straight from the SCOWL tarball, `speller/en.aff:17`.
 `speller/make-hunspell-dict` copies that one file verbatim to every
 generated
 affix file (line 44: `cp $SPELLER/en.aff $1.aff`), so all `en_*` subports
 are
 affected, not just `en_US`.

 This was fixed upstream in the LibreOffice/AOO/Mozilla English
 dictionaries on
 2024-01-29 by Marco A.G. Pinto, whose `en_US.aff` carries the comment

 {{{
 # 2024-01-29 (Marco A.G.Pinto)
 # - Fix: apostrophe handling, by adding: WORDCHARS 0123456789’ to the
 .aff.
 }}}

 and the line `WORDCHARS 0123456789’` (U+2019 RIGHT SINGLE QUOTATION MARK).
 Note
 that adding U+2019 alone is sufficient: it makes hunspell accept both the
 typographic and the ASCII apostrophe, and the affix file already maps one
 to the
 other with `ICONV ’ '`.

 Bumping the port will not pick this up. SCOWL itself has had no release
 since
 2020.12.07 (confirmed against the SourceForge file listing and `port
 livecheck`),
 so the port needs to carry the fix.

 = Suggested fix =

 Add a patchfile against the SCOWL affix template:

 {{{
 --- speller/en.aff.orig
 +++ speller/en.aff
 @@ -14,7 +14,7 @@
  COMPOUNDRULE 2
  COMPOUNDRULE n*1t
  COMPOUNDRULE n*mp
 -WORDCHARS 0123456789
 +WORDCHARS 0123456789’

  PFX A Y 1
  PFX A   0     re         .
 }}}

 `speller/en.aff` is also fed to `munch-list` during the build, so if
 regenerating
 the `.dic` files byte for byte matters, the substitution can be done in a
 `post-build` block against the generated `${worksrcpath}/speller/en_*.aff`
 instead. Patching the template matches what upstream did.

 = Verification =

 Applying the change to the installed affix files and rechecking the same
 412
 apostrophe entries gives zero misspellings, and the contractions above all
 return `*`. Both `en_US` and `en_US-large` behave the same way before and
 after.

 = Impact =

 Anything that consumes the MacPorts hunspell dictionaries: the `hunspell`
 CLI,
 Emacs `ispell`/`flyspell` (which reads the same WORDCHARS line to build
 its own
 word regexp, so it also treats the apostrophe as a word boundary), vim
 `spelllang`, and any application pointed at `${prefix}/share/hunspell`.

 = Workaround =

 Shadow the affix files with corrected copies and point `DICPATH` at them,
 which
 hunspell prepends to its search path:

 {{{
 D=~/Library/Spelling/hunspell; mkdir -p $D
 for d in en_US en_US-large; do
   ln -sf ${prefix}/share/hunspell/$d.dic $D/$d.dic
   perl -CSD -pe "s/^WORDCHARS (.*)\$/WORDCHARS \$1\x{2019}/" \
        ${prefix}/share/hunspell/$d.aff > $D/$d.aff
 done
 export DICPATH=$D
 }}}

 = Environment =

 macOS 26.5.2 (25F84), arm64, MacPorts 2.12.5
 hunspell @1.7.3_0
 hunspell-en_US @2020.12.07_0, hunspell-en_US_large @2020.12.07_0

-- 
Ticket URL: <https://trac.macports.org/ticket/74298>
MacPorts <https://www.macports.org/>
Ports system for macOS