Feedback on SpellCheckerPlugin

Also see discussion in Codev.SpellChecker and related SpellCheckAddOn.

Spell check engines

Lingua::Ispell module at Cpan.org

• http://search.cpan.org/doc/JDPORTER/Lingua-Ispell-0.07/lib/Lingua/Ispell.pm
• This is a Perl front end for the Ispell program

Unix Ispell (international)

• http://fmg-www.cs.ucla.edu/geoff/ispell.html
• curses-based (also a command line executable)
• free

Unix Aspell (English only, but has better suggestions)

• meant as a replacement for Ispell, LaTeX and HTML aware
• Currently in Alpha and is resource-starved
• http://fmg-www.cs.ucla.edu/geoff/ispell.html;
• similar to Ispell (uses same interface such that a TWiki admin could user either Aspell or Ispell?)
• includes Pspell, an abstraction layer

Web-based spell check tools

See WebSpellCheckers.

Functionality

The spell checker checks spelling of words using a widely available open source spell checker (i.e. Ispell or Aspell). Ispell has dictionaries for many languages. Aspell is purported to have better suggested spellings, but is only available for English.

For security reasons, the topic text must not sent over the internet. This is a fundamental requirement driven by the corporate intranet environment. Preferablely, the spell checking is done on the same server.

On the TWiki edit page, the user can press "check spelling" to run the spell checker on the text inside the edit box. When pressed, another web page is presented so the user can correct spelling errors.

For each misspelled words, the suggested spellings are in a drop down list. The user may select one of the suggested correct spellings, leave the word unchanged, or choose to "manually fix". If the user has selected any words to be manually corrected, the user presses "check again." The spell check is performed again. Any words that were to be manually corrected are now text edit fields rather than drop down lists.

The user selects "finished checking" to end the checking and the corrected text is displayed. The user can choose "insert text" to have the corrected text automatically placed in the edit page's text box.

To Do

Next steps

(in approximate order)

Change the prototype over to a TWikiPlugin

• Remove settings from the spell.pl module and make them in the plugin settings
• Integrate the "check spelling" button into the form in the edit.tmpl page

Make compatible with TWiki so that TWiki content isn't marked as spelling errors

• Ignore WikiWords
• Ignore special TWiki syntax
• Ignore HTML

Make the custom dictionary available for editing as a TWiki topic.

Future suggestions

Add an option that allows the user to "add word to dictionary" when the user is correcting their spelling.

Add an ability for users to have their own custom dictionary.

Add an ability for user to add word to the site dictionary or to their personal dictionary when correcting their spelling.

Design ideas and Brainstorming

Suggested words are a drop-down list and shown in context with the sentence. First option in the drop down list is the user's spelling of the word.

After the user has chosen all of the words they want:

1. "Apply changes to buffer" button
2. "Apply changes to buffer and save" button
3. "Cancel"

Consider three HTML form elements on the spell check page:

1. The full text of the message (so the word can be seen in context)
2. A change-to text field that is editable
   • user can type here
   • when change/ignore is selected, the contents of this box replace the word in the original message
3. A Suggestions list box
   • when you select an item in the list box, it updates the text in the change-to field
4. Buttons: Ignore, Ignore All, Change, Change All, Cancel, Finish

-- MikeBarton - 17 Jan 2002

Feedback

Good news that you start to work on the spell checker! This is something many users asked for at work.

We probably need to add some more hooks in the TWikiPluginAPI. Let us know what kind of hooks you need once you get into the design phase.

Brainstorming here. For the UI, how about adding a edit field / selector combo for each misspelled word? Unfortunately HTML has no combo box, but it can be simulated with an edit field and a selector, tied together with some JavaScript. Example (sans JavaScript) :

This text has some ... (learn) (change all) funny fun funky FYI words, but with a ... (learn) (change all) spell spam checker we can fix that.

The Preview screen would show a combo for each misspelled word. Users click/type through the page. When (learn) is picked it will be added to the dic; when (change all) is selected all words will change to the text in the edit field; when a word is picked from the selector it shows up in the edit field. The Save button would apply the changes, then save the topic.

-- PeterThoeny - 18 Jan 2002

Yes. The select for the suggested words is the way many implement it. I hadn't thought of also putting the commands in that box too, that's a good idea. Having another field where the user can type in the corrected spelling is somewhat visually clunky, but is a decent alternative.

For the UI, I see that there are two different design paths that could be taken:

1. Put everything in-line in the preview text
2. Have another small window that is displayed that has the edit/selector box combo with all the other actions that can be done.

I also expect that I (we) will work on this plugin incrementally. I inserted "Increment N" for each section above to provide some development plans. Basically, this gives some order of priority.

-- MikeBarton - 18 Jan 2002

I agree that the editbox/combobox is clunky. How about choosing (edit) from the menu and then submitting the form. This will pull up another display where the mispelled word is in an edit box instead of in a combo box. Problem is how to handle things like change all. Sigh. I really want something that will work reasonably well in a lynx like browser so we don't raise the twiki requirements.

Could this be rolled into the Preview stage of an edit. In another topic I mentioned that check boxes on the bottom of the edit template for "Spell Checking", "Link Suggestions", "Search Other Webs for Topic" could be used to determine what is shown on the following page. If nothing is checked, then it works just like now. Otherwise you get select boxes.

-- JohnRouillard - 18 Jan 2002

Just looked at WBOSS at http://www.dontpokebadgers.com/spellchecker/, spell checker. One of the items in the pull down list of spelling corrections is ---Manually Fix---. When you select that and click continue checking, it brings up an entry box in place of a select box.

Its not to bad too use actually. It is a web based spell checker, and requires a bit of javascript in its current incantation, but it could be adapted to a non-javascript implelemntation with a few more server interactions, I think. It uses aspell or ispell as a spelling engine.

-- JohnRouillard - 27 Jan 2002

Thanks for the reference, John. I did check out WBOSS and it's yet another way to implement the combo-box issue. WBOSS is also in Perl so it might be possible to wrap this in to the plugin.

-- MikeBarton - 28 Jan 2002

I made some updates above on the Xspell entry, hit Diffs to see them. I think it's important to focus on solutions that are OpenSource, like TWiki - this will also enable intranet hosting of spell checking, which is important for performance of spell checking in many companies (sometimes even large telecomms companies with a lot of bandwidth have overloaded Internet firewalls, and the same applies to many large enterprises, not to mention SMEs using dialup access).

If a modular interface can accomodate non-OpenSource tools, that's great, but I think Xspell or similar are worth investigating.

-- RichardDonkin - 18 Jan 2002

I agree -- open source is a basic requirement for the SpellCheckerPlugin. The ones above that are commercial do seem like they would be easy to integrate with TWiki, but they aren't going to be available to all TWiki installations. Also, a fundamental requirement in the corporate environement is that data isn't sent over the Internet.

-- MikeBarton - 28 Jan 2002

I nearly have the Web Based Spell Checker working. I'm following up with the developer because the text isn't being spell checked and I can't figure out why. Slowly making some progress...

-- MikeBarton - 09 Feb 2002

WBOSS does look pretty impressive - is that the one you are working with? The user interface is pretty good - my only concern is that it produced a huge list of alternative spellings, but at least the correct ones were close to the top.

-- RichardDonkin - 10 Feb 2002

Yes, WBOSS is the one I thought I'd try first.

-- MikeBarton - 11 Feb 2002

The first prototype of the SpellCheckerPlugin is available. It's nearly unchanged from the WBOSS package that we found. It does the basic job, but has some work to do to integrate it better with TWiki. It's not packaged well as a plugin yet. Expect to edit the source code file based on your installation details. It should be reasonablely easy to install it.

-- MikeBarton - 27 Mar 2002

Just tried out the prototype am impressed. I had only to install the ispell mod and was up and running. I'd like to integrate it with my other skin's edit template (gnu skin) but am having trouble deciphering which chunks to add into my edit template. Could someone give more info on this?

-- SpatialGuru - 26 Mar 2003

When I ran this I found a lot of preoblems, as reflected by the entries in my http logs. The attached diff is an attempt at cleaning some of them up.

1. Added use strict, which "broke" it. This led to a lot of other changes.
2. Made main a subroutine so I could deal with the scope issues more cleanly.
3. Localized many variables to the package or the subroutine. You can argue about my vs our, but something was neded to define scope
4. Changed bin/spell.pl to bin/spell. I should use %SCRIPTSUFFIX, shouldn't I.
5. Gatuitous changes to layout and indenting as I was figuring out the code.

I appreciate this is "old and unmaintained". Sad. A good spell checker is always a boon.

Now the big problem is that SpellChecker doesn't play well with skins.

What we really need is to abstract the template to be something that skin designers can include or reference in their own skins.

-- AntonAylward - 10 May 2003

It would be good to see a patch or revised plugin after your changes. Also, use of our is not really ideal since it doesn't work with Perl 5.005 (see ReadmeFirst). And yes, you should use %SCRIPTSUFFIX%

-- RichardDonkin - 10 May 2003

I did not find a good way to configure the check language, so I edited line 469 in spell.pl from $pid = open3(\*WRITER,\*READER,\*ERROR,"$path -a -S") or die "Can't open aspell!"; to $pid = open3(\*WRITER,\*READER,\*ERROR,"$path -d deutsch -T Latin1 -a -S") or die "Can't open aspell!";M

Did I miss a better way?

Actually the Plugin cannot deal with special characters like German Umlaute like ö ä. The wordframe is not correctly created. I did a quick hack changing line 156 adding some special chars to those regular expressions:

$wordframe=~s/([A-ZäüößÄÜÖ][äüößÄÜÖA-Za-z\-]*[äüößÄÜÖA-Za-z])|(~~[A-Za-zäüößÄÜÖ][äüößÄÜÖA-Za-z\-]*[äüößÄÜÖAa-z])/_word2label($1)/ge;

Any idea how to fix this propperly?

-- DortheLuebbert - 28 Jun 2003

Some simple changes to the code allow you to use multiple languages ...

in spell.pl

from : $pid = open3(\*WRITER,\*READER,\*ERROR,"$path -a -S") or die "Can't open aspell!";

to: $lang = $query->param('lang'); $pid = open3(\*WRITER,\*READER,\*ERROR,"$path -a -S -d $lang") or die "Can't open aspell!";

then in the template you add something like:

<select name="lang">
  <option value="british">british</option>
  <option value="american">american</option>
  <option value="nederlands">nederlands</option>
  <option value="german">deutsch</option>
</select>

in the <form> part.

-- BartMarescaux - 15 Jan 2004

Can anyone comment on the status of this plugin? Is it maintained?

-- CrawfordCurrie - 15 Mar 2004

It's not maintained, and neither are any of MikeBarton's other plugins. I'm going to update all the cvs versions (with his blessings).

Later: the only difference between the zip and cvs for this plugin is with line-feeds. I left cvs alone.

-- MattWilkie - 24 Mar 2004

Is there a patch available for edit.tmpl which will add a 'spell check' button on the edit page?

thanks

-- SiddharthRam - 04 Oct 2004

I'd love to see this plugin updated. A spell check is one of my users' most-often-requested features. In researching spellcheck options, I discovered that MacOS browsers can utilize the built-in system spell checker as I describe here.

-- LynnwoodBrown - 29 Oct 2004

I was having problems with this plugin with aspell. I would get the new spell check window opening but it would not get any content. I found I needed to comment out the line: print WRITER "!\n"; in the perl script. The check again button still doesn't work so I just hid that from the user.

To make the perl script ignore WikiWords add this to the text2words subroutine:

foreach ($text=~m![A-Z][a-z]+[A-Z][A-Za-z\d]+!g) {
        $wordignore.=" $_";
        }

-- JohnFulton - 22 Feb 2005

Hello - I am also curious as to the status of this plug-in. I understand that it's not maintained, but is it stable and reasonably reliable in it's current form? Are there any alternatives to this? Any advice appreciated! thanks..

-- DaveHecker - 13 Mar 2005

I wouldn't touch it with a fourteen-foot brage-pole. It needs extensive reworking to work with any recent TWiki release.

i don't know of any alternative.

-- CrawfordCurrie - 12 Aug 2005

I'm investigating a JavaScript based spell checker -- http://spellerpages.sourceforge.net/ -- which looks promising. I think it should be able to be used as a plugin in much the same way as the Mishoo DHTML calendar in EditTablePlugin and JSCalendarContrib.

-- MagnusLewisSmith - 10 Oct 2005

Done -- See SpellerPagesPlugin

-- MagnusLewisSmith - 12 Oct 2005

I was wondering if that plugin was working on Dakar? Apparently not looking at the latest release date and the comment above twiki.org uses some kind of spell checker though, is it done using that plugin? (no, it's done by Sven -- sorry, couldn't resist, please remove this)

-- StephaneLenclud - 11 Feb 2006

Forget that the spell check that I thought twiki.org was doing was in fact done by my browser Konqueror

-- StephaneLenclud - 12 Feb 2006

I have managed to get 'Googie Spell' (see http://amix.dk/projects/?page_id=3) working with the TWiki edit page. Unfortunately it does send the text through Google's spelling engine (albeit using an SSL connection) so if you don't trust google it's out of the question. I'm working on modifying the perl script I made to handle the XmlHttpRequest so that it can use aspell or ispell instead but we'll see how we go. I think the author already has a backend for this but is a chargin' for it (the front end is GPL'd).

In the interim the perl script (for twiki/bin) that forwards on the request from Googie Spell to Google is available here: http://shake200.esscc.uq.edu.au/twiki/pub/TWiki/GoogieSpellPlugin/spell.txt, with that and the instructions from http://amix.dk/projects/?page_id=3 you should be able to hack it into the skin for your edit page fairly easily (I hope to bundle it properly at some point).

-- StephenGordon - 30 May 2006

I now have a version of the backend script that works without sending your data to Google (relies on the Lingua::Ispell cpan module which in turn relies on the ispell binary, an ispell wrapper script is available with aspell these days, which is what I am using on my system). I need to clean it up a bit but I'm fairly happy with it thus far, I still haven't looked at how it might work with the WYSIWYG editor but it works wonderfully with the normal one.

-- StephenGordon - 02 Jun 2006

Added as Related: GoogieSpellContribDev

-- CraigMeyer - 25 Oct 2006

Is there a spell check function in 5.1? Can't find it.

-- DanielDWang - 2012-07-04

This plugin is very old and needs some TLC to be used on recent TWiki versions. TWiki is a community driven project. I invite you to get involved with the TWiki community. As a contributor you can actively make a difference and help shape the product. If you are not a developer and need a timely response I recommend to hire a TWiki consultant to enhance TWiki based on your needs..

-- PeterThoeny - 2012-07-05