Feedback on the MathModePlugin

For a comparison of this plugin with other math related plugins, please see ComparisonMathLatexPlugins.

-- AmandaSmith - 01 Mar 2006

GraemeLufkin replied on my email asking if he is still maintaining this plugin

Thanks for your interest!  I am no longer maintaining it, and I would
be happy to pass the reins to you.  I don't follow the area much
anymore, but I think there are other TWiki plugins that supercede
MathModePlugin's functionality.  But I officially give you my blessing
to take MathModePlugin in whatever direction you feel is right.  Do give
me a heads-up email if you make a new release.  Thanks, and good luck!

So I will take over maintenance and see what I can do to keep this piece alive. Although we have the LatexModePlugin which supersedes this one I'd like to have the option to use latex2html instead.

Given the previous release was 2.0 ("Second release") the next will be 2.9 on its way to have a new 3.0 after a few cycles.

So, guys, if you have any interest or any patches that you want to go into that release, please repeat these fixes'n wishes now.

The following things will happen first: OO-ification for lazy compilation, use of TWiki's sandbox security harness, rewrite of configuration not using preference values for things that have to be set once during installation, additional <latex>...</latex> syntax.

-- MichaelDaum - 03 Aug 2006

Micha,

One note of caution: there is a fundamental problem with the way this plugin meshes with latex2html. Specifically, latex2html reformats the markup before rendering-- typically adjusting the spacing. e.g. %$ b a $%, %$b a $%, and $% b a$%, all get mapped to the same image (which is reasonable) but not with this plugin. This is the source of the "image rotation bug". So, if you choose to use latex2html, you will need to incorporate or clone their parsing/respacing code.

Plus, my experience is that dvipng is much more efficient. So I recommend using that program instead of or addition to latex2html.

That said, I think it would be useful to have this plugin upgraded, to provide a super-secure math rendering engine for public wiki, even though this necessitates reduced functionally compared to the LatexModePlugin.

-- ScottHoge - 04 Aug 2006

Hi Scott,

thanks for the warning. I will try to repro the problem you reported with the testcases that you gave above. I think you can either use latex+dvipng or latex2html, the latter produces the graphics already IMHO with no need to run an extra dvipng. One thing I like using latex2html instead of the way you did it is that I only need one single call to latex2html no matter how many formulas there are on one topic. In contrast, you need to call a complete toolchain to be launched for every single formula (am I right?). So even if dvipng is more efficient the resulting plugin is probably not.

Besides, I'd suggest to split up the LatexModePlugin to support lazy compilation and initialization. As things are right now your plugin has a rather heavy initPlugin() which does work that may be delayed til it is really needed. And yes, using system() and backticks within perl is an insecure thing to do. Have a look at TWiki::Sandbox. If you still want to support TWiki/Cairo then have a look at the DakarContrib which backports the security sandbox.

-- MichaelDaum - 04 Aug 2006

In contrast, you need to call a complete toolchain to be launched for every single formula (am I right?).

Well, this is how it is currently implemented, but it doesn't have to be this way. One could call latex+dvipng once and get all of the image s for all of the formulas in one system call.

This reorg of the system calls, and use of the Sandbox, have been on my todo list for awhile. It has just been a low priority compared to other (non-TWiki) work.

And I'm certainly eager to see your lazy compilation code, and always happy to learn how to write better code.

-- ScottHoge - 04 Aug 2006

Uploaded new version.

-- MichaelDaum - 04 Aug 2006

Thanks Micha for maintaining yet another Plugin!

Also here a small feedback: The [[%ATTACHURL%/%TOPIC%.zip][Download]] link fails if this Plugin is installed; it works only here on TWiki.org. Better to write the full URL [[http://twiki.org/p/pub/Plugins/%TOPIC%/%TOPIC%.zip][Download]]. Actually I think it is better to keep the original text "Download the ZIP file from the Plugin web (see below)" (and fix the viewfile issue on TWiki.org.)

-- PeterThoeny - 05 Aug 2006

Micha,

I took at look at your 04aug2006 release, and here is my feedback.

First, this release suffers from the image rotation bug. Here is a simple test. Using

%$\alpha$% %$b = Ax$% %$ \beta $%  %$b=Ax$% %$\kappa $% 

for the topic text, the images will render out of order. This is because the MathModePlugin and latex2html hash tables are mismatched. From the debug output:

- MathModePlugin - using img1 for b=Ax
- MathModePlugin - using img2 for b = Ax
- MathModePlugin - using img3 for \alpha
- MathModePlugin - using img4 for \kappa
- MathModePlugin - using img5 for \beta

img1 => b=Ax
img2 => \alpha
img3 => \kappa
img4 => \beta

Then, on the next call to view, \beta (aka img5) will render again instead of b = Ax. To undo the 'rotation', one must delete images 2-5, render just img2, and then render images 3-5. A major pain for users.

Second, latex2html is significantly slower than latex+dvipng. Using the same topic content (5 image requests), and the following test call (after clearing the pub directory each time)

perl -e 'use Time::HiRes qw(time); $t=time; `lwp-download http://localhost/twiki/bin/view/Sandbox/TestTopicZ /dev/null`; printf("%6.4f\n",time-$t);'

• MathModePlugin w/ latex2html requires 5.12 secs, for 4 images. This is longer than ...
• LatexModePlugin with dvips+convert, which clocks in at 4.05 secs for 5 images.
• LatexModePlugin with dvipng only requires 2.66 secs, for 5 images.

(BTW, I time using lwp-download in order to give a good estimate of rendering time as the images themselves are not requested and the download completes when the rendering is finished.)

Given the above, I see no compelling reason to use latex2html for background math rendering in TWiki.

-- ScottHoge - 06 Aug 2006

It looks as though you are damn right: latex2html is quite slow and applies extra trickery finding out that b=Ax and b = Ax is actually generating the same output ... which MathModePlugin fails to see (why should it). The rendering issue seems striking too. Note, however that you pay that extra time only once and not on every view.

Did performance comparison latex2hml vs latex+dvipng+convert and the results are even more dramatic as the latter is faster by a factor of 10.

-- MichaelDaum - 07 Aug 2006

Micha,

Thanks for abandoning the latex2html path... I'm much relieved.

I'm curious about one thing: why are the config settings labeled as $TWiki::cfg{MathModePlugin}{Latex2Img} instead of $TWiki::cfg{Plugins}{MathModePlugin}{Latex2Img} ?

Also, if you are up for a challenge, dvipng reports alignment data when it generates images. This could be used to put the image on the correct baseline. And go from this:

to something closer to this:

My own tweakinline code is very crude, but necessary if TWiki is to support both dvips+convert and dvipng. I hope to use the dvipng output alignment data at some point.

-- ScottHoge - 07 Aug 2006

Hm, dunno what the policy wrt the cfg namespace is. But IMHO the extra {Plugins} has no obvious benefit.

And wow! Good hint about this dvipng --depth parameter which generates the vertical-align parameter for the html.

However, there is a problem how to make use of that output: our plugins first generate the markup before the actual (set of) images is generated, which would include the extra information to align the image correctly. So how about having a

 <script>@import(url(hashcode.js))</script>

that generates the img tag using dynamic html means. The hashcode.js files can then be generated by the postRenderingHandler together with the actual images.

-- MichaelDaum - 08 Aug 2006

Hmmm. The .js solution might work. I was hoping for a perl solution.

Is it possible to run a post-rendering pass through the topic, like I do with the CITE macros in the BibtexPlugin? That is, put a place holder in the img tag during markup generation, that will be replaced with the correct alignment after the page is completely rendered? This was the direction I was hoping to head.

-- ScottHoge - 08 Aug 2006

The point is that the extra information about the alignment has to be stored somewhere after it has been extracted from stdout of dvipng. That information needs to be added somehow without dvipng when the markup is generated the second time. You can store an maintain that in an extra info file. Or you can encode it into the image filename: _img_<hash-code>_<align>.png Or it gets stored in the js file.

-- MichaelDaum - 08 Aug 2006

Good point: image is cached, alignment information needs to be cached as well. I like the filename approach.

-- ScottHoge - 08 Aug 2006

I am getting the following error in my warning log file:

 exec failed: Permission denied at /usr/local/twiki/lib/TWiki/Sandbox.pm line 319.

Anybody got this before? I am using TWiki 4. Cheers.

-- AndrewRJones - 30 Aug 2006

This means that you don't have the right to execute the requested binary. Check that you have all tools installed and that you can call them on the commandline.

-- MichaelDaum - 30 Aug 2006

Micha, I suggest to make some small mods to the plugin topic:

• Escape links to oneself
• Avoid "click here" text, better to link to the action, such as "you can refresh the formulas in the examples below."
• Use full URL for Plugin Home/Feedback/Appraisal instead of Interwiki links

-- PeterThoeny - 18 Dec 2006

I ran into an interesting problem while using MathModePlugin on a Windows/Apache/Cygwin installation. It seems that stdout of the latest Cygwin distro ImageMagick convert util that I used did not work when executed from the MathModePlugin latex2img script (as user SYSTEM). I have not uncovered the exact reason for this. The initial symptom is that the images have very large margins which indicate that the "-trim" option of convert util does not work. However, after further investigation, it seems that the images are not modified at all.

The only solution that I was able to come up with is to uninstall the Cygwin distro version of ImageMagick and install the win32 native version. Then, I created a symbolic link from the /usr/bin/convert under Cygwin to the win32 version's unnecessarily long default path.

I hope this might prevent someone from wasting as much time as I did.

-- DavidVerbree - 26 Dec 2006

I have installed this plugin, and all dependancies, but I can't seem to get past the following error in my apache logs:

File does not exist: <TWiki root>/pub/TWiki/MathModePlugin/_MathModePlugin_xxx.png, referer: http://www.mydomain.com/twiki/bin/view/TWiki/MathModePlugin?refresh=on

I did have to edit latex2png to point to /usr/local/bin/dvipng. I did this by hand. I did notice a ,v file there. Do I need to check this edit in? Should I have edited this file by hand, or via TWiki?

I can't for the life of me figure out how/where to see the stdout/stderr from the Plugin.

Any help would be much appreciated.

-- AaronScamehorn - 01 May 2007

Check that latex2img is executable

-- PeterJones - 14 Jun 2007

I have installed the latest version of the plugin on Twiki 4.1.2 I have also installed and correctly configured latex2img with /usr/bin/latex /usr/bin/dvipng /usr/bin/convert

However, I get the error on the bottom of all test pages.

Error during latex2img:

ERROR: problems during latex
INPUT:
\documentclass[fleqn,12pt]{article}
\usepackage{amsmath}
\usepackage[normal]{xcolor}
.
.
.

Where should I look in order to solve this problem

-- PeterJones - 14 Jun 2007

Thanks, Friend. That's what my problem was.

Now I'm getting to the same point that you are.... problems during latex.

Best I can tell, my problem is now w/ latex, as the message says. Specifically, no xcolor in my latex distro (tetex-latex-2.0.2-22.EL4.7).

To see if this is your problem too, copy everything between INPUT: and STDERR: from the bottom of your page to a file, and then run latex on it.

I get the following error: ! LaTeX Error: File `xcolor.sty' not found.

I found this site http://www.ukern.de/tex/xcolor.html, but I can't figure out how to get xcolor properly installed....

If you do, please let me know! Good Luck!

-- AaronScamehorn - 15 Jun 2007

Ok, finally got it. From the above mentioned site, get xcolor.ins.gz and xcolor.dtx.gz. Follow instructions in the README. Important Now you must run texhash to tell latex a new file has been installed.

Once I got that working, however, I was still having problems.

So, back to latex2img. On my flavor of Linux (RHEL4), I don't have tempfile. Needed to change it to mktemp.

Phew!

-- AaronScamehorn - 15 Jun 2007

Same Problem with my Distros: Debian-Etch and Ubuntu-Feisty: Please include latex-xcolor in your "Other Dependencies"-Section. If you know what you miss, it is quite easy to Install this in Debian. Thanks for your work, bodo

-- BodoRiedigerKlaus - 12 Jul 2007

I had a similiar experience with my installation - TWiki 4.1.2 on SuSE Linux 10. First of all, I would like to state that I saved days in getting this working due to the posts of AaronScamehorn and PeterJones. Nice work guys!

Here's what I had to do to get this working:

• Install the following RPMs:
  • ImageMagick-6.2.5-16.2
  • ImageMagick-devel-6.2.5-16.2
  • ghostscript-library-8.15.2rc1-20.4
  • tetex-3.0-37.2
  • te_latex3.0-37 (Unlike Aaron's issue with the "color" package, I think that TeLaTeX installs that by default.)
  • latex-ucs-20041017-2 (This is a prerequisite for te_latex3.0-37 RPM.)

• Set permissions on /usr/share/texmf/ls-R (set it to 755)
• Ensure that the environment variable TEXINPUTS is NOT set. I had to comment out the following lines from /etc/profile as it appears as though TeLaTeX inserts it:

  # Set some environment variables for TeX/!LaTeX
  #
  #if test -n "$TEXINPUTS" ; then
  #    TEXINPUTS=":$TEXINPUTS:$HOME/.TeX:/usr/share/doc/.TeX:/usr/doc/.TeX"
  #else
  #    TEXINPUTS=":$HOME/.TeX:/usr/share/doc/.TeX:/usr/doc/.TeX"
  #fi
  #export TEXINPUTS

• Execute command texhash to load the ls-R database with all of the appropriate package files.
• Edit the file: %PATHTOTWIKI%/TWiki/MathModePlugin/latex2img to reflect your environment:

     latex=/usr/bin/latex
     dvipng=/usr/bin/dvipng
     convert=/usr/bin/convert
     grep=/bin/grep
     origpwd=`pwd'
     stderr=`mktemp` *Note: I had to change this from default value of 'tempfile'

• Set executable permission on file: %PATHTOTWIKI%/TWiki/MathModePlugin/latex2img.
• Visit configure in your TWiki installation, and enable the plugin in the {Plugins} section.

My users recently reported that the examples from the plugin failed

<latex>
  {\cal P} & = & \{f_1, f_2, \ldots, f_m\} \\
  {\cal C} & = & \{c_1, c_2, \ldots, c_m\} \\
  {\cal N} & = & \{n_1, n_2, \ldots, n_m\}
</latex>

gives

Error during latex2img:

ERROR: problems during latex
INPUT:
.
.

However the examples work by replacing the latex tags with

%\[ 

%\[{\cal P} & = & \{f_1, f_2, \ldots, f_m\} {\cal C} & = & \{c_1, c_2, \ldots, c_m\} {\cal N} & = & \{n_1, n_2, \ldots, n_m\}\]%

Why would this be the case? I have recently tried out the LatexModePlugin on the same system , would this effect MathModePlugin in any way?

-- PeterJones - 19 Nov 2007

Yes. Use either or the other. Hm, the above example works fine on my machine.

-- MichaelDaum - 19 Nov 2007

I haven't had any luck getting the test page to render. I was wondering if there was a way to debug the plugin? I simply get no errors back, even though I believe I have everything setup.

Are there command line examples I can run to test ImageMagick, Dvipng, or latex2png?

-- EricHanson - 25 Jan 2008

I've installed MathModePlugin on SLC4 and had to do some extra steps to make it work (in addition to official docs):

dvipng was configured with options:

./configure --disable-debug --prefix=/usr

MathModePlugin:

chmod +x pub/TWiki/MathModePlugin/latex2img

LaTeX:
loaded chngpage.sty from http://www.ctan.org/tex-archive/help/Catalogue/entries/chngpage.html put it to /usr/share/texmf/tex/latex/chngpage/ and ran commands:

# mkdir /usr/share/texmf/tex/latex/chngpage
# cp chngpage.sty /usr/share/texmf/tex/latex/chngpage/
# cd /usr/share/texmf
# mktexlsr

loaded xcolor from http://www.ctan.org/tex-archive/macros/latex/contrib/xcolor/ and ran commands after installation:

# mkdir /usr/share/texmf/tex/latex/xcolor
# cp *.sty /usr/share/texmf/tex/latex/xcolor/
# mkdir /usr/share/texmf/dvips/xcolor
# cp *.pro /usr/share/texmf/dvips/xcolor/
# cd /usr/share/texmf
# mktexlsr

-- AlexeyFilin - 23 Oct 2008

Solved problem with

Error during latex2img:

ERROR: problems during latex
INPUT:
.
.

The folliowing settings in /usr/share/texmf/web2c/texmf.cnf prevented any temparary files from being created.

% a (any)        : any file can be opened.
% r (restricted) : disallow opening "dotfiles".
% p (paranoid)   : as 'r' and disallow going to parent directories, and
%                  restrict absolute paths to be under $TEXMFOUTPUT.
openout_any = p
openin_any = p

-- PeterJones - 8 Jan 2010

Really nice plugin but I needed to give execute rights to pub/TWiki/MathModePlugin/latex2img. so this could help if someone else gets a "Permission denied to path" error

-- CarlaReitzenstein - 2011-05-05

Thank you! That fixed my "exec failed: Permission denied path" error.

-- MatthiasRella - 2011-10-17

If you get an error like "fmtutil: [some-dir]/latex.fmt does not exist", do

fmtutil-sys --all 

AND if you finally managed to get rid of all errors but your generated image of the latex formula does not show up, then you probably have encoding troubles. Look into the source of the <img>-tag in your page's source code. The plugin seems not to decode non-ascii characters.

For me it helped to set up localisation for my language (see your TWiki configure page).

-- MatthiasRella - 2011-10-17

Thanks Matthias. I added a troubleshoot section with this info to the plugin doc page and re-released the plugin.

-- PeterThoeny - 2011-10-17