Feedback on XmlQueryPlugin

Updates

Ok that's version 1 out the door then

-- PatrickDiamond - 10 Nov 2004

Ok that's version 2 out the door then

This version is quite a bit more flexible about where it gets data (anything with a DBI interface added) and where it can send it (attachments and DBI interfaces added).

To get an idea of what can be done with this flexibility on the live system I look after (on an intranet) this plugin as well as parsing through TWiki page contents

• reads from a PostgreSQL database using the DBI interface,
• Joins database data with TWiki table contents to produce TWiki output
• writes to a PostgreSQL database using the DBI interface
• writes out Excel XML files as attachments
• loads external XML files using the document function and joins this with TWiki table contents.

-- PatrickDiamond - 9th Sept 2005

Version 2 and a bit ready.

This version provides XSLT functions to directly read topic xml, attachments and cgi parameters. It should be a bit quicker (but getting the latest greatest libxml and libxslt can have more of an impact)

-- PatrickDiamond - 10 Oct 2005

Dakar compatiable release (minor changes)

-- PatrickDiamond - 9th Feb 2006

Minor doc, fixes

-- PatrickDiamond - 28 Feb 2006

XMLDIR setting now removed.

-- PatrickDiamond - 16 Aug 2006

Wish List

Patrick - love the plugin! One thing I would like to be able to do is access all of the other data elements in a topic, especially headings and lists. Are there any plans to add this type of functionality?

-- KevinJarnot - 10 Jun 2005

It makes sense to add some of that into the TWiki core, see ContentAccessSyntax

-- PeterThoeny - 10 Jun 2005

Kevin - I had not any plans to do this. The reason being I couldn't come up with a decent use case that fitted to the design goals. Those goals being providing a mechanism for extracting structured content (it has a name/id or type/class), then flexibly querying it and finally producing output without resticting the format.

That's why tables,forms and actions are extracted to XML then parsed via XSLT.

The problem with the data contained within the scope of headers (all the text in the section) is that it isn't structured (has no names/ids or types/classes) and therefore isn't readily querible. It is searchable but there are better tools and approaches for that job as Peter mentions.

If you have a particular use case in mind let me know more about it and I will see if there is something i can do.

-- PatrickDiamond - 11 Jun 2005

Note that due to the fact that this plugin processes meta-data in the beforeSaveHandler it will not work properly with DakarRelease, as the handler semantics have changed. However the change does not involve a lot of recoding, and should actually simplify the plugin slightly.

-- CrawfordCurrie - 21 Jun 2005

Why change the semantics of the Plugin API? We cannot afford to change the Plugin API, developers do not want a moving target. Adding functionality to the Plugin API is OK, but am am opposed to removing/changing functionality.

• Half a year later and I run into an issue because of the change in beforeSaveHandler, see HandlingCairoDakarPluginDifferences -- PTh - 09 Feb 2006

-- PeterThoeny - 25 Jun 2005

Wasn't this 'change' reverted back (due to other problems)? So there's nothing to fear.

-- FranzJosefSilli - 26 Jun 2005

Hello, I tried the attachment storage and I would like to suggest, that the generated attachment should behave like "normal" attachments - right now it does not seem to be listed in topic's metadata - and so if I try to XSLT a topic with XmlQueryPlugin-generated attachment, I am getting an empty element. On the other hand, it may not be wise to revision the generated data the same way as ordinary content.

-- SvataDedic - 08 Oct 2005

On attachments, the reason I used the attachments area for xmlquery output was to restrict where data could be written, what could be overwritten and to still allow that data to be easily referenced from within TWiki. Unfortunality these output files don't quite fit the TWiki attachment model in that they lack comments, revisions and a meaningful user/creator.

If you wish to use one of these outputs as input to another XSLT script I suggest using the new (10th Oct) xmlquery:readattachment function or the XPath document function (via a URL as file reads are disabled)

-- PatrickDiamond - 10 Oct 2005

I had some problems getting the 10th oct 2005 version of this plugin to work here. Apart from the fact I am running a (almost) dakar release, I think, I found a bug: After negating an expression in line 701, the plugin started working properly.

return $errorstr if defined $errorstr and $errorstr =~ /^\s*$/; should be return $errorstr if defined $errorstr and $errorstr !~ /^\s*$/;

-- OliverKrueger - 20 Dec 2005

Discussions

Patrick: Would you consider keeping this useful Plugin compabile with Cairo and Dakar codebase? The current topic format is 1.1, which is not compatible with Cairo. I have not looked at the code, I do not know if there are code incompatibilities. Learn more at HandlingCairoDakarPluginDifferences.

-- PeterThoeny - 09 Feb 2006

Please watch out, the parameters of the beforeSaveHandler changed, see HandlingCairoDakarPluginDifferences.

-- PeterThoeny - 09 Feb 2006

beforeSaveHandler usage is Ok as the $text parameter continues to contain the metadata (Hurray!

-- PatrickDiamond - 10 Feb 2006

Cario and even the 2003Feb release should work fine. $TWiki::Plugins::VERSION is used to partition the new Dakar specific code from the code which previously carried out these tasks. These amendments are mainly just for preference fetching

-- PatrickDiamond - 10 Feb 2006

Nice to use conditional code. Earlier, I checked the format of the topics. They are format="1.1", e.g. bullets will not render properly in Cairo because bullets have three leading spaces instead of a tab.

-- PeterThoeny - 10 Feb 2006

tabs are gone

-- PatrickDiamond - 13 Feb 2006

Yes, tabs are gone, which means this Plugin topic does not render properly if installed on a Cairo codebase. It also does not reflect the "tested on" info. It would be helpful to revert the topic to the 1.0 format.

-- PeterThoeny - 13 Feb 2006

I think this requires also LWP::UserAgent, at least I couldn't get anything out from this plugin without it. Now I can get error messages..

-- HarriVartiainen - 24 Feb 2006

• Pages are now in format 1.0
• LWP::UserAgent is listed on Plugin page (thanks Will)
• XmlQueryPlugin_install.pl script added to check for and warn about installed modules

-- PatrickDiamond - 28 Feb 2006

The setting Set XMLDIR = /var/tmp/twiki_xml is a major security risk. Plugins should NEVER contain settings with absolute filenames.

Right now we plan to alter the hierarchy that defines how settings are loaded. See MainTWikiPreferencesOverridePluginSettings. Right now you have to unset any TWikiVariable defined in a Plugin topic to be able to set it in Main.TWikiPreferences, WebPreferences, Users Main topic or inside individual topics. When people update a plugin this breaks TWiki Applications because you forget about this crazy limitation. Many people never discover this and assume it is just not possible. So we really want to change this spec back to what it once was. Before we can make this change we need to address the security aspects.

An analysis of all current plugins that are known to work with TWiki 4 shows this plugin as the only plugin with such a very dangerous setting that potentially can allow an attacker to place files with dangerous executable content anywhere on the filesystem.

For the XMLDIR this setting should really be kept in the $LocalSite.cfg.

Even without the changes spec, it is still a very bad idea to have settings in a plugin topic that defines path names.

I hope the author sees this in near future and release an updated plugin within near future so we do not have to worry about this plugin if/when we decide to implement the setting hierarchy.

-- KennethLavrsen - 16 Aug 2006

XMLDIR setting now removed.

-- PatrickDiamond - 16 Aug 2006

That was quick! Thank you for fixing this so speedily.

Where did the settings go? There is no indication in the documentation.

-- PeterThoeny - 16 Aug 2006

For 4.0+ the plugin now uses TWiki::Func::getWorkArea($pluginName) to replace XMLDIR, for pre 4.0 it attempts to use a sensible default that if need be can be overridden within the plugin.

-- PatrickDiamond - 17 Aug 2006

I have now updated the documentation to indicate the default location of the xmldir contents and how to override them.

-- PatrickDiamond - 17 Aug 2006

I've checked in the latest version from twiki.org into svn. The version there was a 2004er one.

-- MichaelDaum - 23 Jan 2007

I fixed two small typos in the dbiselect example: A missing quote and an added >

-- DougClaar - 06 Jun 2007

One other comment: Your dbi interface (I guess? or else this interface) might change case on you. For example, in MySQL, my table has employeeNumber, but I had to specify xsl:value-of select="employeenumber"

-- DougClaar - 06 Jun 2007

Some annoyances in the apache error log:

edit: Use of uninitialized value in pattern match (m//) at /var/www/html/twiki
/lib/TWiki/Plugins/XmlQueryPlugin.pm line 121.

edit: Use of uninitialized value in substitution (s///) at /var/www/html/twiki
/lib/TWiki/Plugins/XmlQueryPlugin.pm line 137.

Looking at the code:

    # Get plugin preferences 
    my ($cl,$ct);
    if( $TWiki::Plugins::VERSION < 1.1 ) {
        # Cairo codebase
        $debug  = &TWiki::Func::getPreferencesFlag("\U$pluginName\E_DEBUG") || $debug;  
    } else {
        # Dakar codebase
        $debug  = TWiki::Func::getPluginPreferencesFlag('DEBUG') || $debug;
    }

    # Setup cache size limit
    my ( $cl_value, $cl_type ) = ( $cl =~ /^\s*([0-9]*)\s*([a-z]*)\s*$/ );        ### 121: $cl not defined
    if ( defined $cl_value ) {
        if ( not defined $cl_type =~ /^m/i ) {
            $cachelimit = $cl_value * 1024 * 1024;    # megabytes default
        }
        else {
            if ( $cl_type =~ /^k/i ) {
                $cachelimit = $cl_value * 1024;       # kilobytes
            }
            elsif ( $cl_type =~ /^g/i ) {
                $cachelimit = $cl_value * 1024 * 1024 * 1024;    # gigabytes!!!
            }
        }
    }

    # set cache default timeout
    $ct =~ s/^\s*//;                                       ### 137: $cl not defined

Trying to figure out what the intent was here. It appears that it is supposed to get parameters for the cache stuff, but clearly it doesn't work. I've attached a quick patch that shuts up the uninitialized value messages.

-- DougClaar - 07 Jun 2007

Just to get some clarification possibly on "When a topic is saved an XML version of the topic is also saved. This contains data held in TWiki tables, actions and metadata."

Does that mean any twiki page will get saved with an XML version, even if the plugin isn't referenced on that page? Like just by having the plugin installed, this will happen?

-- MichelleHedstrom - 16 Aug 2007

Did anyone get this plugin to work on a Mac? I get a server error:

Module XML::LibXML () failed to load Can't locate loadable object for module XML::LibXML::Common in @INC

-- ArthurClemens - 23 Nov 2007

• XmlQueryPlugin.patch: Change to get rid of uninitialized vars in web server log