Feature Proposal: Create way to import external data into TWiki

Motivation

For any number of reasons, users may wish to import data from external source into TWiki, for example creating individual topics for each record in a database. Currently, there is no easy way to do this.

Description and Documentation

Some of the basic features I see as needed in this feature include:

• Automatically create a new topic for each database record, or create new topic for each file in a specified directory.
• Automatic topic-naming based on a defined format.
• Map database fields to fields in a pre-defined TWiki form.
• Appending one or more fields to the "whiteboard" area of the topic.

Additional features which would be "nice to have" include:

• Merging capability - i.e. inserting external db fields into a specific location in the new topic based on variables included in a topic template.
• Testing for duplicate topic names with some rules for merging data.

Representative user requirements:

• "We have created a form template to match the structure of a data table. We'd like to populate the forms by importing the data from the tables. What is the simplest method of automating this? (The data table is a large excel converted to whatever format you recommend: tab-delimited, csv, etc.)" Requested made here.

Examples

Impact and Available Solutions

Implementation

-- Contributors: LynnwoodBrown

Discussion

-- LynnwoodBrown - 15 Mar 2006

Great idea. I could certainly use this!

-- MeredithLesly - 15 Mar 2006

This sounds like a TWikiExtension.

-- RafaelAlvarez - 15 Mar 2006

Just a quick note: MartinCleaver mentioned to me at some point that perhaps we could leverage some of code from his batch registration feature for this purpose since that feature already offers ability to generate multiple topics from return-delineated list.

-- LynnwoodBrown - 15 Mar 2006

Ideally that would be done in a generic way for different types of input data.

For example, paste a spreadsheet into a TWiki form to generate a set of topics, one for each spreadsheet row. This would allow to convert spreadsheets into a TWikiForms based database.

Another example: Take an XML file that contains records of some sort, transform that with help of XSLT into a set of TWiki topics that build a TWikiForms based database.

-- PeterThoeny - 15 Mar 2006

TWiki::Data used by Registration has the code you need already. It will parse a file delimited by any given delimiter and return it as a TWiki table.

It occurred to me that we could turn it on by default looking for tabs - and turning these into tables.

-- MartinCleaver - 15 Mar 2006

There is code that then iterates down the said table and generates topics from it - in registration's case generating user topics containing field entries.

-- MartinCleaver - 16 Mar 2006

I considered doing this a while ago, but decided against it because it is so trivial to write a perl script to do it and (selfishly) that's all I needed

It would be excellent to have a general solution to this.

-- CrawfordCurrie - 16 Mar 2006

Well, in fact, I wrote a perl script to do it once we figured out that the twiki would find properly named and formatted pages (files) we inserted into the directory. For that script, see the attachment to HowToImportDataToForms

-- JaredFreeman - 17 Mar 2006

Martin -- TWiki::Data, by your description, solves an interesting problem, but it is different from the one I raised, which is generating pages in correct format from a data table. See my hack solution (just a script) cited above.

-- JaredFreeman - 17 Mar 2006