WebDAVPlugin just became the latest victim of a legacy "feature" - the conversion of three spaces into tabs on save, and the reconversion of tabs into three spaces on edit. It's not the first victim; all plugins have to be aware of this conversion, and it has caused bugs inall verbatim and forms. The question is, why do we bother doing it? So we can keep the very obscure and nasty piece of code inherited from JosWiki that generates lists?

Here's my understanding of behaviour:

• When a topic is saved via the TWiki::UI::Save interface, groups of three spaces get coverted to tabs.
• When a topic is opened for editing in the TWiki::UI::Edit interface, tabs get converted to three spaces.
• view jumps through hoops to protect tabs in verbatim
• Not sure what raw does, but it has to be aware of the problem as well

Importantly:

• Internal functions such as readTopic deliver topic text with tabs, not spaces.
• Plugins handlers deal with topic text including tabs - except beforeEditHandler, which has to deal with spaces instead.
• The code is peppered with places where tabs are converted to spaces and vice versa to handle all the special cases.
• The code isn't correct. The transformation is asymmetrical:
  • edit -> two_spaces tab space -> save -> edit -> six spaces -> save -> tab tab which probably makes your verbatim and pre output different

Is it all really worth all this confusion?

The proposal is:

1. smarten up the list generating code to treat tabs and groups of three spaces as equivalent
2. stop all other back and forth translation of spaces and tabs
3. provide a script to migrate existings topics (one shot) from tabs to 3 spaces - CN

Yes, there is a legacy of topics that contain tabs that will appear differently in the next edit, but topic views would not change, and the code would get generally simpler and less confusing.

-- CrawfordCurrie - 18 Sep 2004

I agree. There should be identity between what is typed and what is saved. I see two points where this is not the case:

• 3 spaces / tabs in lists: Crawford proposition seems OK
• carriage returns + newline instead of end of lines: we should:
  1. remove all \r anywhere
  2. provide a legacy-migration script to convert data
  3. strip \r from any read data

Any objections?

-- ColasNahaboo - 11 Oct 2004

Oooh - I never intended there to be a 1:1 correspondance between the raw topic and the saved format. All I want to be sure of is that there is a symmetrical transform between the edited topic and the store, so that if I save "X" then I get back "X" the next time, and not "Y". Of course, the easiest way to ensure the transformation is symmetrical is to make both sides identical, but that's only one way.

Colas, I don't think a conversions script is needed. Old data can continue to work fine as long as the transformation is preserving. Because topics are currently stored using tabs, opening them again, editing and saving the tabs will result in identical behaviour as today. All it requires is for a tab to be interpreted as three spaces when laying out lists.

-- CrawfordCurrie - 11 Oct 2004

Should the save topic format be made equivalent to the browser raw format? Please give reasons if you disagree Do you agree or disagree? Please select one of the radio buttons below and click the button. I 1 - Strongly disagree   2 - Disagree   3 - am Neutral   4 - Agree   5 - Strongly Agree

5	PeterKlausner	21 Oct 2004
5	ColasNahaboo	12 Oct 2004
5	AntonAylward	11 Oct 2004
2	CrawfordCurrie	11 Oct 2004
5	MartinCleaver	11 Oct 2004

-- MartinCleaver - 11 Oct 2004

Fine. the restatement has helped elicit clarity. Seems you are neutral about the 1:1 equivalence: I'd prefer we simplify the core and move compatibility with the old way to a Plugins.CairoLegacyPlugin.

-- MartinCleaver - 11 Oct 2004

Old data can continue to work fine ... in the main TWiki engine, but it will continue to bother addons & plugins. And editing & saving will mess the dates. For instance I was bitten by this just yesterday: if you have cell contents in a table with more than 3 padding spaces in them (some people hand-edited the table for nicer looks), then the EditTablePlugin failed to read the data properly!

We should be 1:1 or we will be continuously bitten by unexpected behavior in many places.

-- ColasNahaboo - 12 Oct 2004

Polls have limitations. I am not going to vote since it would be a 4 and a 1.

Agree: Saved file format should be the same as edit format, e.g. three spaces instead of tabs.

Disagree strongly: Upgrade script to change existing content and TWiki code understanding only new format.

An upgrade script could be supplied, but the TWiki code needs to be aware of legacy tab format (I stated my reasons several times). Going forward, TWiki should only generates space format. That way we will get consisted data over time, and old data (from backups and different TWikis) still render correctly.

-- PeterThoeny - 12 Oct 2004

Given that the question in the poll is "Should the save topic format be made equivalent to the browser raw format?". Am I to read your answer "Saved file format should be the same as edit format" as being an "Agree"?

It does sounds like you are in favour of ImplicitUpgrade, but dislike the need for ExplicitUpgrade: I agree with you. Should we even consider downgrades?

Procedural matter, belonging on VotingVsDiscussion:

• This poll didn't capture the full extent of all the issues presented (i.e. of whether it should be an ExplicitUpgrade script). We'd have hundreds of polls if we did that. Yet, you can always vote on the poll and add your comment or even invalidate or add a poll if you think there is something more pertainent.
  • (Invalidating existing votes makes me think we need a "notify me if my name was mentioned less times on page than before" option).
    • "was my name in the diff of a recent revision" the recent bit is important because it couldn't just be latest, it would need proper date range searching. -- SamHasler - 20 Oct 2004

-- MartinCleaver - 20 Oct 2004

(1) Strongly agree to have (edit format == stored format).
(2) Strongly agree to render same as now, i.e. (1 tab -> 3 spaces).
-->

• Old topics need not be converted.
• Re-edited topics will loose the tabs.
• This may or may not result in the proper alignment -- like with most apps and proportional fonts.
• Rule (1) would be violated in the case of typing tabs,

These are not real problems anyway. Most browsers make it very hard to even enter a tab into the textarea. Maybe a config switch $expandTab would do it: While in legacy mode, render them 3 tabs. Otherwise, leave them alone.

In other words:

• remove conversion on the input
• keep current conversions on output (including store to edit widget)
• turn them off, when legacy free (read: in the far future...)

-- PeterKlausner - 21 Oct 2004

This would imply that you couldn't have tabs at all - any time you saved a tab, it would save as a tab, then get rendered as 3 spaces when output to the edit widget, then saved back as 3 spaces. At each revision, the user would be required to explicitly replace any instances of 3 spaces that should be tabs with tabs.

I think for simplicity, saved text should be identical to desired text. Even the minimal performance improvement (1 regexp per topic read) would be a pro. Personally, I'd be much more comfortable with no tabs than with no sets of 3 spaces. I think it smart to eliminate the transformation altogether, but to read a bullet point (and preferences) as preceded by either sets of 3 spaces or tabs.

Finally, don't make legacy functionality a plugin - there's an increasing pressure on the ordering of plugin execution, and it becomes very confusing to administer (since at times, it seems that two plugins need to manipulate each other's output via the same hook). Given this, a script would be nice that would create a new version of any topic that has a leading tab (only at the front of the line) in it, replacing all leading tabs with 3 spaces. The second part of the script could be a user-interactive part to replace non-leading tabs on a per-instance basis.

Note though that this would likely break a lot of Plugins, and somebody would likely need to go through the entire Plugin codebase and insert the (?:\t|   ) regexp everywhere.

-- OrrBernstein - 30 Nov 2004

I agree with Orr. The specific proposal here is as the topic is named - to stop space-tab conversion, everywhere. There would be no legacy conversion script (it should not be required).

-- CrawfordCurrie - 13 Feb 2005

Implemented in r3636 (DevelopBranch)

-- CrawfordCurrie - 16 Feb 2005

I've written a small plugin to replace tabs befor editing as a workarround for konqueror (see KonquerorBrowserIssues). Nevertheless, no browser can edit tabs: hitting tab will change the focus to the next input field but not insert a tab. So updating old web content becomes a nightmare. Actually, I'd still favour the one-stop sed script solution to convert all content offline. Why was that dropped?

-- MichaelDaum - 02 Apr 2005

Why not re-instate tab->space conversion in the edit script (but not the reverse space->tab transform)?

There are other editors (for example Kate via WebDAV) that will do a smarter job, but in this case we might as well just go with the flow.

-- CrawfordCurrie - 02 Apr 2005

Yep!!!

-- MichaelDaum - 02 Apr 2005