In trying to track down the cause of UnresolvedVarsInURL I've started looking at HTTP 1.0 vs 1.1. I'm getting really confused as to whether or not twiki is 1.0 or 1.1 compliant. Or neither, as some comments seem to indicate.

The topics ProxiedIncludesBrokenImplementationBug, ChunkedTransferEncoding, HttpRequestAndVirtualHosts, NetCantGetURLFromVhosts, ProperIncludeUrls all seem to inidicate that twiki is claiming 1.1 compliance but is really only 1.0 compliant. Yet all of those topics are also marked FeatureDone or BugResolved, but very little info on how this was fixed.

Would somebody who understands this stuff give a definitive answer?

-- MattWilkie - 03 Jan 2005

From IRC:

[16:29] <MS-> For MattWilkie: TWiki is barely 1.0 compliant, let alone 1.1 compliant
[16:30] <MS-> Where it matters is when TWiki is making remote includes
[16:30] <MS-> Specifically when acting as a client, it sends a line which is of the form:
[16:30] <MS-> GET /some/page.html HTTP/1.0
[16:30] <MS-> Host: Somehost.com
[16:31] <MS-> the 1.0 there indicates to the _server_ what HTTP capabilities the _client_ (twiki in this instance) can handle
[16:31] <MS-> Since TWiki does not use LWP or similar it will stay this way since TWiki doesn't implement a number of MUSTs in the HTTP/1.1 specification
[16:31] <MS-> (MUSTs for _clients_)
[16:31] <MS-> As a wiki though twiki is a CGI application, not an HTTP server
[16:32] <MS-> And hence in *that* instance HTTP version is irrelevant
[16:34] <MS-> Oh I suppose there is another area where HTTP compliance is an issue
[16:35] <MS-> That specifically of valid http URLs
[16:36] <MS-> If you create a free link of the form [[free & link]] IIRC that will cause twiki to create a link to "http://yourserver.com/twiki/bin/view/Someweb/Free Link" - note the space. This is actually a breach of the http spec since the link should be: http://yourserver.com/twiki/bin/view/Someweb/Free%20Link
[16:36] <MS-> The reason for that is because spaces aren't allowed in URLs

Question from me; if TWiki did use CPAN:LWP, is that the route to HTTP1.1 compliance?

-- CrawfordCurrie - 04 Jan 2005

iiuc, not only would CPAN:LWP provide a route to HTTP/1.1 compliance, it should also be a way to get https:// includes to work

-- WillNorris - 19 Jan 2005

I found this on the W3C forums:

> The http specs mention that content-length should not be zero... though
> people are using it... and netscape takes it to mean
> that 'download until disconnect'

That is a bug in Navigator. "Content-length: 0" means no message body.

> Is there a way to tell the browser without using this unpredictable
> method that, the data size is dynamic.

Not sending a Content-Length in HTTP/1.x. Using Transfer-Encoding: chunked in HTTP/1.1.

This basically says the current TWiki implementation is wrong - it sets a 0 content-length when the content length is not known. Several of us have noticed incomplete page downloads from TWiki, and I am suspicious that this may be related. When I fix the code (see DevelopBranch) to not send a content-length when the content length is not known, the problem seems to go away. Perhaps browsers are finally catching up with the HTTP1.1 spec?

-- CrawfordCurrie - 10 Jan 2005

I put in the Content-Length originally because it can improve performance - however it has caused problems before, so if it's tough to get the actual length right it can probably be omitted.

LWP would help for all the TWiki INCLUDE directives where TWiki's current HTTP support is quite odd and causes problems. We shouldn't need to do much on the normal page side apart perhaps from Content-Length.

-- RichardDonkin - 11 Jan 2005

After a bunch of to-ing and fro-ing, I discovered three things:

1. TWiki should have had $| = 1 for a long, long time now (switches off output buffering)
2. Setting STDOUT->blocking(0) is a really bad idea (my fault, I re-used code from elsewhere)
3. Setting content-length is a good idea, and can be done easily enough. I changed the way the header functions work in TWiki.pm to make it much more obvious and natural.

The DEVELOP branch r3471 now works, and the changes made during the debug process seem to have improved Matt's position as well, as an unintended but welcome side-effect.

Many thanks to MichaelSparks and SvenDowideit for invaluable help when debugging this.

-- CrawfordCurrie - 12 Jan 2005

How does that work when the content length is unknown in the beginning? Such as in a formatted search that takes minutes to search all webs in a large TWiki and returns hits bits by bits?

-- PeterThoeny - 18 Jan 2005

A search doesn't get a content-length. Only pages where the length is known get the content length set. Note that inline searches have to be fully satisified before the page is rendered, so the page length is known and the content-length can be set.

-- CrawfordCurrie - 18 Jan 2005