Includes of remote pages using includes fail

Using a proxy with TWiki fails.

• It doesn't send a proxy-HTTP request.
• It claims to send an HTTP 1.1 request, but fails to send a suitable minimal 1.0 request
• All requests are marked 1.1 compliant but are in fact at best 1.0 compliant

This request is sent:

-    my $req = "GET $theUrl HTTP/1.1\r\n";

But the HTTP/1.1 spec requires the inclusion of a Host header. Downgrading this to:

+    my $req = "GET $theUrl HTTP/1.0\r\n";

Results in a minimal valid HTTP/1.0 request. Given responses for 1.1 could be chunked encoding etc, which TWiki does not handle it is more appropriate in this case to downgrade the request - most webservers will do so anyway.

The request is however totally invalid for a web proxy - an explicit web proxy expects a request of the form:

+        $req = "GET http://$theHost$theUrl HTTP/1.0\r\n";

Also this needs to be done before the host is added to the request, and any authentication tokens are added.

The attached patch fixes these issues. I suspect based on the small code review though that there might be further issues with this code.

Test case

%INCLUDE{"http://www.unitedmedia.com/comics/dilbert/"}%

Environment

TWiki version:	TWikiRelease01Feb2003
TWiki plugins:	DefaultPlugin EmptyPlugin InterwikiPlugin
Server OS:	SuSE Linux 8.0 (Kernel 2.4.18)
Web server:	Apache 1.3.23
Perl version:	5.6.1 built for i586-linux
Client OS:	Any
Web Browser:	Any

-- MichaelSparks - 09 Jul 2003

Follow up

• Patch is against BeijingRelease ( TWikiRelease01Feb2003 )
• To use patch:
  cd ~twiki # or whereever your twiki root is
cd lib/TWiki
patch -p0 < /tmp/PROXYBUG.patch

I originally set this bug to BugResolved, since the fix is available below. I've changed this to BugAssigned since it's not checked in (obviously).

---

An alternate description with an alterante implementation is at: ProxyUsing . A minor change from that bug fix needs to be merged with this bug fix. Unless these changes are made, TWiki's proxy support is broken beyond use.

-- MichaelSparks - 20 Jul 2003

Fix record

I've applied this fix into CVS. A couple of small changes that make a big difference. In my testing using Apache 2.0 configured as a proxy the patch got the functionality working. Similarly the change to HTTP 1.0 made includes of HTTP work.

-- JohnTalintyre - 28 Sep 2003

Can anyone test this fix to see if it works with virtual hosts? Virtual hosts seem to require HTTP/1.1. See related topics:

• HttpRequestAndVirtualHosts enhancement, CVS change http://cvs.sourceforge.net/viewcvs.py/twiki/twiki/lib/TWiki/Net.pm?r1=1.11&r2=1.12
• NetCantGetURLFromVhosts fix, CVS change http://cvs.sourceforge.net/viewcvs.py/twiki/twiki/lib/TWiki/Net.pm?r1=1.17&r2=1.18

-- PeterThoeny - 30 Sep 2003

I installed this a couple of months ago. It works reliably, accessing both standalone and virtual hosts.

-- MartinCleaver - 01 Oct 2003

Great, thanks Martin for the clarification. This bug is closed now.

-- PeterThoeny - 30 Sep 2003

Category: TWikiPatches