Archive of HeadlinesPluginDev Discussions

First version posted. This plugin is the counterpart to TWikiSyndication, and it probably makes the SlashdotPlugin obsolete.

Technical detail: The plugin is a quick hack (that should work ). No XML parser is used, Perl regex is used instead to extract parts of the RSS feed.

Possible enhancements:

• Introduce a cache to improve the performance:
  • Store RSS XML source in file cache. File name could be simplified URL, e.g. slashdot_org_slashdot_rdf.xml for http://slashdot.org/slashdot.rdf.
  • At topic view, check for the timestamp of the cache file. Update cache file if older then REFRESH minutes.

• Introduce new official TWiki::Func::getUrl() function

• Make getUrl() more robust

This plugin does not work if installed at SF, I get this error: Can't read http://slashdot.org/slashdot.rdf (TWiki::Net::getUrl connect: No route to host). Any idea why?

-- PeterThoeny - 12 Jun 2002

Some plugin benchmarks on a large installation:

There is no measurable performance hit for topics that do not have %HEADLINES{...}%.

-- PeterThoeny - 12 Jun 2002

Good stuff - will have a go at this when I get time. The error message on SF has been reported before, I think it's due to the lack of the Host header in HTTP transactions generated by TWiki code such as getUrl().

Presumably you are using this at Wind River? Would be interesting to hear how you are using it, e.g. whether there is a specific goal in using RSS, and which RSS readers you have tried...

-- RichardDonkin - 14 Jun 2002

At work we have lots of product marketing and marketing material in TWiki. RSS feeds are a logical enhancement to display domain specific news for specific groups, for example pulling syndicated Telecom News from Moreover, http://www.syndic8.com/feedinfo.php?FeedID=686.

-- PeterThoeny - 15 Jun 2002

Interesting... BTW there is a useful patch for the Dec2001 TWiki code that helps this plugin work with virtual-hosted RSS feeds, see Support.HeadlinesPluginCantReadError - although you are probably running a TWiki beta at least

-- RichardDonkin - 18 Jul 2002

Hi! Thanks for this plugin, Peter. I added a few lines of code and implemented the caching that you were talking about. It uses the refresh attribute that you had hidden in an HTML comment in the HeadlinesPlugin configuration page to determine if the cached version is too old and should be retrieved again. The cache file is stored in the pub/WEB directory - where WEB is the name of a web that uses the plugin. The filename looks like "rss-" followed by an MD5 hash of the RSS URL. So, the two requirements for using my changes are that (a) there is a writeable pub/WEB directory for any web that uses the plugin and (b) that the MD5 perl module is installed. I'm attaching a patch file for the HeadlinesPlugin.pm file.. you'd also need to uncomment the refresh bit in the data/TWiki/HeadlinesPlugin.txt file. Let me know what else you need from me if you decide to integrate these changes.

-- RobDuarte - 30 Jul 2002

Implemented Rob's caching with some enhancements. Latest Plugin is attached.

-- PeterThoeny - 02 Aug 2002

The caching looks very useful, since most RSS readers don't have a cache themselves. As for the MD5 module, the Digest::MD5 module is required for TWiki on Windows at least (for register to work) - see WindowsInstallCookbook.

-- RichardDonkin - 02 Aug 2002

It looks like Digest::MD5 is a standard lib since Perl 5.005_03, see http://www.perldoc.com/perl5.005_03/

-- PeterThoeny - 02 Aug 2002

Just an ObAOL on caching -- very nice. Page loads with a couple of RSS feeds were getting outrageously long, but headlines are nice, it's a tough tradeoff to make between the two -- and now I don't need to! Page load is now very fast even with four headlines feeds. Kudos.

-- KarstenSelf - 04 Aug 2002

Actually, it's the Digest::SHA1 module that's needed on Windows, not Digest::MD5 (or at least latter is not installed specially in WindowsInstallCookbook.

Seems like MIME::Base64 is standard in both Perl 5.005_03 and 5.6.1 - in which case, how come so many Windows Perl users had problems? I thought it was an ActivePerl thing, but my copy of ActivePerl 5.6.1, build 631, includes both the Base64 and SHA1 modules... In any case, WindowsInstallCookbook can probably be simplified.

If someone has a spare machine where they can do a vanilla install of Cygwin Perl, it would be great to check whether this includes these two modules, as it should.

-- RichardDonkin - 04 Aug 2002

Is anybody else seeing a problem with the feed: http://www.moreover.com/cgi-local/page?o=rss&c=Aerospace%20and%20defense%20industry%20news In my headline plugin installation it comes up with a truncated error message like Error 302, which I think (and from some tcpdump snooping) is a redirect. I also tried it with p.moreover.com and the rest of the URL and came up with the same problem. Is this a known limitation of TWiki::Net::GetUrl?

-- JohnRouillard - 10 Aug 2002

When using this plugin with a twiki under mod_perl, I noticed that it implements the wonderful concept of "eternal caching" - or to put it otherwise, it seems not to refresh at all. It only seems to refresh after a server restart (which makes sense if the problem is related to mod_perl/twiki). Don't have the time to look into it right now - but I will over the weekend if there is no known solution.

-- JeroenVanDongen - 02 Sep 2002

I think this is great! A couple of issues/questions: 1) I can't seem to get the DEBUG to work. I set the debug fine, but don't seem to get anything in the debug log... anything I should know? 2) I'd like to make it simpler for people to add rss feeds to a page. Is there a way to embed a form on the bottom of a page that could do a simple reformat of a url and paste it into the page as if the user had edited it? Just not sure where to begin on this. Thanks.

-- RickMangi - 09 Jun 2003

Good plugin - I notice that it doesn't display feeds that include HTML in their $description correctly. Any chance to get this added?

-- PaulPetterson - 14 Jul 2003

I've made a modification to handle feeds encoded in UTF-8 and transform them into ISO-8859-1.. This needs the perl module Unicode::String.

-- LudovicDubost - 28 Oct 2003

Ludovic: Any chance to get a patch that makes this feature optional? Not everyone needs this; not all have Unicode::String installed; each additional module slows down TWiki a bit. Could be loaded conditionally based on a DECODE_UTF8 setting.

New version 29 Oct 2004 posted on Plugin topic:

• Fixed issue of eternal caching if ModPerl or SpeedyCGI is used
• Allow namless "http://..." parameter besides the href="http://..." parameter

-- PeterThoeny - 29 Oct 2004

Running HeadlinesPlugin with Perl 5.8.0 (on RedHat 9) yields Insecure dependency in open while running with -T switch at $TWIKIROOT/lib/TWiki/Store.pm line 986.

• $cacheFilename must be explicitely untainted. I include a patch as attachment.

-- AdrianWeiler - 31 Oct 2004

New version 31 Oct 2004 posted on Plugin topic:

• Fixed taint issue, thanks AdrianWeiler
• Small performance improvement

Used a modified taint2.patch with small performance improvement

-- PeterThoeny - 31 Oct 2004

As a result of work on EncodeURLsWithUTF8, there is some translation infrastructure within SVNget:lib/TWiki.pm - this uses modules such as CPAN:Encode and CPAN:Unicode::MapUTF8, which are loaded on demand when really needed, avoiding slowdowns for the most common cases. In fact, for the common case of UTF-8 to ISO-8859-1, there is no need to load a module as this is done algorithmically. Would be worth looking at this and refactoring slightly (mainly just renaming it) to make it more generally reusable as part of the plugin API.

-- RichardDonkin - 18 Nov 2004

Hi, I've extended this plugin a bit. See http://nats-www.informatik.uni-hamburg.de/NewsFeeds for examples. Get my hacks from here. As I started on an older version we might want to merge stuff.

-- MichaelDaum - 26 Nov 2004

Just a heads up...

It appears that HeadlinesPlugin can't handle rss feeds on https connections. And it can't handle 302 Moved messages.

I don't ahve time to isolate/fix these problems, but thought someone might like to know.

-- BruceDawson - 10 Jan 2005

HeadlinesPlugin doesn't handle CDATA sections. Specifically, it doesn't strip the CDATA wrapper from the various item fields. For instance, http://news.search.yahoo.com/news/rss?ei=UTF-8&p=slashdot uses CDATA wrappers around their text field values. Without stripping the CDATA tags, the browser won't display the contents of those fields.

-- MikeSalisbury - 14 Feb 2005

Here's my hack-allong on the HeadlinesPlugin which adds

• dublin core
• cdata rendering
• mapping of an article's subject to an image
• css to customize feeds

Examples: SlashdotNewsFeed, BoinBoingFeed, PennyArcadeNewsFeed

-- MichaelDaum - 07 Mar 2005

Thanks WillNorris for creating a zip that is Develop Branch compatable. I am fine with that even though the Plugin is flagged a "contact author first".

I can hand over the ownership of this Plugin to an interested person. Guidelines:

• Do not introduce mandatory CPAN dependencies
• Keep Plugin compatible with older TWiki releases (use SharedCodeDev if needed)
• Keep good performance

-- PeterThoeny - 11 May 2005

I'm getting a

Bad arg length for Socket::pack_sockaddr_in, length is 0, should be 4 at /usr/lib/perl5/5.8/cygwin/Socket.pm line 373.
 at /home/httpd/twiki/lib/TWiki/UI.pm line 138
...

-- FranzJosefSilli - 11 May 2005

does a previous version work for you (in CairoRelease)? here's the change i made, i can't see how it would break a working setup:

wbniv@ubuntu:~/twiki/twikiplugins/HeadlinesPlugin/lib/TWiki/Plugins$ cvs diff -r1.2 -r1.3 HeadlinesPlugin.pm
Index: HeadlinesPlugin.pm
===================================================================
RCS file: /cvsroot/twikiplugins/twikiplugins/HeadlinesPlugin/lib/TWiki/Plugins/HeadlinesPlugin.pm,v
retrieving revision 1.2
retrieving revision 1.3
diff -r1.2 -r1.3
128c128,133
<     my $text = TWiki::Net::getUrl( $host, $port, $path );
---
>     # figure out how to get to TWiki::Net which is wide open in Cairo and before,
>     # but Dakar uses the session object.
>     my $text = $TWiki::Plugins::SESSION->{net}
>       ? $TWiki::Plugins::SESSION->{net}->getUrl( $host, $port, $path )
>       : TWiki::Net::getUrl( $host, $port, $path );
>

i have seen that kind of error before, such as when running on sourceforge.net, which doesn't allow outbound HTTP requests. does a remote %INCLUDE% work for you?

• You were right about outbound HTTP requests. A remote %INCLUDE% produces the same error. What (where) is my problem? How do I solve it? -- FJ
  • sorry, other than the possibility of a firewall blocking those outbound requests, i have no idea. perhaps you should open a Support request? -- WN

while i have access to linux, solaris, and mac os x boxes ( see a pattern here? ), i ("un")fortunately have no way to test windows versions. are you able to research more?

-- WillNorris - 12 May 2005

Will: Conditional code should not be done with variables that do not exist in earlier TWiki versions or you risk warnings in the Apache log. Use conditional if( $TWiki::Plugins::VERSION >= 1.020 ) { (with whatever Plugins version cutover number you require)

-- PeterThoeny - 12 May 2005

getUrl() isn't provided by TWiki::Func, so there's no version of $TWiki::Plugins::VERSION i can check. the conditional code is exactly that, conditional, and should not produce any warnings in the apache error_log

• I disagree respectfully. First, some $unititalized{item} produces a warning, and even an error when using strict. Second, TWikiPlugins and TWikiFuncModule describe how to use $TWiki::Plugins::VERSION to detect a TWiki version at runtime. Even when using undocumented calls you can rely on this since core code enhancements and Plugin API enhancements usually go hand in hand (it might be out of sync for Alpha releases but that is OK since those are not used in a production environment). -- PeterThoeny - 17 May 2005

LynnwoodBrown, you tested this plugin under CairoRelease for me. would you please double check that there are no error messages? thanks.

-- WillNorris - 12 May 2005

Will - Sorry for not doing more detailed testing for you on this. At first, I was simply looking at whether rss feeds displayed OK and they do. Just now, I loaded the HeadlinesPlugin in a Cairo installation and got this error message in the apache log:

[Thu May 12 07:48:31 2005] view: Argument "slashdot.org" isn't numeric in subroutine entry at /usr/lib/perl/5.6.1/Socket.pm line 286.

However, I tried reloading the topic several times to see if I got the error message each time but did not.

-- LynnwoodBrown - 12 May 2005

Many sites are converting to ATOM and this reader is uanble to handle them

My SAGE plugin for the FirefoxBrowser can handle.

For example, the website http://taosecurity.blogspot.com/ has a ATOM feed http://taosecurity.blogspot.com/atom.xml but using this in

%HEADLINES{href="http://taosecurity.blogspot.com/atom.xml"}%

What does it take to have this plugin accomodate ATOM?

-- AntonAylward - 03 Jul 2005

Supporting ATOM sounds like a useful enhancement. ATOM was not on the radar screen at the time this Plugin was written. It should not be too difficult to add the new protocol. Ideally without any CPAN dependency.

Anton, you gave a poor rating for the Plugin documentation and examples. What do you miss in the docs? How would you improve the docs?

-- PeterThoeny - 04 Jul 2005

I used MichaelDaum's excellent HeadlinesPlugin-Hack in CairoRelease, but it doesn't seem to work in DakarRelease. I get this error message:

Can't use string ("www.xyz.de") as a HASH ref while "strict refs" in use at /export/webroot/laut/intranet/lib/TWiki/Net.pm line 80.

Does anybody know how to make this work in Dakar?

-- RainerHenze - 17 Nov 2005

Here's: Micha's latest HeadlinesPlugin hack for dakar including support for ATOM as well as a bunch of dirt to work around the most annoying utf8 chars.

-- MichaelDaum - 17 Nov 2005

Dankeschön Michael! Now I get another error-message:

Can't locate object method "default_header" via package "TWiki::Plugins::HeadlinesPlugin::RequestAgent" (perhaps you forgot to load "TWiki::Plugins::HeadlinesPlugin::RequestAgent"?) at /export/webroot/laut/intranet/lib/TWiki/Plugins/HeadlinesPlugin.pm line 769.

Can you help again?

-- RainerHenze - 18 Nov 2005

Try the new one. Changes:

• cleaned up getUrl()'s use of LWP::UserAgent
• cleaned up unicode to html entities conversion

Peter, can you take a look at this stuff? I'd like to get this into the release. If you don't have the time I'll package a new HeadlinesPlugin release.

-- MichaelDaum - 18 Nov 2005

Thanks Micha on providing the useful enhancements. Yes, please release a new version. I replied to your e-mail with details.

-- PeterThoeny - 18 Nov 2005

Thanks very much again. I'm glad that this is going to be in the official release.

-- RainerHenze - 21 Nov 2005

Uploaded a first pre-release.

-- MichaelDaum - 21 Nov 2005

Uploaded a second pre-release: use getWorkArea() if available.

The current pre-releases shape the features to be present in the upcomming 2.0 release of the HeadlinesPlugin.

Uploaded pre-release 4: added web.topic shortcuts for internal feeds; some utf8 recodings (still needs to be completed)

-- MichaelDaum - 02 Dec 2005

Hi Michael. Installed the 02 Dec version, got HeadlinesPlugin: invalid format of the href parameter when I put in feed from Webname.WebRss (and Webname.WebRss?skin=rss ). Full line is
%HEADLINES{"Sandbox.WebRss" limit="10"}%

-- MarcusLeonard - 04 Dec 2005

If you are using a recent dakar engine from svn then this got broken last weekend during the big relative-urls change. Fixed the HeadlinesPlugin to generate absolute urls again in the latest pre-release 5.

-- MichaelDaum - 05 Dec 2005