Subject: KampalaMeetingLog2015x12x19.txt Date: December 17, 2015 4:13:49 PM PST [2:58pm]  HaraldJoerg joined the chat room. [2:59pm] HaraldJoerg: Good evening! [3:01pm] PeterThoeny: guten abend HaraldJoerg! [3:02pm] PeterThoeny: what's new? [3:03pm] HaraldJoerg: Lots of ideas, lack of time [3:04pm] PeterThoeny: sounds very familiar [3:04pm] PeterThoeny: my daughter is now in prague and will be in switzerland and munich later on [3:05pm] PeterThoeny: she likes to travel, wondering where she got that bug from [3:06pm] HaraldJoerg: I'd like to travel more, but shun the organizational overhead [3:08pm] PeterThoeny: wondering if hideyo-san joins us [3:11pm] PeterThoeny: time check: +11 min [3:11pm] PeterThoeny: i think we should start [3:11pm] PeterThoeny: informal meeting, or based on agenda? [3:12pm] HaraldJoerg: As you prefer... [3:13pm] PeterThoeny: so let's do informal discussion [3:13pm] PeterThoeny: http://twiki.org/cgi-bin/view/Codev/TestingTWikiThePerlWay [3:13pm] PeterThoeny: your proposal [3:14pm] PeterThoeny: looks good [3:14pm] PeterThoeny: feel free to hack away on this! [3:14pm] HaraldJoerg: Ok [3:14pm] PeterThoeny: i have no further input on this [3:14pm] PeterThoeny: http://twiki.org/cgi-bin/view/Codev/TWikiFunclogFilePath [3:14pm] HaraldJoerg: This is bugging me for some time, whenever I come from "traditional" Perl to TWiki [3:15pm] PeterThoeny: yea, it could help us also get additional contributors [3:15pm] PeterThoeny: on a related note, switching over to git would help too [3:16pm] PeterThoeny: nothing urgent, but eventually needed [3:16pm] PeterThoeny: back to: [3:16pm] PeterThoeny: http://twiki.org/cgi-bin/view/Codev/TWikiFunclogFilePath [3:16pm] PeterThoeny: proposed by hideyo-san [3:16pm] PeterThoeny: i have concerns abou exposing file path on the api [3:17pm] PeterThoeny: can be addresses in other ways [3:17pm] PeterThoeny: i'd like to get input from hideyo-san on my latest feedback [3:17pm] HaraldJoerg: If it is readonly - I would't have too much concern [3:18pm] PeterThoeny: well, eventually we shoud use file locking [3:18pm] HaraldJoerg: An extension developer could always easily hack his way into %TWiki::cfg [3:18pm] PeterThoeny: yes [3:18pm] PeterThoeny: the specific need for hideyo-san is to output debug info of ploticus to twiki's debug.txt [3:19pm] HaraldJoerg: Ah - I see that's your current comment, I had not read this before the meeting [3:19pm] PeterThoeny: any other feedback on this, or on other features? [3:20pm] HaraldJoerg: I have another idea, still undocumented... [3:20pm] HaraldJoerg: Similar to perldoc, I'd like to have an easy access to "readable" documentation of TWiki modules [3:21pm] PeterThoeny: we have a plugin for that [3:21pm] HaraldJoerg: Oh? I seem to have missed that [3:21pm] PeterThoeny: http://twiki.org/cgi-bin/view/Plugins/PerlDocPlugin [3:22pm] PeterThoeny: if installed it shows perl docs of any module [3:22pm] HaraldJoerg: Ah, that's the wrong way around.  I want to see the twiki style documentation, not POD [3:23pm] PeterThoeny: can you elaborate? [3:23pm] HaraldJoerg: And right now I see that the plugin is supposed to do that [3:23pm] HaraldJoerg: I need to have a look at that. Thanks for the pointer! [3:24pm] PeterThoeny: feel free to pick up on the plugin with additional features [3:24pm] PeterThoeny: we used to have it enabled on twiki.org long time ago [3:25pm] PeterThoeny: in 2002 was the last real dev work [3:25pm] PeterThoeny: after that just cosmetic [3:25pm] HaraldJoerg: So maybe it is already perfect [3:25pm] PeterThoeny: on to extensions [3:25pm] PeterThoeny: well, we are on extensions [3:25pm] PeterThoeny: http://twiki.org/cgi-bin/view/Plugins/WatchlistPlugin [3:25pm] PeterThoeny: only this plugin has been updated since last meeting [3:26pm] PeterThoeny: bug fix [3:26pm] PeterThoeny: actually, one more: [3:26pm] PeterThoeny: http://twiki.org/cgi-bin/view/Codev/ExportCompleteWikiData [3:26pm] PeterThoeny: http://twiki.org/cgi-bin/view/Plugins/TwikiLibForPhpAddOn [3:26pm] PeterThoeny: it has been updated in github [3:27pm] PeterThoeny: the add-on has no package attached, e.g. not visible to twiki's installer [3:27pm] PeterThoeny: wondering if we should package it [3:28pm] PeterThoeny: i will provide feedback to viliam simko [3:28pm] PeterThoeny: that is all on extensions i have [3:28pm] PeterThoeny: anything? [3:28pm] PeterThoeny: on to misc: [3:28pm] HaraldJoerg: It is PHP, not Perl... usual TWiki extension installation procedure would't work [3:28pm] PeterThoeny: good point [3:29pm] PeterThoeny: so just leave it [3:30pm] PeterThoeny: however, it likely will show up in the installer, and complain when installing due to missing package [3:30pm] PeterThoeny: some inconsistency we can live with [3:30pm] PeterThoeny: anything misc? [3:31pm] HaraldJoerg: Nothing from me [3:31pm] PeterThoeny: i have something [3:31pm] PeterThoeny: i do consulting work for broadcom [3:31pm] PeterThoeny: they use twiki [3:32pm] PeterThoeny: but my first project is not twiki related [3:32pm] PeterThoeny: i lead a midsize internal tool project [3:32pm] PeterThoeny: we use mongodb as the database [3:32pm] PeterThoeny: and node.js for the app logic on the server [3:33pm] PeterThoeny: browser side is jquery, ajax and such [3:33pm] PeterThoeny: i really like this dev env [3:33pm] PeterThoeny: node.js with express webserver is very easy to use [3:33pm] PeterThoeny: and powerful [3:33pm] HaraldJoerg: Yes, that's pretty agile [3:35pm] PeterThoeny: there is mongoose to tie data structures directly to docs in mongodb [3:35pm] PeterThoeny: really cool and easy [3:35pm] PeterThoeny: say, you have a collection with users in mongodb [3:36pm] PeterThoeny: you create a javascript object of any hierarchy (depth), and tie it to mongodb [3:36pm] PeterThoeny: that's it [3:36pm] HaraldJoerg: Yes, for some practical problems this non-SQL stuff is a very clean solution [3:36pm] PeterThoeny: now you simply create new user docs and change user docs simply by manipulating javascript objects [3:37pm] PeterThoeny: which brings me to twiki: [3:37pm] PeterThoeny: i think mongodb would be an ideal database backend for twiki [3:37pm] HaraldJoerg: Object persistence through RDBMS is always a bit convoluted compared to mongodb [3:37pm] PeterThoeny: because mongodb is a schemaless db [3:38pm] HaraldJoerg: I guess this would need work on a TWiki topic object model [3:38pm] PeterThoeny: this allows us to simply store a twiki topic as a mongodb doc,and include the twiki form data, and other page structure in a queriable way [3:39pm] PeterThoeny: yes and no [3:40pm] HaraldJoerg: And with all database-oriented backends, I'm concerned about efficient version management [3:41pm] PeterThoeny: no: we could store the twiki topic text sans meta data as a blob, and all meta data as objects. in addition, we can parse page data and store some of that as objects, such as headings, wikiwords (for backlinks). etc [3:42pm] PeterThoeny: yes: full twiki object model - that could mean more strict structure for content in a twiki page [3:42pm] PeterThoeny: so i think the former is more flexible [3:43pm] PeterThoeny: with the former, we could simply keep the current file-based store and versionning, and use the mongodo doc as a fast cache [3:43pm] PeterThoeny: e.g. master of record is still the twiki topic text [3:44pm] PeterThoeny: another approach would be also the former (e.g. no: ), but with mongodb doc as the actual store [3:45pm] PeterThoeny: in that case we need to create a mongodb doc for each version, and keep track of the top version for fast queries [3:46pm] HaraldJoerg: Yes, that emphasises the point about efficiency of versioning [3:46pm] PeterThoeny: the mongodb as cache is faster to implement than the mongodb as store [3:47pm] PeterThoeny: in mongodb you can create multiple indices, so a with an indexed "top version" flag it would be quick to query all top revs [3:48pm] PeterThoeny: one catch: mongodb keeps all indexes in memory, so be careful not to create indexes on too big datasets [3:48pm] HaraldJoerg: Sure.  But every version is a full copy in the database [3:49pm] PeterThoeny: i am not concerned about data size on disk [3:49pm] HaraldJoerg: RCS may be old, but is efficient for topics with hundreds of versions of small changes [3:49pm] PeterThoeny: a typical twiki installation has 10:1 or bigger ratio of attachments : topic text [3:50pm] PeterThoeny: so, as long as attachments are stored in the file system we are fine [3:50pm] PeterThoeny: actually, attachments need to be considered as well [3:50pm] HaraldJoerg: Almost agreed: For attachments, I seldom see more than one version, for topics, we have several which are continuously updated [3:51pm] PeterThoeny: in fact versioning attachments is the biggest thing, space-wise [3:51pm] HaraldJoerg: Yes, because they're binary, RCS isn't nearly as efficient as with text [3:52pm] HaraldJoerg: For most attachments in "my" productive TWiki, versioning is not required at all [3:52pm] PeterThoeny: so in this case, storing attachments as blobs in mongodb would be feasable too [3:53pm] PeterThoeny: one important detail is twiki's query language [3:53pm] PeterThoeny: that needs to be mapped to mongodb queries [3:54pm] PeterThoeny: twiki's regex search will not be possible unless we store the whole topic text verbatim as a blob in mongodb, or keep the topic text in the file system [3:54pm] PeterThoeny: so there are details to be worked out [3:55pm] PeterThoeny: one advantage of mongodb is that you can create repliacted databases [3:55pm] PeterThoeny: https://docs.mongodb.org/v3.0/core/replica-set-architecture-geographically-distributed/ [3:56pm] HaraldJoerg: File systems can do that, too [3:56pm] PeterThoeny: so that could be intersting for morgan stanley and other companies that have twiki installed in multiple datacenters [3:58pm] PeterThoeny: yes, possible on file system, but with manual plumbing [3:58pm] PeterThoeny: time check: +58 min [3:59pm] PeterThoeny: let me write up a proposal without committing to it (lack of time) [4:00pm] PeterThoeny: ok, shall we close the meeting? [4:01pm] HaraldJoerg: Fine for me... without Hideyo-san it doesn't make much sense to discuss the POSIX::tzset issue [4:01pm] PeterThoeny: a more technical description on replication at https://docs.mongodb.org/v3.0/tutorial/deploy-geographically-distributed-replica-set/ [4:01pm] PeterThoeny: yes [4:01pm] PeterThoeny: ok, i'll post he logs as usual [4:01pm] PeterThoeny: thank you and good night harald! [4:03pm] HaraldJoerg: TTYL - perhaps I won't be able to attend in two weeks [4:04pm]  HaraldJoerg left the chat room.