Tags:
create new tag
, view all tags

Feature Proposals » Mdrepo to use Sereal

Summary

Current State: Developer: Reason: Date: Concerns By: Bug Tracking: Proposed For:
AcceptedProposal HideyoImazu AcceptedByReleaseMeeting 2016-07-21     KampalaRelease

Edit Form

TopicSummary:
CurrentState:
CommittedDeveloper:
ReasonForDecision:
DateOfCommitment:   Format: YYYY-MM-DD
ConcernRaisedBy:
BugTracking:
OutstandingIssues:
RelatedTopics:
InterestedParties:
ProposedFor:
TWikiContributors:
 

Motivation

Mdrepo may lose data if the mdrepo data files are shared with NFS.

Description and Documentation

Mdrepo is a mechanism in TWiki core to store site and web metadata. Use of it is optional (not used by default). It's primarily for a large installation having thousands of webs.

So far, mdrepo has been using Berkeley DB. But Berkely DB is not designed to store data on a shared NFS file. There was a case where several records were lost on a TWiki installation consisted of multiple servers sharing files by NFS.

Mdrepo is for a large installation, which is likely to consist of multiple servers behind load balancer. So using an NFS-unsafe mechanism is bad. That said, instead of tying a hash to a Berkely DB file, storing a serealized hash data and retrieving it is better. Still, there is a chance of a record insertion/change/deletion being lost if multiple servers update simultaneously. But that's better than the current situation where records not touched may be lost or altered.

Among ways to serealize/de-serialize data in Perl, Sereal::Encoder and Sereal::Decoder seem promising. The webs mdrepo file having 5,000 records occupies 700 kilobytes of Berkeley DB file. The same hash data serealized by Sereal::Encoder should be similar in size. In this time and age, handling something smaller than 1 megabyte shouldn't be a problem.

Examples

Impact

Implementation

-- Contributors: Hideyo Imazu - 2016-07-21

Discussion

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r2 - 2016-07-22 - HideyoImazu
 
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.