[PROPOSAL] TWiki and mirroring

Jeroen Coumans jeroen at linuxfromscratch.org
Mon Aug 2 01:32:51 PDT 2004

Anderson Lizardo said the following on 01-08-2004 21:52:
> Hi guys,
> I know this had been discussed before, but I still have some doubts/concerns 
> regarding how the TWiki mirroring will happen, and maybe some mirror admins 
> share them too. Here they are:

Well, as far as your concerns are shared with mirror admins; I haven't 
had such feedback yet on lfs-admin. So they're either silent or they 
were in agreement with our proposal :)

> - The mirrors will require a TWiki installation to be able to process the data 
> files. Although TWiki will be mirrored along with the data, **the partition 
> where the mirrored data is stored will need to be mounted with "exec" 
> option**. This is default for "mount" but some mirrors would have been used 
> "noexec" for improved security and therefore the Perl scripts will not run on 
> them. The same occurs for chrooted Apache installations that may not have 
> access to /usr/bin/perl (except if they install Perl itself on the chroot...)
> - Although we can garantee that we will always keep the TWiki installation 
> up-to-date with the most recent fixes, we cannot garantee that TWiki itself 
> is bug-free. The mirrors have to know that. Some may even argue that this is 
> a strong point against allowing executable data to be stored on their 
> servers, and I will have to agree here.
> - AFAIK, there are mainly three types of Apache installations: the "global", 
> non-chroot one, where all VHOSTs share the same Apache installation (although 
> they have separate directory trees and file permissions); the global one, but 
> inside a chroot environment; and the very restrictive one, where each single 
> VHOST has its own root tree inside a chroot environment. I don't know if any 
> mirroring site uses this last, but some services offering shell access 
> (Savannah?) lock-in each user account on its own chroot jail.
> - http://test.linuxfromscratch.org/twiki/bin/view/Main/WebsiteMirrors says 
> that port 873 will be used by the rsync daemon. AFAIK, this is a low port and 
> requires special privileges to be bound. Can rsync's bind port be changed so 
> it can be run by a regular user?

I'm sure it can.

> Now I'll stop complaining and offer some suggestions for solutions :)
> - Adapt TWiki so we can completely separate the non-executable data from the 
> executable one (e.g. scripts, modules [do they really need to be 
> executable?]). Then host the static data, **already processed to HTML** 
> (possibly using http://twiki.org/cgi-bin/view/Plugins/PublishAddOn), on a 
> separate VHOST (e.g. www.linuxfromscratch.org). The scripts will be on 
> another VHOST (edit.lfs.org, scripts.lfs.org or just lfs.org, but see below).

That seems very reasonable, and in general a good thing to do.

> - mirror the static data using the pushing method suggested by J. Huntwork. 
> Even if we have to continue to use a non-CMS solution, I think we should move 
> to push rsync'ing as it resolves another long-standing problem (mirrors being 
> constantly outdated).

Yes, this should probably be our first priority. Once we have the 
necessary mechanisms in place for this, it will also be easier to setup 

> - I remember some people arguing that enabling edits only for belgarath would 
> increase bandwidth usage. I offer a possible solution for this: additionally 
> to having mirrors hosting static data, some which have no concerns regarding 
> the items above can co-operate with belgarath on hosting the scripts and 
> non-processed data (i.e. a complete TWiki installation). Those include 
> possible mirrors which already host TWiki-powered projects and feel confident 
> about it (as anyone else should :). The requirements for these "special" 
> mirrors will be no different from what we have currently on 
> http://test.linuxfromscratch.org/twiki/bin/view/Main/WebsiteMirrors.

If we allow edits on multiple servers, how will they all synchronize? 
And how will conflicts be resolved in case two people are independantly 
editing the same file on different servers? There are lots of 
possibilities for races here which, AFAIK, TWiki can't handle or wasn't 
built/tested for.

Also, I have my doubts about the bandwidth savings we'd get from 
allowing edits on other servers. I made an estimate some time ago (can't 
seem to find it right now), but IIRC 10 edits on a day would be 0.01% of 
the current traffic (this includes actual edit traffic and the 
subsequent push to mirrors). And it seems highly unlikely that we'd have 
10 edits a day.

In short: this seems to be more trouble than it

> - Move from our current CGI/Apache authentication method to https, as 
> suggested by Jeroen. I have my doubts if the current method will work well 
> with these cross-site redirections from "edit mode" to "view mode" and 
> vice-versa. Additionally, CGI::Session likes to create small cookie files on 
> server's /tmp dir (some belgarath users may have noticed how messed 
> is /tmp :) and is not scalable enough, IMHO (the docs explain that Apache 
> authentication becomes slower the bigger the .htpasswd database is).

Yes, we already discussed this a bit when Jeremy was researching methods 
to integrate the current LFS user database with TWiki. Althoug I 
originally only suggested to use HTTPS for edit traffic, not for 
authentication. But does TWiki allow secure authentication?

> The HTTPS authentication will (hopefully) work like some popular Webmail 
> services which offer secure logins. You start with a non encrypted login 
> form, which sends your login data through POST to the login server and then 
> is redirected to the service's main page, no authenticated. Of course, we can 
> keep the user authenticated during all session duration (like SourceForge), 
> but I wonder if this will have a big impact on server's performance.
> Sorry for the long post. I hope to have expressed all possible future concerns 
> regarding TWiki, and to have presented some food for possible solutions. 
> Comments, please.

As long as it's all feasable, most of it seems reasonable. And, like you 
said, we can already work out some things (like rsync push) before we 
migrate to TWiki.

Jeroen Coumans

More information about the website mailing list