[PROPOSAL] TWiki and mirroring

Anderson Lizardo lizardo at linuxfromscratch.org
Sun Aug 1 12:52:23 PDT 2004

Hi guys,

I know this had been discussed before, but I still have some doubts/concerns 
regarding how the TWiki mirroring will happen, and maybe some mirror admins 
share them too. Here they are:

- The mirrors will require a TWiki installation to be able to process the data 
files. Although TWiki will be mirrored along with the data, **the partition 
where the mirrored data is stored will need to be mounted with "exec" 
option**. This is default for "mount" but some mirrors would have been used 
"noexec" for improved security and therefore the Perl scripts will not run on 
them. The same occurs for chrooted Apache installations that may not have 
access to /usr/bin/perl (except if they install Perl itself on the chroot...)

- Although we can garantee that we will always keep the TWiki installation 
up-to-date with the most recent fixes, we cannot garantee that TWiki itself 
is bug-free. The mirrors have to know that. Some may even argue that this is 
a strong point against allowing executable data to be stored on their 
servers, and I will have to agree here.

- AFAIK, there are mainly three types of Apache installations: the "global", 
non-chroot one, where all VHOSTs share the same Apache installation (although 
they have separate directory trees and file permissions); the global one, but 
inside a chroot environment; and the very restrictive one, where each single 
VHOST has its own root tree inside a chroot environment. I don't know if any 
mirroring site uses this last, but some services offering shell access 
(Savannah?) lock-in each user account on its own chroot jail.

- http://test.linuxfromscratch.org/twiki/bin/view/Main/WebsiteMirrors says 
that port 873 will be used by the rsync daemon. AFAIK, this is a low port and 
requires special privileges to be bound. Can rsync's bind port be changed so 
it can be run by a regular user?

Now I'll stop complaining and offer some suggestions for solutions :)

- Adapt TWiki so we can completely separate the non-executable data from the 
executable one (e.g. scripts, modules [do they really need to be 
executable?]). Then host the static data, **already processed to HTML** 
(possibly using http://twiki.org/cgi-bin/view/Plugins/PublishAddOn), on a 
separate VHOST (e.g. www.linuxfromscratch.org). The scripts will be on 
another VHOST (edit.lfs.org, scripts.lfs.org or just lfs.org, but see below).

- mirror the static data using the pushing method suggested by J. Huntwork. 
Even if we have to continue to use a non-CMS solution, I think we should move 
to push rsync'ing as it resolves another long-standing problem (mirrors being 
constantly outdated).

- I remember some people arguing that enabling edits only for belgarath would 
increase bandwidth usage. I offer a possible solution for this: additionally 
to having mirrors hosting static data, some which have no concerns regarding 
the items above can co-operate with belgarath on hosting the scripts and 
non-processed data (i.e. a complete TWiki installation). Those include 
possible mirrors which already host TWiki-powered projects and feel confident 
about it (as anyone else should :). The requirements for these "special" 
mirrors will be no different from what we have currently on 

- Move from our current CGI/Apache authentication method to https, as 
suggested by Jeroen. I have my doubts if the current method will work well 
with these cross-site redirections from "edit mode" to "view mode" and 
vice-versa. Additionally, CGI::Session likes to create small cookie files on 
server's /tmp dir (some belgarath users may have noticed how messed 
is /tmp :) and is not scalable enough, IMHO (the docs explain that Apache 
authentication becomes slower the bigger the .htpasswd database is).

The HTTPS authentication will (hopefully) work like some popular Webmail 
services which offer secure logins. You start with a non encrypted login 
form, which sends your login data through POST to the login server and then 
is redirected to the service's main page, no authenticated. Of course, we can 
keep the user authenticated during all session duration (like SourceForge), 
but I wonder if this will have a big impact on server's performance.

Sorry for the long post. I hope to have expressed all possible future concerns 
regarding TWiki, and to have presented some food for possible solutions. 
Comments, please.
Anderson Lizardo
lizardo at linuxfromscratch.org

More information about the website mailing list