rsync push method: plan of attack

Anderson Lizardo lizardo at linuxfromscratch.org
Mon Aug 2 13:12:41 PDT 2004


Hi,

I'm going to list here the changes I'm planning to do on our current website 
installation to be able to enable rsync push for mirroring. The main reason 
to implement this setup even before switching to a CMS is because it 
(hopefully) fixes a long-standing problem with mirroring: there is a 
considerable delay between the changes on the repository getting effective on 
the main website, and some more between the website being synced with the 
mirrors.

The first delay is due the fact the LFS main site is updated by a cron job run 
periodically. A change to the repository (e.g. a news item added) does not 
get on-line on real time, except if one of the website team members run the 
script manually. This stage can be fixed by using the post-commit hook 
feature on Subversion.

The second delay is caused by the current rsync "pull method" (is this the 
right name?). For each mirror is given a "time slot" with which they setup a 
cron job to run rsync at that time and fetch the website changes from the 
main site. This setup is problematic because we cannot garantee the mirrors 
will become up-to-date on a reasonable time. Additionally, the many mirrors 
we have, the bigger is the discrepancy between the first updated mirror and 
the last one. And lastly, we have no control if mirrors become offline for 
some reason, or cannot successfully finish the rsync'ing, leading to broken 
pages.

A solution for the last mentioned problem would be a push method for rsync. 
Every time a change is made to the repository, the post-commit hook script 
updates the affected files accordingly, and then iterates through each 
mirror, "pushing" the changes to them. If some mirror does not seem to 
respond to the push request, or the rsync command exits with error, that 
specific mirror is disabled on the mirrors list. Note the this last feature 
would require the mirror list being only on the main LFS site indexes, and 
the mirrors indexes just redirect to the local <project>/news.html 
(using .htaccess?). This is nice IMHO, because it removes the redundant 
"select mirror" page from the mirrors.

My next message (yes, another :) will give more details of the changes I'll 
make to the current website update script to support the post-commit hook 
adequately.

Comments?
-- 
Anderson Lizardo
lizardo at linuxfromscratch.org
http://www.linuxfromscratch.org/



More information about the website mailing list