New mirrors requeriments. Was: Re: so long and thanks for the fish

Jeroen Coumans jeroen at
Mon Jul 19 03:43:14 PDT 2004 said the following on 19-07-2004 12:14:
> El Lun 19 Jul 2004 11:22 Jeroen Coumans escrbi�:
>>If anyone has problems with the new requirements, then please speak up.
> I have problems. Isn't possible to me run the rsync daemon or create an ssh 
> account. The mirror is hosted at a commercial server that 
> don't want activate that services.

We have a third option: static HTML export. We can let TWiki export HTML 
pages and let mirrors who can't use the "push" mechanisms mirror these 
HTML pages instead. It would mean that we'll have to seperate mirrors 
into two categories:
1. primary mirrors (TWiki mirrors; always up-to-date)
2. secondary mirrors (HTML; may be 12 hours out-of-date due to 
time-based synchronizing)

We'll have to figure out a selection mechanism though. My preference is 
to make it very transparant: you'll always get a primary mirror if 
you're logged in, otherwise you'll get a random mirror from the complete 
pool. Perhaps we can offer a select box for users to always get a 
primary mirror.

Anyway, in case of the HTML export, you won't have to change anything on 
your mirror. It would mean you aren't always up-to-date, but at least 
the user will be aware of that, which is better then the current 
mechanism, where mirrors can get out-of-date and we don't have any 
control over it, nor can we notify users of it. This has caused user 
frustration multiple times in the past.

> On the other side, this continous rsyncing with each mirror to synchronize each 
> TWiki edited page (and the direct connection to belgarath to make the the login 
> & page editions), don't could will increase the bandwhitd usage on belgarath?

Rsync is very bandwidth-friendly: it will only transfer the actual 
differences. Thus, if one page is changed, only that page will get 
transferred to all mirrors. How many pages will get changed per day? If 
the current rate of updating is any indication, we'll have to count per 
week instead of per day. But lets say that we'll have 10 updates per 
day. Pages are only 4-20Kb each.

Hypothetically speaking, we'll have:
10 updates * 20Kb * 20 mirrors = 2000Kb per day. That's less then 2Mb 
per day. For June, we used 27.860.946 Kb bandwidth on belgerath (see 30x2000=60.000 Kb 
which is (60.000/27.860.946)*100=0.2% of the bandwidth.

Hope that relieves your concerns!

Jeroen Coumans

More information about the website mailing list