[Netarchivesuite-curator] BnF NAS update for October

[peter.stirling at bnf.fr]

Hello all,

That's it! We started our 2015 broad crawl on September 22nd with 4.5 
million seed domains.

We need to be careful with the bandwidth: only ten threads are possible by 
crawler during the day, so we do not go over the 400 Mbits/s allocated for 
web archiving at the BnF. During the night, each crawler has 250 threads.

For the same attention to bandwidth, we also started the crawl with only 
five crawlers, and have added by groups of five to go up to 40 crawlers. 
In case of any problem, we can decrease the number of crawlers in the same 
way.

Each domain has a maximum budget of 2,300 URL, and each job has a maximum 
time of three days.

We hope to finish this crawl in two months with a total budget of 60 TB.

Best regards,
The BnF digital legal deposit team


Entrez dans l'Histoire des rois de France en participant à l'acquisition du bréviaire royal de Saint-Louis de Poissy Avant d'imprimer, pensez à l'environnement. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://ml.sbforge.org/pipermail/netarchivesuite-curator/attachments/20151002/33974eb8/attachment.html>