[Netarchivesuite-curator] BnF NAS update for October

[Pérez Morillo, Mar]

Thank you, Peter!
Very interesting and helpful for us all this information, as we are planning to launch our first domain crawl by the beginning of next year.
Good luck and we wish you all the success with it!
Best,

Mar Pérez Morillo
Jefe del Área de Gestión del Depósito de las Publicaciones en Línea
Dirección de Biblioteca Digital y Sistemas de Información
Tfno.: 91 516 89 92
Biblioteca Nacional de España

De: Netarchivesuite-curator [mailto:netarchivesuite-curator-bounces at ml.sbforge.org] En nombre de peter.stirling at bnf.fr
Enviado el: viernes, 02 de octubre de 2015 14:30
Para: netarchivesuite-curator at ml.sbforge.org
Asunto: [Netarchivesuite-curator] BnF NAS update for October

Hello all,

That's it! We started our 2015 broad crawl on September 22nd with 4.5 million seed domains.

We need to be careful with the bandwidth: only ten threads are possible by crawler during the day, so we do not go over the 400 Mbits/s allocated for web archiving at the BnF. During the night, each crawler has 250 threads.

For the same attention to bandwidth, we also started the crawl with only five crawlers, and have added by groups of five to go up to 40 crawlers. In case of any problem, we can decrease the number of crawlers in the same way.

Each domain has a maximum budget of 2,300 URL, and each job has a maximum time of three days.

We hope to finish this crawl in two months with a total budget of 60 TB.

Best regards,
The BnF digital legal deposit team
________________________________

Entrez dans l'Histoire des rois de France en participant ? l'acquisition du br?viaire royal de Saint-Louis de Poissy<http://www.bnf.fr/fr/acces_dedies/mecenat_partenariat/s.mecenat_saint_louis.html>

Avant d'imprimer, pensez ? l'environnement.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://ml.sbforge.org/pipermail/netarchivesuite-curator/attachments/20151002/1a8da790/attachment.html>