[Netarchivesuite-curator] BnF NAS update May/June

[peter.stirling at bnf.fr]

Hello all,

Here is our update for May/June :

1) We launched the "annual" part of our focussed crawl on 14th May - this 
is the biggest section of the sites selected by our librarians, with over 
4000 configurations. We once again experienced some problems due to the 
fact that we cannot control in advance the number of jobs created by NAS, 
as we had 42 jobs for these HarvestDefinitions, in addition to the jobs 
for other HarvestDefinitions that are active at the same time (elections, 
daily, weekly and monthly crawls). 
- To overcome the problem of crawler capacity Bert was able to add 20 
extra crawlers using our virtual server system. This allowed us to 
continue with all the different crawls, but can only be a short-term 
solution as it puts a lot of strain on our pool of resources.
- To find a long-term solution to this problem, particularly in view of 
our next broad crawl which will be performed in the autumn, we aim to work 
on the way NAS creates jobs to give us more control over the process, so 
we can plan our crawls to make the most efficient use of our resources. 
Nicolas should start working on this in June.

2) We have also started testing how to collect password-protected 
resources using the "Credential Store" settings of Heritrix. Two different 
options are possible, for sites that use a login form ("HTMLLoginForm") 
and those where the server controls the connection directly 
("HTTPBasicAccessAuthentication"). The first results are positive, the 
logs show that robot is able to pass the login page and collect files. As 
these test crawls are not indexed we haven't yet been able to test access 
in the Wayback.

If you want any more details please don't hesitate to ask.

Best regards,
The BnF digital legal deposit team


Journée portes ouvertes le jeudi 31 mai - François-Mitterrand  dans le cadre de la  Semaine du Nouveau Quartier Latin .   
Visites gratuites des salles de lecture et des expositions.  
Cette journée se terminera avec un  concert du Brass Band du Conservatoire Municipal du 13 e  arrondissement  qui aura lieu à 18h30 dans le Grand auditorium. Avant d'imprimer, pensez à l'environnement. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://ml.sbforge.org/pipermail/netarchivesuite-curator/attachments/20120531/bfd437ab/attachment.html>