[Netarchivesuite-curator] BnF NAS update for September
peter.stirling at bnf.fr
peter.stirling at bnf.fr
Fri Sep 11 15:07:07 CEST 2015
Hello all,
First of all, we're pleased to announce that Marie Chouleur arrived on the
1st September to take up the post of head of digital legal deposit at the
BnF.
During July and August, we have performed technical tests and started a
trial run for the broad crawl:
- we have tried to include IDNs: it was necessary to rewrite some of them
in correct UTF8 syntax, but this did not work with NAS and Heritrix. So we
will have to wait for Heritrix 3 to crawl these specific domains.
- the new storage array was delivered and different kinds of
configurations were tried before finding the right one. There were some
communication problems between the crawlers and the array.
- we changed the operating system of the servers from CentOS 5 to CentOS
6, which turned out to be a lot of work. At first, we put CentOS 6 on the
crawlers but access to the indexer was much less powerful than under the
old system. The consequence was that each job almost stopped working after
two or three hours. We tried several configurations before we eventually
moved the index to an external nfs server on the storage array.
Right now, our engineers have solved these problems and are doing some
final tests. We hope to be able to start the real crawl mid-september.
Best regards,
The BnF digital legal deposit team
Entrez dans l'Histoire des rois de France en participant à l'acquisition du bréviaire royal de Saint-Louis de Poissy Avant d'imprimer, pensez à l'environnement.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://ml.sbforge.org/pipermail/netarchivesuite-curator/attachments/20150911/a4f2cf82/attachment.html>
More information about the Netarchivesuite-curator
mailing list