[Netarchivesuite-users] Problem wih IndexAggregator

[Colin Samuel Rosenthal]

Hi!


That's hard to diagnose exactly because the unix sort command gives return code 2 for all errors. What you can try to do is run the same sort command by hand, something like


sort <all the files in batchOutputDir>  -o  aggregator_tempdir/temp.index -T aggregator_tempdir


then you should be able to see the actual cause of the error. My guess is that you have one of

i) a drive that is too full to write to

ii) an error in your write permissions

iii) a problem with too many files to be sorted at once


cheers,

Colin


--
Colin Rosenthal PhD
Senior IT Consultant
Royal Danish Library (Aarhus)


________________________________
From: NetarchiveSuite-users <netarchivesuite-users-bounces at ml.sbforge.org> on behalf of García Arratia, Juan Carlos <juancarlos.garcia at bne.es>
Sent: Friday, December 13, 2019 11:24 AM
To: netarchivesuite-users at ml.sbforge.org
Cc: Serrano Benavente, Nuria; Archivoweb
Subject: [Netarchivesuite-users] Problem wih IndexAggregator


For some time we have problems with the Application Aggregator. Therefore we do not have the updated Openwayback indexes and the collections made do not appear.

This message appears in the logs:



“[AggregatorThread] ERROR dk.netarkivet.wayback.aggregator.IndexAggregator - Failed to sort index files, sort exited with return code 2”



The configuration we have is as follows:

·         settings_AggregatorApplication.xml:

<aggregator>

  <indexFileInputDir>batchOutputDir</indexFileInputDir>

   <indexFileOutputDir>indexDir</indexFileOutputDir>

   <tempAggregatorDir>aggregator_tempdir</tempAggregatorDir>

   <aggregationInterval>2400000</aggregationInterval>

   <maxIntermediateIndexFileSize>2048576</maxIntermediateIndexFileSize>

   <maxMainIndexFileSize>304857600</maxMainIndexFileSize>

</aggregator>



·         File size:

o    -rw-rw-r--  1 netarchive netarchive 79356293120 dic 11 22:58 wayback.index

o    -rw-rw-r--  1 netarchive netarchive  1279525993 dic 13 06:55 wayback_intermediate.index



·         Server:

o    Java version:

§   openjdk version "1.8.0_131"

§   OpenJDK Runtime Environment (build 1.8.0_131-b12)

§   OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode)

o    Memory:

§   MemTotal:       131903488 kB

§   MemFree:        48043984 kB

§   MemAvailable:   128086540 kB

o    File system:

§   76 GB usados

§   75 GB disponibles



Regards



Un saludo

-------------------------------------------------------------------------

Juan Carlos García Arratia

Jefe de Servicio de Tecnologías de la Información

Biblioteca Nacional de España

Paseo de Recoletos 20-22

28071    Madrid

Tlfno:    91 580 78 19

Fax:       91 516 89 07

e-mail:   jcgarcia at bne.es

------------------------------------------------------------------------



________________________________
Este mensaje y cualquier fichero adjunto están dirigidos únicamente a sus destinatarios y contiene información confidencial. Si usted ha recibido este correo electrónico por error, le informamos que no puede realizar ninguna revisión, alteración, impresión, copia, transmisión, difusión ni utilización alguna de este mensaje ni de cualquier fichero adjunto que pudiese contener. La realización de cualquiera de los actos indicados está expresamente prohibida por las Normas que regulan estas materias. Por todo ello se solicita que, en caso de existir error en la recepción de este mensaje, se lo notifique al remitente respondiendo a este e-mail y elimine el mensaje y su contenido inmediatamente. La Biblioteca Nacional de España se reserva las acciones legales que le correspondan en el caso de que se infrinja lo indicado anteriormente.
________________________________
The information in this e-mail and any attachments is confidential and it is intended for the addressee only. If you have received this e-mail in error, you are notified that any revision, amendment, print, copy, disclosure, distribution or use of the contents is unauthorized. Carrying out any of the above actions, is expressly banned by rules governing this matter. Hence we request that if you are not the intended recipient, please notify the sender answering this e-mail, and delete the message and any attachments. The National Library of Spain reserves itself the right to take the appropriate legal actions in the event of the above mentioned matter is being infringed.
________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://ml.sbforge.org/pipermail/netarchivesuite-users/attachments/20191213/e17eae6e/attachment.html>