[Netarchivesuite-users] NAS - Problems with Broker

[PE-BDH-002]

Hi!
I'm Miguel Soleto, from the National Library of Spain. I will try to explain the problems We are having with the Broker.
First, this is the architecture of our installation (each module on a different machine). NAS Version: 5.4.2:

·         4 BitArchives.

·         1 Aggregator.

·         2 Indexers.

·         3 OpenWayback.

·         1 Broker.

·         1 Postgresql database (1 Master and 1 Slave).

·         1 IndexServerApplication and 1 ViewerProxyApplication.

·         1 GUIApplication, 1 ArcRepositoryApplication, 1 BitarchiveMonitorApplication and 1 HarvestJobManagerApplication.

·         110 Spiders, Working 80, the other 30 doing nothing.

Here are the issues We are experiencing:
This Sunday (2nd May), the Broker seemed to work, but no Jobs were running. With a ps command on the machine, We saw the Java process of the Broker, but We had lots of "Connection refused" on the logs of the GUIApplication. After that, We killed the Java process (Broker machine) and a few minutes later, it seemed to be OK again.
This Tuesday, the Java process of the Broker died with no apparent reason. In that moment, We realized that the file "config.properties" were overwritten, and every line on that file was commented, except for the last one:
#Last Update:
#Tue May 04 09:22:52 CEST 2021
imq.instanceconfig.version=300

After that, We re-launched the Java process again, and everything were OK.

Yesterday, We had the same problem: the Broker died with apparently no reason.

On the GUIApplication logs, We have the error code "C4056". I have searched for that code, and this is what Oracle says: Cause A Message Queue client received a GOOD_BYE message from broker.

So, We are worried about the performance of the Broker, beacuse when It dies, all the platform stops working. We had a BASH script on the Broker machine that looks for the process: if It isn't running, It send us an e-mail.

Has anyone had a problem like this? Any idea about what is happening in our NAS installation? May be a memory problem, or just too much conections for the Broker? Should I send our "default.properties" file to you? Any guide or documentation about the config.properties or default.properties parameters?
The Broker process is configured with this parameters: -Xms24g -Xmx24g -Xss192m -XX:MaxGCPauseMillis=5000.

Thank you all!
Best regards,
Miguel.
________________________________
Este mensaje y cualquier fichero adjunto están dirigidos únicamente a sus destinatarios y contiene información confidencial. Si usted ha recibido este correo electrónico por error, le informamos que no puede realizar ninguna revisión, alteración, impresión, copia, transmisión, difusión ni utilización alguna de este mensaje ni de cualquier fichero adjunto que pudiese contener. La realización de cualquiera de los actos indicados está expresamente prohibida por las Normas que regulan estas materias. Por todo ello se solicita que, en caso de existir error en la recepción de este mensaje, se lo notifique al remitente respondiendo a este e-mail y elimine el mensaje y su contenido inmediatamente. La Biblioteca Nacional de España se reserva las acciones legales que le correspondan en el caso de que se infrinja lo indicado anteriormente.
________________________________
The information in this e-mail and any attachments is confidential and it is intended for the addressee only. If you have received this e-mail in error, you are notified that any revision, amendment, print, copy, disclosure, distribution or use of the contents is unauthorized. Carrying out any of the above actions, is expressly banned by rules governing this matter. Hence we request that if you are not the intended recipient, please notify the sender answering this e-mail, and delete the message and any attachments. The National Library of Spain reserves itself the right to take the appropriate legal actions in the event of the above mentioned matter is being infringed.
________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://ml.sbforge.org/pipermail/netarchivesuite-users/attachments/20210506/2d2a2311/attachment.html>