[Netarchivesuite-users] Messaging (openmq) problems: looping (?) messages, > 6000 per second in and out

[Peter Svanberg]

More info.

Looking with another program  (https://github.com/imsnif/bandwhich) you could see (see image) that the admin server is communicating with the harvest server instances. And all the harvest servers instance processes are busy, (four of them, 25 % each). But all but one of them has "Starts to listen ..." as last line in the log. The fourth has "Requested to check the validity of harvest channel 'WEEKLY'", which is strange. But there are earlier "broker error" messages in logs. Will look more.

-----
Peter Svanberg, Sweden

[cid:image001.png at 01DA125D.7BCAAEB0]


Från: NetarchiveSuite-users <netarchivesuite-users-bounces at ml.sbforge.org> För Peter Svanberg
Skickat: den 8 november 2023 14:05
Till: netarchivesuite-users at ml.sbforge.org
Ämne: [Netarchivesuite-users] Messaging (openmq) problems: looping (?) messages, > 6000 per second in and out

We have recurring problems (in July in both test and production and now just in test) with lots of messaging in the  queue for sending HarvesterRegistrationResponse.

Over 6000 messages per second (see below) in both directions, figures which are normally 0.

Is cured by stopping all harvesters, restarting openq on the admin server and then starting harvesters.

Anyone else have had this? Any hints on what it could be? Or if we should debug in some way before we restart?

Currently the openmq process gets 120 % CPU while the NAS processes get less than 1 % now and then. I suppose this indicates that something is looping internally in the openmq process. But why ...?

Displaying destination metrics where:

-----------------------------------------------
Destination Name               Destination Type
-----------------------------------------------
PLIKT_COMMON_HCHAN_VAL_RESP    Queue

On the broker specified by:

-------------------------
Host         Primary Port
-------------------------
localhost    7676

----------------------------------------------------------------------------------
Msgs/sec    Msg Bytes/sec       Msg Count         Total Msg Bytes (k)     Largest
In   Out     In      Out    Current  Peak  Avg  Current   Peak     Avg    Msg (k)
----------------------------------------------------------------------------------
0     0      0        0        9      16   10      6       12       8       < 1
5848  5847  4650296  4649825    10      16   10      7       12       8       < 1
5920  5921  4707657  4707976    10      16   10      7       12       8       < 1
5818  5817  4626266  4625949    12      16   10      9       12       8       < 1
5784  5784  4599286  4599763     8      16   10      6       12       8       < 1
5970  5969  4747221  4746267    15      16   10     11       12       8       < 1
5830  5831  4635982  4636777    10      16   10      7       12       8       < 1

[KB Logo]<https://www.kb.se/>

Peter Svanberg
Technical officer
Aquisitions and Metadata Department
Film, Games, Sheet Music and Web Unit

National Library of Sweden
PO Box 5039, SE-102 41 Stockholm
Visits: Karlavägen 96, Stockholm
+46 10-709 32 78
Peter.Svanberg at kb.se<mailto:Peter.Svanberg at kb.se>
www.kb.se<https://www.kb.se/>



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://ml.sbforge.org/pipermail/netarchivesuite-users/attachments/20231108/164f0e18/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 423315 bytes
Desc: image001.png
URL: <https://ml.sbforge.org/pipermail/netarchivesuite-users/attachments/20231108/164f0e18/attachment-0001.png>