[Netarchivesuite-devel] # messages on HARVEST_MON is growing all the time
Søren Vejrup Carlsen
svc at kb.dk
Mon Feb 7 11:36:15 CET 2011
Hi all.
We have just rolled out 3.14.0 in the Netarkivet PROD, and immediately stumbled upon two major problems.
The consumers to the MONITOR queues cannot keep up with the producers of these queues.
After just a few days, we have the following broker status:
PROD_COMMON_ANY_HIGHPRIORITY_HACO Queue RUNNING 1 8 0 0 0.0
PROD_COMMON_ANY_LOWPRIORITY_HACO Queue RUNNING 0 16 0 0 0.0
PROD_COMMON_FRONTIERMON Queue RUNNING 29 2 1 1 1739.0
PROD_COMMON_HARVESTMON Queue RUNNING 30 2 28538 2 2376.3276
PROD_COMMON_INDEX_SERVER Queue RUNNING 30 1 0 0 0.0
PROD_COMMON_MONITOR Queue RUNNING 158 1 91543 1 937.5843
PROD_COMMON_THE_REPOS Queue RUNNING 47 1 0 0 0.0
PROD_COMMON_THE_SCHED Queue RUNNING 29 1 0 0 0.0
Notice, that there is 28538 waiting messages (from 30 producers) in the COMMON_HARVESTMON queue, and 91543 (from 158 producers) waiting message in the COMMON_MONITOR queue.
We have increased the crawlLoppWaitTime from the default 20 secs to 60 seconds, and the reregisterDelay from 1 minute to 5 minutes.
These increases were hoped to fix the problem with congestion in the MONITOR message-queues.
Have anyone have similar problems in their production environment?
Best Regards
Søren V. Carlsen, QA of netarchiveSuite
---------------------------------------------------------------------------
Søren Vejrup Carlsen, Department of Digital Preservation, Royal Library, Copenhagen, Denmark
tlf: (+45) 33 47 48 41
email: svc at kb.dk
----------------------------------------------------------------------------
Non omnia possumus omnes
--- Macrobius, Saturnalia, VI, 1, 35 -------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://ml.sbforge.org/pipermail/netarchivesuite-devel/attachments/20110207/b5cbff9d/attachment-0002.html>
More information about the Netarchivesuite-devel
mailing list