[Netarchivesuite-devel] # messages on HARVEST_MON is growing all the time

aponb at gmx.at aponb at gmx.at
Mon Feb 7 13:42:46 CET 2011


I had this Problem in Version 3.8. Then I used reregisterDelay setting 
with 10 minutes and I had no problems anymore in Version 3.10 and 3.12. 
Can't tell about Version 3.14 as I am starting testing it now!
>
> We have just rolled out 3.14.0 in the Netarkivet PROD, and immediately 
> stumbled upon two major problems.
>
> The consumers to the MONITOR queues cannot keep up with the producers 
> of these queues.
>
> After just a few days, we have the following broker status:
>
> PROD_COMMON_ANY_HIGHPRIORITY_HACO                              Queue  
> RUNNING  1          8          0      0      0.0
>
> PROD_COMMON_ANY_LOWPRIORITY_HACO                               Queue  
> RUNNING  0          16         0      0      0.0
>
> PROD_COMMON_FRONTIERMON                                        Queue  
> RUNNING  29         2          1      1      1739.0
>
> PROD_COMMON_HARVESTMON                                         Queue  
> RUNNING  30         2          28538  2      2376.3276
>
> PROD_COMMON_INDEX_SERVER                                       Queue  
> RUNNING  30         1          0      0      0.0
>
> PROD_COMMON_MONITOR                                            Queue  
> RUNNING  158        1          91543  1      937.5843
>
> PROD_COMMON_THE_REPOS                                          Queue  
> RUNNING  47         1          0      0      0.0
>
> PROD_COMMON_THE_SCHED                                          Queue  
> RUNNING  29         1          0      0      0.0
>
> Notice, that there is 28538 waiting messages (from 30 producers) in 
> the COMMON_HARVESTMON queue, and 91543 (from 158 producers) waiting 
> message in the COMMON_MONITOR queue.
>
> We have increased the crawlLoppWaitTime from the default 20 secs to 60 
> seconds, and the reregisterDelay from 1 minute to 5 minutes.
>
> These increases were hoped to fix the problem with congestion in the 
> MONITOR message-queues.
>
> Have anyone have similar problems in their production environment?
>
> Best Regards
>
> Søren V. Carlsen, QA of netarchiveSuite
>
> ---------------------------------------------------------------------------
>
> Søren Vejrup Carlsen, Department of Digital Preservation, Royal 
> Library, Copenhagen, Denmark
>
> tlf: (+45) 33 47 48 41
>
> email: svc at kb.dk
>
> ----------------------------------------------------------------------------
>
> Non omnia possumus omnes
>
> --- Macrobius, Saturnalia, VI, 1, 35 -------
>
>
> _______________________________________________
> Netarchivesuite-devel mailing list
> Netarchivesuite-devel at lists.gforge.statsbiblioteket.dk
> https://lists.gforge.statsbiblioteket.dk/mailman/listinfo/netarchivesuite-devel
>    

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://ml.sbforge.org/pipermail/netarchivesuite-devel/attachments/20110207/8416291f/attachment-0002.html>


More information about the Netarchivesuite-devel mailing list