[Netarchivesuite-users] Netarchive Suite 3.21.0 MessageQueue and backup questions

Meelis Mihhailov meelis at nlib.ee
Wed Apr 17 14:41:52 CEST 2013


Success :)
After killing applications and those harvesters that were not occupied 
with jobs I saw in the MQ list nothing. Empty as it should be. So I 
started up all the services that were killed and this is the MQ result:


Listing all the destinations on the broker specified by:

-------------------------
Host         Primary Port
-------------------------
localhost    7676

------------------------------------------------------------------------------------------------------------------------------------
                        Name                         Type    State 
Producers        Consumers                  Msgs
 
Total  Wildcard  Total  Wildcard  Count  Remote  UnAck  Avg Size
------------------------------------------------------------------------------------------------------------------------------------
NLIB_A_ALL_BA_TOPIC                                 Topic  RUNNING  1 
    0         1      0         0      0       0      0.0
NLIB_A_ANY_BA                                       Queue  RUNNING  1 
    -         1      -         0      0       0      0.0
NLIB_A_THE_BAMON                                    Queue  RUNNING  2 
    -         1      -         0      0       0      0.0
NLIB_COMMON_ANY_HIGHPRIORITY_HACO                   Queue  RUNNING  1 
    -         15     -         0      0       0      0.0
NLIB_COMMON_ANY_LOWPRIORITY_HACO                    Queue  RUNNING  0 
    -         1      -         0      0       0      0.0
NLIB_COMMON_HARVESTER_STATUS_TOPIC                  Topic  RUNNING  17 
    0         1      0         0      0       0      0.0
NLIB_COMMON_HARVESTMON                              Queue  RUNNING  2 
    -         1      -         0      0       0      0.0
NLIB_COMMON_INDEX_SERVER                            Queue  RUNNING  0 
    -         1      -         0      0       0      0.0
NLIB_COMMON_MONITOR                                 Queue  RUNNING  24 
    -         1      -         0      0       0      0.0
NLIB_COMMON_THE_REPOS                               Queue  RUNNING  3 
    -         1      -         0      0       0      0.0
NLIB_COMMON_THE_SCHED                               Queue  RUNNING  2 
    -         1      -         0      0       0      0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH10  Queue  RUNNING  0 
    -         1      -         0      0       0      0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH11  Queue  RUNNING  0 
    -         1      -         0      0       0      0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH12  Queue  RUNNING  0 
    -         1      -         0      0       0      0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH14  Queue  RUNNING  0 
    -         1      -         0      0       0      0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH15  Queue  RUNNING  0 
    -         1      -         0      0       0      0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH16  Queue  RUNNING  0 
    -         1      -         0      0       0      0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH17  Queue  RUNNING  0 
    -         1      -         0      0       0      0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH19  Queue  RUNNING  0 
    -         1      -         0      0       0      0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH1   Queue  RUNNING  0 
    -         1      -         0      0       0      0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH20  Queue  RUNNING  0 
    -         1      -         0      0       0      0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH4   Queue  RUNNING  0 
    -         1      -         0      0       0      0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH5   Queue  RUNNING  0 
    -         1      -         0      0       0      0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH6   Queue  RUNNING  1 
    -         1      -         0      0       0      0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH7   Queue  RUNNING  0 
    -         1      -         0      0       0      0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH8   Queue  RUNNING  0 
    -         1      -         0      0       0      0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH9   Queue  RUNNING  0 
    -         1      -         0      0       0      0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_LOW     Queue  RUNNING  0 
    -         1      -         0      0       0      0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_IS          Queue  RUNNING  0 
    -         1      -         0      0       0      0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_VP          Queue  RUNNING  0 
    -         1      -         0      0       0      0.0
mq.sys.dmq                                          Queue  RUNNING  0 
    -         0      -         0      0       0      0.0

Successfully listed destinations.

Everything seems to work and the job that was pending to start, launched 
successfully :)

Thank you Bjarne and Søren.

Now I just have to wait untill these jobs that are currently fetching 
pages finish so I can restart remaining harvester applications.

Thank you again :)

Meelis Mihhailov
----------------
National Library Of Estonia
meelis at nlib.ee


On 17.04.2013 15:11, Meelis Mihhailov wrote:
> Thank you Bjarne
>
> Yes, we are running on debian linux :)
> I'll try the steps and let you know the results.
>
> Meelis Mihhailov
> -----------------
> National Library Of Estonia
> meelis at nlib.ee
>
> On 17.04.2013 15:03, Bjarne Andersen wrote:
>> The BAMON queue is already full with 100.000 messages again (still
>> strange if you restarted with reset store parameter).
>> I think your installation died too much (both applications and MQ) to
>> come alive again without a restart of everything (maybe except
>> harvesters)
>>
>> I would recommend
>> 1. stop MQ
>> 2. stop all appliations / stop all applications except harvesters
>> 3. restart MQ with the "reset store option"
>> 4. check with the list queues that all queues are empty (actually no
>> queues should exist)
>> 5. start all applications stopped under (2)
>>
>> Individual applications can be killed / started with individual
>> scripts in the conf-folder of the installation. There is a startall.sh
>> and killall.sh script and inside them you can see that they only call
>> all the individual kill/start scripts
>> (I assume you are on some kind of linux OS)
>>
>> best
>> Bjarne
>> ________________________________________
>> Fra: netarchivesuite-users-bounces at ml.sbforge.org
>> [netarchivesuite-users-bounces at ml.sbforge.org] På vegne af Meelis
>> Mihhailov [meelis at nlib.ee]
>> Sendt: 17. april 2013 13:24
>> Til: netarchivesuite-users at ml.sbforge.org
>> Emne: Re: [Netarchivesuite-users] Netarchive Suite 3.21.0 MessageQueue
>> and      backup questions
>>
>> Thank you Bjarne for the command.
>> After running it I can see the following result:
>>
>> Listing all the destinations on the broker specified by:
>>
>> -------------------------
>> Host         Primary Port
>> -------------------------
>> localhost    7676
>>
>> ------------------------------------------------------------------------------------------------------
>>
>>          Name          Type    State      Producers        Consumers
>>                Msgs
>>                                        Total  Wildcard  Total  Wildcard
>> Count   Remote  UnAck  Avg Size
>> ------------------------------------------------------------------------------------------------------
>>
>> NLIB_A_ALL_BA_TOPIC  Topic  RUNNING  0      0         1      0         0
>>         0       0      0.0
>> NLIB_A_ANY_BA        Queue  RUNNING  0      -         1      -         0
>>         0       0      0.0
>> NLIB_A_THE_BAMON     Queue  RUNNING  1      -         0      -
>> 100000  0       0      683.0
>> mq.sys.dmq           Queue  RUNNING  0      -         0      -         0
>>         0       0      0.0
>>
>> Successfully listed destinations.
>>
>> Our setup is done by QuickStart manual and everything is running in the
>> same server and started/stopped with a start/stop script created after
>> the install. How can I restart individual applications in my
>> installation? In NAS administration interface I can see under
>> "systemstate" that there are only "remove application" buttons and no
>> "restart" or similar options.
>>
>> Also when I restart IMQ I can see that bitarchive is the only
>> application that actually sees the MQ working and connects to it. All
>> other applications give connection errors.
>>
>>
>>
>> Meelis Mihhailov
>> ----------------
>> National Library Of Estonia
>> meelis at nlib.ee
>>
>> On 17.04.2013 14:10, Bjarne Andersen wrote:
>>> Ah - yes. The broker is best to restart when all other applications
>>> is not running - otherwise they seem to hold all the messages they
>>> couldn't deliver. Everything but harvesters (running heritrix jobs)
>>> should be fine to restart - you could leave running harvesters and
>>> restart everything else.
>>> BUT I dont actually understand how more than 37.000 messages could be
>>> waiting for the BA_MON queue - do you have many Bitarchive Machines
>>> running ?
>>> There is a command for listing consumers, producers and number of
>>> messages on each queue
>>> /opt2/sun/mq/bin/imqcmd list dst -u admin -passfile
>>> /home/prod/.imq_passfile
>>> (imq_passfile is just a textfile with the password:
>>> imq.imqcmd.password=admin) (admin is the default password)
>>>
>>> Here you could also inspect how many aplications is generating
>>> messages for each queue and how many applications is "eating"
>>> messages from each queue. for the BA_MON queue there should be
>>> exactly one consumer (the BitarchiveMonitor Application) - if that is
>>> not running properly the number of messages on that queue will grow
>>> quite rapidly (but not to 37.000 in few seconds)
>>>
>>>
>>> best
>>> Bjarne
>> _______________________________________________
>> NetarchiveSuite-users mailing list
>> NetarchiveSuite-users at ml.sbforge.org
>> http://ml.sbforge.org/mailman/listinfo/netarchivesuite-users
>>
>> _______________________________________________
>> NetarchiveSuite-users mailing list
>> NetarchiveSuite-users at ml.sbforge.org
>> http://ml.sbforge.org/mailman/listinfo/netarchivesuite-users
>>
> _______________________________________________
> NetarchiveSuite-users mailing list
> NetarchiveSuite-users at ml.sbforge.org
> http://ml.sbforge.org/mailman/listinfo/netarchivesuite-users


More information about the NetarchiveSuite-users mailing list