[Netarchivesuite-users] Netarchive Suite 3.21.0 MessageQueue and backup questions
Meelis Mihhailov
meelis at nlib.ee
Wed Apr 17 14:41:52 CEST 2013
Success :)
After killing applications and those harvesters that were not occupied
with jobs I saw in the MQ list nothing. Empty as it should be. So I
started up all the services that were killed and this is the MQ result:
Listing all the destinations on the broker specified by:
-------------------------
Host Primary Port
-------------------------
localhost 7676
------------------------------------------------------------------------------------------------------------------------------------
Name Type State
Producers Consumers Msgs
Total Wildcard Total Wildcard Count Remote UnAck Avg Size
------------------------------------------------------------------------------------------------------------------------------------
NLIB_A_ALL_BA_TOPIC Topic RUNNING 1
0 1 0 0 0 0 0.0
NLIB_A_ANY_BA Queue RUNNING 1
- 1 - 0 0 0 0.0
NLIB_A_THE_BAMON Queue RUNNING 2
- 1 - 0 0 0 0.0
NLIB_COMMON_ANY_HIGHPRIORITY_HACO Queue RUNNING 1
- 15 - 0 0 0 0.0
NLIB_COMMON_ANY_LOWPRIORITY_HACO Queue RUNNING 0
- 1 - 0 0 0 0.0
NLIB_COMMON_HARVESTER_STATUS_TOPIC Topic RUNNING 17
0 1 0 0 0 0 0.0
NLIB_COMMON_HARVESTMON Queue RUNNING 2
- 1 - 0 0 0 0.0
NLIB_COMMON_INDEX_SERVER Queue RUNNING 0
- 1 - 0 0 0 0.0
NLIB_COMMON_MONITOR Queue RUNNING 24
- 1 - 0 0 0 0.0
NLIB_COMMON_THE_REPOS Queue RUNNING 3
- 1 - 0 0 0 0.0
NLIB_COMMON_THE_SCHED Queue RUNNING 2
- 1 - 0 0 0 0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH10 Queue RUNNING 0
- 1 - 0 0 0 0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH11 Queue RUNNING 0
- 1 - 0 0 0 0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH12 Queue RUNNING 0
- 1 - 0 0 0 0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH14 Queue RUNNING 0
- 1 - 0 0 0 0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH15 Queue RUNNING 0
- 1 - 0 0 0 0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH16 Queue RUNNING 0
- 1 - 0 0 0 0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH17 Queue RUNNING 0
- 1 - 0 0 0 0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH19 Queue RUNNING 0
- 1 - 0 0 0 0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH1 Queue RUNNING 0
- 1 - 0 0 0 0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH20 Queue RUNNING 0
- 1 - 0 0 0 0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH4 Queue RUNNING 0
- 1 - 0 0 0 0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH5 Queue RUNNING 0
- 1 - 0 0 0 0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH6 Queue RUNNING 1
- 1 - 0 0 0 0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH7 Queue RUNNING 0
- 1 - 0 0 0 0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH8 Queue RUNNING 0
- 1 - 0 0 0 0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_HIGH9 Queue RUNNING 0
- 1 - 0 0 0 0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_HCS_LOW Queue RUNNING 0
- 1 - 0 0 0 0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_IS Queue RUNNING 0
- 1 - 0 0 0 0.0
NLIB_COMMON_THIS_REPOS_CLIENT_127_0_1_1_VP Queue RUNNING 0
- 1 - 0 0 0 0.0
mq.sys.dmq Queue RUNNING 0
- 0 - 0 0 0 0.0
Successfully listed destinations.
Everything seems to work and the job that was pending to start, launched
successfully :)
Thank you Bjarne and Søren.
Now I just have to wait untill these jobs that are currently fetching
pages finish so I can restart remaining harvester applications.
Thank you again :)
Meelis Mihhailov
----------------
National Library Of Estonia
meelis at nlib.ee
On 17.04.2013 15:11, Meelis Mihhailov wrote:
> Thank you Bjarne
>
> Yes, we are running on debian linux :)
> I'll try the steps and let you know the results.
>
> Meelis Mihhailov
> -----------------
> National Library Of Estonia
> meelis at nlib.ee
>
> On 17.04.2013 15:03, Bjarne Andersen wrote:
>> The BAMON queue is already full with 100.000 messages again (still
>> strange if you restarted with reset store parameter).
>> I think your installation died too much (both applications and MQ) to
>> come alive again without a restart of everything (maybe except
>> harvesters)
>>
>> I would recommend
>> 1. stop MQ
>> 2. stop all appliations / stop all applications except harvesters
>> 3. restart MQ with the "reset store option"
>> 4. check with the list queues that all queues are empty (actually no
>> queues should exist)
>> 5. start all applications stopped under (2)
>>
>> Individual applications can be killed / started with individual
>> scripts in the conf-folder of the installation. There is a startall.sh
>> and killall.sh script and inside them you can see that they only call
>> all the individual kill/start scripts
>> (I assume you are on some kind of linux OS)
>>
>> best
>> Bjarne
>> ________________________________________
>> Fra: netarchivesuite-users-bounces at ml.sbforge.org
>> [netarchivesuite-users-bounces at ml.sbforge.org] På vegne af Meelis
>> Mihhailov [meelis at nlib.ee]
>> Sendt: 17. april 2013 13:24
>> Til: netarchivesuite-users at ml.sbforge.org
>> Emne: Re: [Netarchivesuite-users] Netarchive Suite 3.21.0 MessageQueue
>> and backup questions
>>
>> Thank you Bjarne for the command.
>> After running it I can see the following result:
>>
>> Listing all the destinations on the broker specified by:
>>
>> -------------------------
>> Host Primary Port
>> -------------------------
>> localhost 7676
>>
>> ------------------------------------------------------------------------------------------------------
>>
>> Name Type State Producers Consumers
>> Msgs
>> Total Wildcard Total Wildcard
>> Count Remote UnAck Avg Size
>> ------------------------------------------------------------------------------------------------------
>>
>> NLIB_A_ALL_BA_TOPIC Topic RUNNING 0 0 1 0 0
>> 0 0 0.0
>> NLIB_A_ANY_BA Queue RUNNING 0 - 1 - 0
>> 0 0 0.0
>> NLIB_A_THE_BAMON Queue RUNNING 1 - 0 -
>> 100000 0 0 683.0
>> mq.sys.dmq Queue RUNNING 0 - 0 - 0
>> 0 0 0.0
>>
>> Successfully listed destinations.
>>
>> Our setup is done by QuickStart manual and everything is running in the
>> same server and started/stopped with a start/stop script created after
>> the install. How can I restart individual applications in my
>> installation? In NAS administration interface I can see under
>> "systemstate" that there are only "remove application" buttons and no
>> "restart" or similar options.
>>
>> Also when I restart IMQ I can see that bitarchive is the only
>> application that actually sees the MQ working and connects to it. All
>> other applications give connection errors.
>>
>>
>>
>> Meelis Mihhailov
>> ----------------
>> National Library Of Estonia
>> meelis at nlib.ee
>>
>> On 17.04.2013 14:10, Bjarne Andersen wrote:
>>> Ah - yes. The broker is best to restart when all other applications
>>> is not running - otherwise they seem to hold all the messages they
>>> couldn't deliver. Everything but harvesters (running heritrix jobs)
>>> should be fine to restart - you could leave running harvesters and
>>> restart everything else.
>>> BUT I dont actually understand how more than 37.000 messages could be
>>> waiting for the BA_MON queue - do you have many Bitarchive Machines
>>> running ?
>>> There is a command for listing consumers, producers and number of
>>> messages on each queue
>>> /opt2/sun/mq/bin/imqcmd list dst -u admin -passfile
>>> /home/prod/.imq_passfile
>>> (imq_passfile is just a textfile with the password:
>>> imq.imqcmd.password=admin) (admin is the default password)
>>>
>>> Here you could also inspect how many aplications is generating
>>> messages for each queue and how many applications is "eating"
>>> messages from each queue. for the BA_MON queue there should be
>>> exactly one consumer (the BitarchiveMonitor Application) - if that is
>>> not running properly the number of messages on that queue will grow
>>> quite rapidly (but not to 37.000 in few seconds)
>>>
>>>
>>> best
>>> Bjarne
>> _______________________________________________
>> NetarchiveSuite-users mailing list
>> NetarchiveSuite-users at ml.sbforge.org
>> http://ml.sbforge.org/mailman/listinfo/netarchivesuite-users
>>
>> _______________________________________________
>> NetarchiveSuite-users mailing list
>> NetarchiveSuite-users at ml.sbforge.org
>> http://ml.sbforge.org/mailman/listinfo/netarchivesuite-users
>>
> _______________________________________________
> NetarchiveSuite-users mailing list
> NetarchiveSuite-users at ml.sbforge.org
> http://ml.sbforge.org/mailman/listinfo/netarchivesuite-users
More information about the NetarchiveSuite-users
mailing list