[Netarchivesuite-users] Netarchive Suite 3.21.0 MessageQueue and backup questions

Wed Apr 17 13:10:36 CEST 2013

Ah - yes. The broker is best to restart when all other applications is not running - otherwise they seem to hold all the messages they couldn't deliver. Everything but harvesters (running heritrix jobs) should be fine to restart - you could leave running harvesters and restart everything else.
BUT I dont actually understand how more than 37.000 messages could be waiting for the BA_MON queue - do you have many Bitarchive Machines running ?
There is a command for listing consumers, producers and number of messages on each queue
/opt2/sun/mq/bin/imqcmd list dst -u admin -passfile /home/prod/.imq_passfile
(imq_passfile is just a textfile with the password: imq.imqcmd.password=admin) (admin is the default password)

Here you could also inspect how many aplications is generating messages for each queue and how many applications is "eating" messages from each queue. for the BA_MON queue there should be exactly one consumer (the BitarchiveMonitor Application) - if that is not running properly the number of messages on that queue will grow quite rapidly (but not to 37.000 in few seconds)

best
Bjarne
________________________________________
Fra: netarchivesuite-users-bounces at ml.sbforge.org [netarchivesuite-users-bounces at ml.sbforge.org] På vegne af Meelis Mihhailov [meelis at nlib.ee]
Sendt: 17. april 2013 12:30
Til: netarchivesuite-users at ml.sbforge.org
Emne: Re: [Netarchivesuite-users] Netarchive Suite 3.21.0 MessageQueue and      backup questions

Bjarne:

Do you mean "imqbrokerd -reset store"?
I used the following : /opt/MessageQueue4_5/mq/bin/imqbrokerd -reset store &
but it does not work as we can see below. It seems that NAS or something
else is filling up the queue quite fast...

Søren:

Changed to the values you provided and this is the result in log:

----------------------------------------------------------------
[17/Apr/2013:13:25:18 EEST] [B1065]: Accepting:
guest at 127.0.0.1:41812->jms:43020. Count: service=1 broker=1
[17/Apr/2013:13:25:18 EEST] [B1132]: Auto-creating destination
NLIB_A_ANY_BA [Queue]
[17/Apr/2013:13:25:18 EEST] [B1132]: Auto-creating destination
NLIB_A_ALL_BA_TOPIC [Topic]
[17/Apr/2013:13:25:18 EEST] [B1132]: Auto-creating destination
NLIB_A_THE_BAMON [Queue]
[17/Apr/2013:13:25:18 EEST] imq.persist.file.minimizeWrites=false

[17/Apr/2013:13:25:41 EEST] [B1089]: In low memory condition, Broker is
attempting to free up resources
[17/Apr/2013:13:25:41 EEST] [B1088]: Entering Memory State YELLOW from
previous state GREEN  - allocated memory is 430759K, 83% of total memory
used
[17/Apr/2013:13:25:42 EEST] [B1088]: Entering Memory State GREEN  from
previous state YELLOW - allocated memory is 412956K, 79% of total memory
used
[17/Apr/2013:13:25:43 EEST] [B1089]: In low memory condition, Broker is
attempting to free up resources
[17/Apr/2013:13:25:43 EEST] [B1088]: Entering Memory State YELLOW from
previous state GREEN  - allocated memory is 446344K, 86% of total memory
used
[17/Apr/2013:13:25:44 EEST] [B1089]: In low memory condition, Broker is
attempting to free up resources
[17/Apr/2013:13:25:44 EEST] [B1088]: Entering Memory State ORANGE from
previous state YELLOW - allocated memory is 469519K, 90% of total memory
used
[17/Apr/2013:13:25:44 EEST] [B1088]: Entering Memory State YELLOW from
previous state ORANGE - allocated memory is 463464K, 89% of total memory
used
[17/Apr/2013:13:25:45 EEST] [B1089]: In low memory condition, Broker is
attempting to free up resources
[17/Apr/2013:13:25:45 EEST] [B1088]: Entering Memory State ORANGE from
previous state YELLOW - allocated memory is 467659K, 90% of total memory
used
[17/Apr/2013:13:25:46 EEST] [B1089]: In low memory condition, Broker is
attempting to free up resources
[17/Apr/2013:13:25:46 EEST] [B1088]: Entering Memory State  RED  from
previous state ORANGE - allocated memory is 507496K, 98% of total memory
used
[17/Apr/2013:13:26:16 EEST] [B1088]: Entering Memory State GREEN  from
previous state  RED  - allocated memory is 385779K, 74% of total memory used
[17/Apr/2013:13:26:21 EEST] [B1089]: In low memory condition, Broker is
attempting to free up resources
[17/Apr/2013:13:26:21 EEST] [B1088]: Entering Memory State YELLOW from
previous state GREEN  - allocated memory is 421926K, 81% of total memory
used
[17/Apr/2013:13:26:23 EEST] [B1089]: In low memory condition, Broker is
attempting to free up resources
[17/Apr/2013:13:26:23 EEST] [B1088]: Entering Memory State ORANGE from
previous state YELLOW - allocated memory is 471546K, 91% of total memory
used
[17/Apr/2013:13:26:26 EEST] [B1089]: In low memory condition, Broker is
attempting to free up resources
[17/Apr/2013:13:26:26 EEST] [B1088]: Entering Memory State  RED  from
previous state ORANGE - allocated memory is 507642K, 98% of total memory
used
[17/Apr/2013:13:26:46 EEST] [B1088]: Entering Memory State ORANGE from
previous state  RED  - allocated memory is 475077K, 91% of total memory used
[17/Apr/2013:13:26:48 EEST] [B1088]: Entering Memory State YELLOW from
previous state ORANGE - allocated memory is 465644K, 89% of total memory
used
[17/Apr/2013:13:26:48 EEST] [B1089]: In low memory condition, Broker is
attempting to free up resources
[17/Apr/2013:13:26:48 EEST] [B1088]: Entering Memory State ORANGE from
previous state YELLOW - allocated memory is 471659K, 91% of total memory
used
[17/Apr/2013:13:26:49 EEST] [B1089]: In low memory condition, Broker is
attempting to free up resources
[17/Apr/2013:13:26:49 EEST] [B1088]: Entering Memory State  RED  from
previous state ORANGE - allocated memory is 507927K, 98% of total memory
used
[17/Apr/2013:13:26:59 EEST] [B1088]: Entering Memory State ORANGE from
previous state  RED  - allocated memory is 507392K, 97% of total memory used
[17/Apr/2013:13:26:59 EEST] [B1089]: In low memory condition, Broker is
attempting to free up resources
[17/Apr/2013:13:26:59 EEST] [B1088]: Entering Memory State  RED  from
previous state ORANGE - allocated memory is 507508K, 98% of total memory
used
[17/Apr/2013:13:27:09 EEST] [B1088]: Entering Memory State ORANGE from
previous state  RED  - allocated memory is 492675K, 95% of total memory used
[17/Apr/2013:13:27:12 EEST] WARNING [B2011]: Storing of JMS message from
IMQConn[AUTHENTICATED,guest at 127.0.0.1:41812,null] failed:
com.sun.messaging.jmq.jmsserver.util.BrokerException: [B4120]: Can not
add message 1145015-127.0.1.1(96:e4:57:50:b6:79)-41812-1366194432079 to
destination NLIB_A_THE_BAMON [Queue]. The destination message count
limit (maxNumMsgs) of 100000 has been reached.
[17/Apr/2013:13:27:12 EEST] WARNING [B2011]: Storing of JMS message from
IMQConn[AUTHENTICATED,guest at 127.0.0.1:41812,null] failed:
com.sun.messaging.jmq.jmsserver.util.BrokerException: [B4120]: Can not
add message 1145015-127.0.1.1(96:e4:57:50:b6:79)-41812-1366194432079 to
destination NLIB_A_THE_BAMON [Queue]. The destination message count
limit (maxNumMsgs) of 100000 has been reached.

----------------------------------------------------------------

Meelis Mihhailov
------------------------
National Library Of Estonia
meelis at nlib.ee

On 17.04.2013 13:04, Bjarne Andersen wrote:
> It looks like you are not emptying the queues when you restart the IMQ-broker (looks like the server tries to re-load messages after the restart)
> Its always recommended to empty all queues when you restart (this is a parameter for the startup-script and should be documented in the NAS manual)
>
> The re-loading of old messages looks like killing the broker again.
>
> If you did not configure the MQ to allow more heap space you should also do that - I think we are running with 512Mb
>
> best
> Bjarne Andersen
> ________________________________________
> Fra: netarchivesuite-users-bounces at ml.sbforge.org [netarchivesuite-users-bounces at ml.sbforge.org] På vegne af Meelis Mihhailov [meelis at nlib.ee]
> Sendt: 17. april 2013 11:56
> Til: netarchivesuite-users at ml.sbforge.org
> Emne: Re: [Netarchivesuite-users] Netarchive Suite 3.21.0 MessageQueue and      backup questions
>
> Hi Søren and thank you for the replay
>
> We are currently using MessageQueue 4.5.2 and I belive it crashed
> because of memory issues. Last log that I was able to see was the following:
>
> [11/Apr/2013:15:12:40 EEST] [B1088]: Entering Memory State GREEN  from
> previous state ORANGE - allocated memory is 132378K, 70% of total memory
> used
> [11/Apr/2013:15:12:43 EEST] [B1089]: In low memory condition, Broker is
> attempting to free up resources
> [11/Apr/2013:15:12:43 EEST] [B1088]: Entering Memory State ORANGE from
> previous state GREEN  - allocated memory is 173208K, 91% of total memory
> used
> [11/Apr/2013:15:12:43 EEST] [B1088]: Entering Memory State YELLOW from
> previous state ORANGE - allocated memory is 169490K, 89% of total memory
> used
> [11/Apr/2013:15:12:44 EEST] [B1089]: In low memory condition, Broker is
> attempting to free up resources
> [11/Apr/2013:15:12:44 EEST] [B1088]: Entering Memory State ORANGE from
> previous state YELLOW - allocated memory is 175270K, 93% of total memory
> used
> [11/Apr/2013:15:12:47 EEST] [B1088]: Entering Memory State YELLOW from
> previous state ORANGE - allocated memory is 165555K, 87% of total memory
> used
> [11/Apr/2013:15:12:47 EEST] [B1089]: In low memory condition, Broker is
> attempting to free up resources
> [11/Apr/2013:15:12:47 EEST] [B1088]: Entering Memory State ORANGE from
> previous state YELLOW - allocated memory is 172148K, 91% of total memory
> used
>
> -------------------------------------------------------------
> And this is where it ends. Next lines are after restart :
> -------------------------------------------------------------
>
> [13/Apr/2013:13:48:44 EEST] [B1151]: Loading destination
> NLIB_A_THE_BAMON [Queue] with 37720 messages
> [13/Apr/2013:13:48:45 EEST] [B1153]: Loading of destination
> NLIB_A_THE_BAMON [Queue] in process, 10000 of 37720 messages loaded, 26%
> complete
> [13/Apr/2013:13:48:46 EEST] [B1153]: Loading of destination
> NLIB_A_THE_BAMON [Queue] in process, 20000 of 37720 messages loaded, 53%
> complete
> [13/Apr/2013:13:48:48 EEST] [B1153]: Loading of destination
> NLIB_A_THE_BAMON [Queue] in process, 30000 of 37720 messages loaded, 79%
> complete
> [13/Apr/2013:13:48:50 EEST] [B1089]: In low memory condition, Broker is
> attempting to free up resources
> [13/Apr/2013:13:48:50 EEST] [B1088]: Entering Memory State YELLOW from
> previous state GREEN  - allocated memory is 164796K, 87% of total memory
> used
> [13/Apr/2013:13:48:50 EEST] [B1153]: Loading of destination
> NLIB_A_THE_BAMON [Queue] in process, 37720 of 37720 messages loaded,
> 100% complete
> [13/Apr/2013:13:48:53 EEST] [B1089]: In low memory condition, Broker is
> attempting to free up resources
> [13/Apr/2013:13:48:53 EEST] [B1088]: Entering Memory State ORANGE from
> previous state YELLOW - allocated memory is 177593K, 94% of total memory
> used
> [13/Apr/2013:13:49:03 EEST] ERROR [B2085]: Loading Destination
> NLIB_A_THE_BAMON [Queue] failed. Messages stored on that destination
> will not be available.:
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>           at
> com.sun.messaging.jmq.jmsserver.core.Destination.load(Destination.java:4449)
>           at
> com.sun.messaging.jmq.jmsserver.core.Destination.load(Destination.java:4227)
>           at
> com.sun.messaging.jmq.jmsserver.core.Destination.load(Destination.java:4208)
>           at
> com.sun.messaging.jmq.jmsserver.core.Destination.addProducer(Destination.java:3711)
>           at
> com.sun.messaging.jmq.jmsserver.data.handlers.ProducerHandler.addProducer(ProducerHandler.java:295)
>           at
> com.sun.messaging.jmq.jmsserver.data.handlers.ProducerHandler.handle(ProducerHandler.java:179)
>           at
> com.sun.messaging.jmq.jmsserver.data.PacketRouter.handleMessage(PacketRouter.java:188)
>           at
> com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readData(IMQIPConnection.java:1362)
>           at
> com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.process(IMQIPConnection.java:548)
>           at
> com.sun.messaging.jmq.jmsserver.service.imq.OperationRunnable.process(OperationRunnable.java:176)
>           at
> com.sun.messaging.jmq.jmsserver.util.pool.BasicRunnable.run(BasicRunnable.java:499)
>           at java.lang.Thread.run(Thread.java:636)
> [13/Apr/2013:13:49:04 EEST] [B1088]: Entering Memory State YELLOW from
> previous state ORANGE - allocated memory is 151801K, 80% of total memory
> used
> [13/Apr/2013:13:49:04 EEST] [B2085]: Loading Destination
> NLIB_A_THE_BAMON [Queue] failed. Messages stored on that destination
> will not be available.
> [13/Apr/2013:13:49:04 EEST] [B1066]:   Closing:
> guest at 127.0.0.1:42246->jms:41225 because "[B0059]: Client closed the
> connection". Count: service=0 broker=0
> [13/Apr/2013:13:49:04 EEST] [B1065]: Accepting:
> guest at 127.0.0.1:42270->jms:41225. Count: service=1 broker=1
> [13/Apr/2013:13:53:04 EEST] [B1151]: Loading destination
> NLIB_A_THE_BAMON [Queue] with 37720 messages
> [13/Apr/2013:13:53:04 EEST] [B1153]: Loading of destination
> NLIB_A_THE_BAMON [Queue] in process, 10000 of 37720 messages loaded, 26%
> complete
> [13/Apr/2013:13:53:05 EEST] [B1153]: Loading of destination
> NLIB_A_THE_BAMON [Queue] in process, 20000 of 37720 messages loaded, 53%
> complete
> [13/Apr/2013:13:53:06 EEST] [B1089]: In low memory condition, Broker is
> attempting to free up resources
> [13/Apr/2013:13:53:06 EEST] [B1088]: Entering Memory State ORANGE from
> previous state YELLOW - allocated memory is 173401K, 92% of total memory
> used
> [13/Apr/2013:13:53:14 EEST] [B1088]: Entering Memory State GREEN  from
> previous state ORANGE - allocated memory is 136363K, 72% of total memory
> used
> [13/Apr/2013:13:53:14 EEST] [B1153]: Loading of destination
> NLIB_A_THE_BAMON [Queue] in process, 30000 of 37720 messages loaded, 79%
> complete
> [13/Apr/2013:13:53:17 EEST] [B1153]: Loading of destination
> NLIB_A_THE_BAMON [Queue] in process, 37720 of 37720 messages loaded,
> 100% complete
> [13/Apr/2013:13:53:20 EEST] [B1089]: In low memory condition, Broker is
> attempting to free up resources
> [13/Apr/2013:13:53:20 EEST] [B1088]: Entering Memory State ORANGE from
> previous state GREEN  - allocated memory is 178471K, 94% of total memory
> used
> [13/Apr/2013:13:53:23 EEST] ERROR [B2085]: Loading Destination
> NLIB_A_THE_BAMON [Queue] failed. Messages stored on that destination
> will not be available.:
> java.lang.OutOfMemoryError: Java heap space
>
> -----------------------------------------------------------------------
> Seems like it was a memory problem. However I have seen those lines
> before and to fix it have done the following:
>
> 1. shut down NAS
> 2. kill mqbroker process
> 3. restart mqborker process
> 4. start NAS
>
> After that it's OK for some unexplained reason but as we have jobs
> running I cannot kill NAS at the moment.
>
> We are not using Netarchive Suite 4 because there is a new server
> ordered and version 4 is planned to be installed there.
>
> For the database I think I'm using derby. I was planning to start using
> MySQL or other but this is also planned for the new server. External DB
> seems to be more flexible to back up.
>
> So if I shut down NAS and restart it those jobs that heritrix is
> currently working on should restart(because their information is saved
> in the harvester's directory) and the new job that is at the moment
> waiting to start should also commit?
>
> Meelis Mihhailov
> -----------------
> National Library Of Estonia
> meelis at nlib.ee
>
> On 17.04.2013 12:25, Søren Vejrup Carlsen wrote:
>> Hi Meelis.
>> First, we recommend you upgrade to NetarchiveSuite 4.0, the latest stable release with WARC support.
>> 1) About your broker, what version of OpenMQ are you using, and how did it crash? (Lack of memory, or queues filled up). We're currently using OpenMQ 4.5.2. Some apps do (and some don't) reconnect automatically to the JMSbroker. But except for the harvesters, it is not a problem to restart these applications. Most apps read their states from the database, or react on messages from the JMS broker.
>> 2) When we used an embedded Derby database, backup was integrated in NetarchiveSuite. Now when the database in all cases (Derby, PostgreSQL, MySQL) is external, you need to use the tools available for the specific database type.
>>
>> What database are you using?
>>
>> Søren V. Carlsen
>> Developer of NetarchiveSuite
>> Royal Library, Denmark
>>
>>
>> -----Oprindelig meddelelse-----
>> Fra: netarchivesuite-users-bounces at ml.sbforge.org [mailto:netarchivesuite-users-bounces at ml.sbforge.org] På vegne af Meelis Mihhailov
>> Sendt: 17. april 2013 08:01
>> Til: netarchivesuite-users at ml.sbforge.org
>> Emne: [Netarchivesuite-users] Netarchive Suite 3.21.0 MessageQueue and backup questions
>>
>> Hi all!
>>
>> I have two questions about Netarchive Suite with what I hope you can help me with.
>>
>> 1. MessageQueue
>>
>> Some days ago our MessageQueue crashed and therefore Netarchive Suite applications cannot communicate. I restarted the mqbroker service but NAS does not understand the new instance. So none of the messages go to MessageQueue.
>> Is there a way to make NAS understand the new instance of mqbroker and continue to work without restarting the whole system? We have some jobs running but heritrix will not end them when they are finished and we have new jobs pending to start thanks to the crash.
>>
>> 2. NAS backup
>>
>> We are currently working on a backup plan for our archive system and as we have had some issues with NAS database corruption (mainly because of my mistake and testing) we would like to know how to back up the NAS system so if something happens we can just install a new NAS and restore the DB from the backup.
>>
>> Meelis Mihhailov
>> ------------------------
>> National Library Of Estonia
>> meelis at nlib.ee
>>
>> _______________________________________________
>> NetarchiveSuite-users mailing list
>> NetarchiveSuite-users at ml.sbforge.org
>> http://ml.sbforge.org/mailman/listinfo/netarchivesuite-users
>>
>> _______________________________________________
>> NetarchiveSuite-users mailing list
>> NetarchiveSuite-users at ml.sbforge.org
>> http://ml.sbforge.org/mailman/listinfo/netarchivesuite-users
>>
> _______________________________________________
> NetarchiveSuite-users mailing list
> NetarchiveSuite-users at ml.sbforge.org
> http://ml.sbforge.org/mailman/listinfo/netarchivesuite-users
>
> _______________________________________________
> NetarchiveSuite-users mailing list
> NetarchiveSuite-users at ml.sbforge.org
> http://ml.sbforge.org/mailman/listinfo/netarchivesuite-users
>
_______________________________________________
NetarchiveSuite-users mailing list
NetarchiveSuite-users at ml.sbforge.org
http://ml.sbforge.org/mailman/listinfo/netarchivesuite-users