[Netarchivesuite-users] Netarchive Suite 3.21.0 MessageQueue and backup questions

Wed Apr 17 12:15:58 CEST 2013

Hi Meelis.

The default memory setting for the OpenMQ 4.5.2 is much too low in our experience. 
We change the memory setting by changing the definition of _def_jvm_args in the imqbrokerd script 
as follows: _def_jvm_args="-Xms512m -Xmx512m -Xss192k -XX:+UseConcMarkSweepGC"

About restarting NAS, we recommend that you stop the harvesters as the last thing, so that they can 
finish their harvesting. Note that harvest-data not uploaded from the harvesters will be uploaded to your archive, when the HarvestServers start again.

/Søren

-----Oprindelig meddelelse-----
Fra: netarchivesuite-users-bounces at ml.sbforge.org [mailto:netarchivesuite-users-bounces at ml.sbforge.org] På vegne af Meelis Mihhailov
Sendt: 17. april 2013 11:57
Til: netarchivesuite-users at ml.sbforge.org
Emne: Re: [Netarchivesuite-users] Netarchive Suite 3.21.0 MessageQueue and backup questions

Hi Søren and thank you for the replay

We are currently using MessageQueue 4.5.2 and I belive it crashed because of memory issues. Last log that I was able to see was the following:

[11/Apr/2013:15:12:40 EEST] [B1088]: Entering Memory State GREEN  from previous state ORANGE - allocated memory is 132378K, 70% of total memory used
[11/Apr/2013:15:12:43 EEST] [B1089]: In low memory condition, Broker is attempting to free up resources
[11/Apr/2013:15:12:43 EEST] [B1088]: Entering Memory State ORANGE from previous state GREEN  - allocated memory is 173208K, 91% of total memory used
[11/Apr/2013:15:12:43 EEST] [B1088]: Entering Memory State YELLOW from previous state ORANGE - allocated memory is 169490K, 89% of total memory used
[11/Apr/2013:15:12:44 EEST] [B1089]: In low memory condition, Broker is attempting to free up resources
[11/Apr/2013:15:12:44 EEST] [B1088]: Entering Memory State ORANGE from previous state YELLOW - allocated memory is 175270K, 93% of total memory used
[11/Apr/2013:15:12:47 EEST] [B1088]: Entering Memory State YELLOW from previous state ORANGE - allocated memory is 165555K, 87% of total memory used
[11/Apr/2013:15:12:47 EEST] [B1089]: In low memory condition, Broker is attempting to free up resources
[11/Apr/2013:15:12:47 EEST] [B1088]: Entering Memory State ORANGE from previous state YELLOW - allocated memory is 172148K, 91% of total memory used

-------------------------------------------------------------
And this is where it ends. Next lines are after restart :
-------------------------------------------------------------

[13/Apr/2013:13:48:44 EEST] [B1151]: Loading destination NLIB_A_THE_BAMON [Queue] with 37720 messages
[13/Apr/2013:13:48:45 EEST] [B1153]: Loading of destination NLIB_A_THE_BAMON [Queue] in process, 10000 of 37720 messages loaded, 26% complete
[13/Apr/2013:13:48:46 EEST] [B1153]: Loading of destination NLIB_A_THE_BAMON [Queue] in process, 20000 of 37720 messages loaded, 53% complete
[13/Apr/2013:13:48:48 EEST] [B1153]: Loading of destination NLIB_A_THE_BAMON [Queue] in process, 30000 of 37720 messages loaded, 79% complete
[13/Apr/2013:13:48:50 EEST] [B1089]: In low memory condition, Broker is attempting to free up resources
[13/Apr/2013:13:48:50 EEST] [B1088]: Entering Memory State YELLOW from previous state GREEN  - allocated memory is 164796K, 87% of total memory used
[13/Apr/2013:13:48:50 EEST] [B1153]: Loading of destination NLIB_A_THE_BAMON [Queue] in process, 37720 of 37720 messages loaded, 100% complete
[13/Apr/2013:13:48:53 EEST] [B1089]: In low memory condition, Broker is attempting to free up resources
[13/Apr/2013:13:48:53 EEST] [B1088]: Entering Memory State ORANGE from previous state YELLOW - allocated memory is 177593K, 94% of total memory used
[13/Apr/2013:13:49:03 EEST] ERROR [B2085]: Loading Destination NLIB_A_THE_BAMON [Queue] failed. Messages stored on that destination will not be available.:
java.lang.OutOfMemoryError: GC overhead limit exceeded
         at
com.sun.messaging.jmq.jmsserver.core.Destination.load(Destination.java:4449)
         at
com.sun.messaging.jmq.jmsserver.core.Destination.load(Destination.java:4227)
         at
com.sun.messaging.jmq.jmsserver.core.Destination.load(Destination.java:4208)
         at
com.sun.messaging.jmq.jmsserver.core.Destination.addProducer(Destination.java:3711)
         at
com.sun.messaging.jmq.jmsserver.data.handlers.ProducerHandler.addProducer(ProducerHandler.java:295)
         at
com.sun.messaging.jmq.jmsserver.data.handlers.ProducerHandler.handle(ProducerHandler.java:179)
         at
com.sun.messaging.jmq.jmsserver.data.PacketRouter.handleMessage(PacketRouter.java:188)
         at
com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readData(IMQIPConnection.java:1362)
         at
com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.process(IMQIPConnection.java:548)
         at
com.sun.messaging.jmq.jmsserver.service.imq.OperationRunnable.process(OperationRunnable.java:176)
         at
com.sun.messaging.jmq.jmsserver.util.pool.BasicRunnable.run(BasicRunnable.java:499)
         at java.lang.Thread.run(Thread.java:636)
[13/Apr/2013:13:49:04 EEST] [B1088]: Entering Memory State YELLOW from previous state ORANGE - allocated memory is 151801K, 80% of total memory used
[13/Apr/2013:13:49:04 EEST] [B2085]: Loading Destination NLIB_A_THE_BAMON [Queue] failed. Messages stored on that destination will not be available.
[13/Apr/2013:13:49:04 EEST] [B1066]:   Closing: 
guest at 127.0.0.1:42246->jms:41225 because "[B0059]: Client closed the connection". Count: service=0 broker=0
[13/Apr/2013:13:49:04 EEST] [B1065]: Accepting: 
guest at 127.0.0.1:42270->jms:41225. Count: service=1 broker=1
[13/Apr/2013:13:53:04 EEST] [B1151]: Loading destination NLIB_A_THE_BAMON [Queue] with 37720 messages
[13/Apr/2013:13:53:04 EEST] [B1153]: Loading of destination NLIB_A_THE_BAMON [Queue] in process, 10000 of 37720 messages loaded, 26% complete
[13/Apr/2013:13:53:05 EEST] [B1153]: Loading of destination NLIB_A_THE_BAMON [Queue] in process, 20000 of 37720 messages loaded, 53% complete
[13/Apr/2013:13:53:06 EEST] [B1089]: In low memory condition, Broker is attempting to free up resources
[13/Apr/2013:13:53:06 EEST] [B1088]: Entering Memory State ORANGE from previous state YELLOW - allocated memory is 173401K, 92% of total memory used
[13/Apr/2013:13:53:14 EEST] [B1088]: Entering Memory State GREEN  from previous state ORANGE - allocated memory is 136363K, 72% of total memory used
[13/Apr/2013:13:53:14 EEST] [B1153]: Loading of destination NLIB_A_THE_BAMON [Queue] in process, 30000 of 37720 messages loaded, 79% complete
[13/Apr/2013:13:53:17 EEST] [B1153]: Loading of destination NLIB_A_THE_BAMON [Queue] in process, 37720 of 37720 messages loaded, 100% complete
[13/Apr/2013:13:53:20 EEST] [B1089]: In low memory condition, Broker is attempting to free up resources
[13/Apr/2013:13:53:20 EEST] [B1088]: Entering Memory State ORANGE from previous state GREEN  - allocated memory is 178471K, 94% of total memory used
[13/Apr/2013:13:53:23 EEST] ERROR [B2085]: Loading Destination NLIB_A_THE_BAMON [Queue] failed. Messages stored on that destination will not be available.:
java.lang.OutOfMemoryError: Java heap space

-----------------------------------------------------------------------
Seems like it was a memory problem. However I have seen those lines before and to fix it have done the following:

1. shut down NAS
2. kill mqbroker process
3. restart mqborker process
4. start NAS

After that it's OK for some unexplained reason but as we have jobs running I cannot kill NAS at the moment.

We are not using Netarchive Suite 4 because there is a new server ordered and version 4 is planned to be installed there.

For the database I think I'm using derby. I was planning to start using MySQL or other but this is also planned for the new server. External DB seems to be more flexible to back up.

So if I shut down NAS and restart it those jobs that heritrix is currently working on should restart(because their information is saved in the harvester's directory) and the new job that is at the moment waiting to start should also commit?

Meelis Mihhailov
-----------------
National Library Of Estonia
meelis at nlib.ee

On 17.04.2013 12:25, Søren Vejrup Carlsen wrote:
> Hi Meelis.
> First, we recommend you upgrade to NetarchiveSuite 4.0, the latest stable release with WARC support.
> 1) About your broker, what version of OpenMQ are you using, and how did it crash? (Lack of memory, or queues filled up). We're currently using OpenMQ 4.5.2. Some apps do (and some don't) reconnect automatically to the JMSbroker. But except for the harvesters, it is not a problem to restart these applications. Most apps read their states from the database, or react on messages from the JMS broker.
> 2) When we used an embedded Derby database, backup was integrated in NetarchiveSuite. Now when the database in all cases (Derby, PostgreSQL, MySQL) is external, you need to use the tools available for the specific database type.
>
> What database are you using?
>
> Søren V. Carlsen
> Developer of NetarchiveSuite
> Royal Library, Denmark
>
>
> -----Oprindelig meddelelse-----
> Fra: netarchivesuite-users-bounces at ml.sbforge.org 
> [mailto:netarchivesuite-users-bounces at ml.sbforge.org] På vegne af 
> Meelis Mihhailov
> Sendt: 17. april 2013 08:01
> Til: netarchivesuite-users at ml.sbforge.org
> Emne: [Netarchivesuite-users] Netarchive Suite 3.21.0 MessageQueue and 
> backup questions
>
> Hi all!
>
> I have two questions about Netarchive Suite with what I hope you can help me with.
>
> 1. MessageQueue
>
> Some days ago our MessageQueue crashed and therefore Netarchive Suite applications cannot communicate. I restarted the mqbroker service but NAS does not understand the new instance. So none of the messages go to MessageQueue.
> Is there a way to make NAS understand the new instance of mqbroker and continue to work without restarting the whole system? We have some jobs running but heritrix will not end them when they are finished and we have new jobs pending to start thanks to the crash.
>
> 2. NAS backup
>
> We are currently working on a backup plan for our archive system and as we have had some issues with NAS database corruption (mainly because of my mistake and testing) we would like to know how to back up the NAS system so if something happens we can just install a new NAS and restore the DB from the backup.
>
> Meelis Mihhailov
> ------------------------
> National Library Of Estonia
> meelis at nlib.ee
>
> _______________________________________________
> NetarchiveSuite-users mailing list
> NetarchiveSuite-users at ml.sbforge.org
> http://ml.sbforge.org/mailman/listinfo/netarchivesuite-users
>
> _______________________________________________
> NetarchiveSuite-users mailing list
> NetarchiveSuite-users at ml.sbforge.org
> http://ml.sbforge.org/mailman/listinfo/netarchivesuite-users
>
_______________________________________________
NetarchiveSuite-users mailing list
NetarchiveSuite-users at ml.sbforge.org
http://ml.sbforge.org/mailman/listinfo/netarchivesuite-users