[Netarchivesuite-users] Netarchive Suite 3.21.0 MessageQueue and backup questions

Meelis Mihhailov meelis at nlib.ee
Wed Apr 17 11:56:42 CEST 2013


Hi Søren and thank you for the replay

We are currently using MessageQueue 4.5.2 and I belive it crashed 
because of memory issues. Last log that I was able to see was the following:

[11/Apr/2013:15:12:40 EEST] [B1088]: Entering Memory State GREEN  from 
previous state ORANGE - allocated memory is 132378K, 70% of total memory 
used
[11/Apr/2013:15:12:43 EEST] [B1089]: In low memory condition, Broker is 
attempting to free up resources
[11/Apr/2013:15:12:43 EEST] [B1088]: Entering Memory State ORANGE from 
previous state GREEN  - allocated memory is 173208K, 91% of total memory 
used
[11/Apr/2013:15:12:43 EEST] [B1088]: Entering Memory State YELLOW from 
previous state ORANGE - allocated memory is 169490K, 89% of total memory 
used
[11/Apr/2013:15:12:44 EEST] [B1089]: In low memory condition, Broker is 
attempting to free up resources
[11/Apr/2013:15:12:44 EEST] [B1088]: Entering Memory State ORANGE from 
previous state YELLOW - allocated memory is 175270K, 93% of total memory 
used
[11/Apr/2013:15:12:47 EEST] [B1088]: Entering Memory State YELLOW from 
previous state ORANGE - allocated memory is 165555K, 87% of total memory 
used
[11/Apr/2013:15:12:47 EEST] [B1089]: In low memory condition, Broker is 
attempting to free up resources
[11/Apr/2013:15:12:47 EEST] [B1088]: Entering Memory State ORANGE from 
previous state YELLOW - allocated memory is 172148K, 91% of total memory 
used

-------------------------------------------------------------
And this is where it ends. Next lines are after restart :
-------------------------------------------------------------

[13/Apr/2013:13:48:44 EEST] [B1151]: Loading destination 
NLIB_A_THE_BAMON [Queue] with 37720 messages
[13/Apr/2013:13:48:45 EEST] [B1153]: Loading of destination 
NLIB_A_THE_BAMON [Queue] in process, 10000 of 37720 messages loaded, 26% 
complete
[13/Apr/2013:13:48:46 EEST] [B1153]: Loading of destination 
NLIB_A_THE_BAMON [Queue] in process, 20000 of 37720 messages loaded, 53% 
complete
[13/Apr/2013:13:48:48 EEST] [B1153]: Loading of destination 
NLIB_A_THE_BAMON [Queue] in process, 30000 of 37720 messages loaded, 79% 
complete
[13/Apr/2013:13:48:50 EEST] [B1089]: In low memory condition, Broker is 
attempting to free up resources
[13/Apr/2013:13:48:50 EEST] [B1088]: Entering Memory State YELLOW from 
previous state GREEN  - allocated memory is 164796K, 87% of total memory 
used
[13/Apr/2013:13:48:50 EEST] [B1153]: Loading of destination 
NLIB_A_THE_BAMON [Queue] in process, 37720 of 37720 messages loaded, 
100% complete
[13/Apr/2013:13:48:53 EEST] [B1089]: In low memory condition, Broker is 
attempting to free up resources
[13/Apr/2013:13:48:53 EEST] [B1088]: Entering Memory State ORANGE from 
previous state YELLOW - allocated memory is 177593K, 94% of total memory 
used
[13/Apr/2013:13:49:03 EEST] ERROR [B2085]: Loading Destination 
NLIB_A_THE_BAMON [Queue] failed. Messages stored on that destination 
will not be available.:
java.lang.OutOfMemoryError: GC overhead limit exceeded
         at 
com.sun.messaging.jmq.jmsserver.core.Destination.load(Destination.java:4449)
         at 
com.sun.messaging.jmq.jmsserver.core.Destination.load(Destination.java:4227)
         at 
com.sun.messaging.jmq.jmsserver.core.Destination.load(Destination.java:4208)
         at 
com.sun.messaging.jmq.jmsserver.core.Destination.addProducer(Destination.java:3711)
         at 
com.sun.messaging.jmq.jmsserver.data.handlers.ProducerHandler.addProducer(ProducerHandler.java:295)
         at 
com.sun.messaging.jmq.jmsserver.data.handlers.ProducerHandler.handle(ProducerHandler.java:179)
         at 
com.sun.messaging.jmq.jmsserver.data.PacketRouter.handleMessage(PacketRouter.java:188)
         at 
com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readData(IMQIPConnection.java:1362)
         at 
com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.process(IMQIPConnection.java:548)
         at 
com.sun.messaging.jmq.jmsserver.service.imq.OperationRunnable.process(OperationRunnable.java:176)
         at 
com.sun.messaging.jmq.jmsserver.util.pool.BasicRunnable.run(BasicRunnable.java:499)
         at java.lang.Thread.run(Thread.java:636)
[13/Apr/2013:13:49:04 EEST] [B1088]: Entering Memory State YELLOW from 
previous state ORANGE - allocated memory is 151801K, 80% of total memory 
used
[13/Apr/2013:13:49:04 EEST] [B2085]: Loading Destination 
NLIB_A_THE_BAMON [Queue] failed. Messages stored on that destination 
will not be available.
[13/Apr/2013:13:49:04 EEST] [B1066]:   Closing: 
guest at 127.0.0.1:42246->jms:41225 because "[B0059]: Client closed the 
connection". Count: service=0 broker=0
[13/Apr/2013:13:49:04 EEST] [B1065]: Accepting: 
guest at 127.0.0.1:42270->jms:41225. Count: service=1 broker=1
[13/Apr/2013:13:53:04 EEST] [B1151]: Loading destination 
NLIB_A_THE_BAMON [Queue] with 37720 messages
[13/Apr/2013:13:53:04 EEST] [B1153]: Loading of destination 
NLIB_A_THE_BAMON [Queue] in process, 10000 of 37720 messages loaded, 26% 
complete
[13/Apr/2013:13:53:05 EEST] [B1153]: Loading of destination 
NLIB_A_THE_BAMON [Queue] in process, 20000 of 37720 messages loaded, 53% 
complete
[13/Apr/2013:13:53:06 EEST] [B1089]: In low memory condition, Broker is 
attempting to free up resources
[13/Apr/2013:13:53:06 EEST] [B1088]: Entering Memory State ORANGE from 
previous state YELLOW - allocated memory is 173401K, 92% of total memory 
used
[13/Apr/2013:13:53:14 EEST] [B1088]: Entering Memory State GREEN  from 
previous state ORANGE - allocated memory is 136363K, 72% of total memory 
used
[13/Apr/2013:13:53:14 EEST] [B1153]: Loading of destination 
NLIB_A_THE_BAMON [Queue] in process, 30000 of 37720 messages loaded, 79% 
complete
[13/Apr/2013:13:53:17 EEST] [B1153]: Loading of destination 
NLIB_A_THE_BAMON [Queue] in process, 37720 of 37720 messages loaded, 
100% complete
[13/Apr/2013:13:53:20 EEST] [B1089]: In low memory condition, Broker is 
attempting to free up resources
[13/Apr/2013:13:53:20 EEST] [B1088]: Entering Memory State ORANGE from 
previous state GREEN  - allocated memory is 178471K, 94% of total memory 
used
[13/Apr/2013:13:53:23 EEST] ERROR [B2085]: Loading Destination 
NLIB_A_THE_BAMON [Queue] failed. Messages stored on that destination 
will not be available.:
java.lang.OutOfMemoryError: Java heap space

-----------------------------------------------------------------------
Seems like it was a memory problem. However I have seen those lines 
before and to fix it have done the following:

1. shut down NAS
2. kill mqbroker process
3. restart mqborker process
4. start NAS

After that it's OK for some unexplained reason but as we have jobs 
running I cannot kill NAS at the moment.

We are not using Netarchive Suite 4 because there is a new server 
ordered and version 4 is planned to be installed there.

For the database I think I'm using derby. I was planning to start using 
MySQL or other but this is also planned for the new server. External DB 
seems to be more flexible to back up.

So if I shut down NAS and restart it those jobs that heritrix is 
currently working on should restart(because their information is saved 
in the harvester's directory) and the new job that is at the moment 
waiting to start should also commit?

Meelis Mihhailov
-----------------
National Library Of Estonia
meelis at nlib.ee

On 17.04.2013 12:25, Søren Vejrup Carlsen wrote:
> Hi Meelis.
> First, we recommend you upgrade to NetarchiveSuite 4.0, the latest stable release with WARC support.
> 1) About your broker, what version of OpenMQ are you using, and how did it crash? (Lack of memory, or queues filled up). We're currently using OpenMQ 4.5.2. Some apps do (and some don't) reconnect automatically to the JMSbroker. But except for the harvesters, it is not a problem to restart these applications. Most apps read their states from the database, or react on messages from the JMS broker.
> 2) When we used an embedded Derby database, backup was integrated in NetarchiveSuite. Now when the database in all cases (Derby, PostgreSQL, MySQL) is external, you need to use the tools available for the specific database type.
>
> What database are you using?
>
> Søren V. Carlsen
> Developer of NetarchiveSuite
> Royal Library, Denmark
>
>
> -----Oprindelig meddelelse-----
> Fra: netarchivesuite-users-bounces at ml.sbforge.org [mailto:netarchivesuite-users-bounces at ml.sbforge.org] På vegne af Meelis Mihhailov
> Sendt: 17. april 2013 08:01
> Til: netarchivesuite-users at ml.sbforge.org
> Emne: [Netarchivesuite-users] Netarchive Suite 3.21.0 MessageQueue and backup questions
>
> Hi all!
>
> I have two questions about Netarchive Suite with what I hope you can help me with.
>
> 1. MessageQueue
>
> Some days ago our MessageQueue crashed and therefore Netarchive Suite applications cannot communicate. I restarted the mqbroker service but NAS does not understand the new instance. So none of the messages go to MessageQueue.
> Is there a way to make NAS understand the new instance of mqbroker and continue to work without restarting the whole system? We have some jobs running but heritrix will not end them when they are finished and we have new jobs pending to start thanks to the crash.
>
> 2. NAS backup
>
> We are currently working on a backup plan for our archive system and as we have had some issues with NAS database corruption (mainly because of my mistake and testing) we would like to know how to back up the NAS system so if something happens we can just install a new NAS and restore the DB from the backup.
>
> Meelis Mihhailov
> ------------------------
> National Library Of Estonia
> meelis at nlib.ee
>
> _______________________________________________
> NetarchiveSuite-users mailing list
> NetarchiveSuite-users at ml.sbforge.org
> http://ml.sbforge.org/mailman/listinfo/netarchivesuite-users
>
> _______________________________________________
> NetarchiveSuite-users mailing list
> NetarchiveSuite-users at ml.sbforge.org
> http://ml.sbforge.org/mailman/listinfo/netarchivesuite-users
>


More information about the NetarchiveSuite-users mailing list