[Netarchivesuite-users] IOException while querying Heritrix statusvia JMX
Søren Vejrup Carlsen
svc at kb.dk
Mon May 18 13:54:22 CEST 2009
Hi Nicolas.
Is there anything to see in the Heritrix logs that might indicate what the problem is?
First, you could try to increase the value of the setting "settings.common.jmx.timeout".
The default value is 120 seconds. You could try increasing it to 300 seconds (5 minutes)
/Søren
Fra: netarchivesuite-users-bounces at lists.gforge.statsbiblioteket.dk [mailto:netarchivesuite-users-bounces at lists.gforge.statsbiblioteket.dk] På vegne af nicolas.giraud at bnf.fr
Sendt: 18. maj 2009 10:34
Til: netarchivesuite-users at lists.gforge.statsbiblioteket.dk
Emne: [Netarchivesuite-users] IOException while querying Heritrix statusvia JMX
Hi,
I think the bug that existed in 1.12 is not fully fixed in 1.14. I'm experiencing trouble right at the start of broad crawl jobs now, with a new harvest template I've been using. My goal was to take the prodiction configuration of Heritrix used by Internet Archive when they performed or last broad crawl in 2008. I started from the default_obeyrobots config, and then started adding the IA settings. This is the harvest template I obtained :
I don't know what configuration parameter causes this, but Heritrix takes a long time to compute the initial frontier, and uses a lot of CPU doing so, hence, even if I can connect to the admin console, I obtain the following error after a couple of minutes:
GRAVE: Mailing netarkivet error: Fatal error while operating job 'Job 13 (state = SUBMITTED, HD = 1, priority = LOWPRIORITY, forcemaxcount = -1, forcemaxbyt
es = 20000000, orderxml = ia_large_2008, numconfigs = 1000)'
dk.netarkivet.common.exceptions.IOFailure: Error during crawling. The crawl may have been only partially completed.
at dk.netarkivet.harvester.harvesting.distribute.HarvestControllerServer$HarvesterThread.run(HarvestControllerServer.java:657)
Caused by: dk.netarkivet.common.exceptions.IOFailure: Failed to find MBean org.archive.crawler:name=13-1-20090515153903599,type=CrawlService.Job,jmxport=817
0,mother=Heritrix,host=acheron2.bnf.fr for getting attribute Status after 17 attempts
at dk.netarkivet.common.utils.JMXUtils.getAttribute(JMXUtils.java:317)
at dk.netarkivet.common.utils.JMXUtils.getAttribute(JMXUtils.java:459)
at dk.netarkivet.harvester.harvesting.JMXHeritrixController.getCrawlJobAttribute(JMXHeritrixController.java:905)
at dk.netarkivet.harvester.harvesting.JMXHeritrixController.crawlIsEnded(JMXHeritrixController.java:487)
at dk.netarkivet.harvester.harvesting.JMXHeritrixController.atFinish(JMXHeritrixController.java:354)
at dk.netarkivet.harvester.harvesting.HeritrixLauncher.doCrawl(HeritrixLauncher.java:193)
at dk.netarkivet.harvester.harvesting.HarvestController.runHarvest(HarvestController.java:221)
at dk.netarkivet.harvester.harvesting.distribute.HarvestControllerServer$HarvesterThread.run(HarvestControllerServer.java:650)
Caused by: javax.management.InstanceNotFoundException: org.archive.crawler:name=13-1-20090515153903599,type=CrawlService.Job,jmxport=8170,mother=Heritrix,ho
st=acheron2.bnf.fr
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:662)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
at com.sun.jmx.remote.security.MBeanServerAccessController.getAttribute(MBeanServerAccessController.java:299)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1403)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1264)
at java.security.AccessController.doPrivileged(Native Method)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1366)
at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:600)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:255)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)
at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl_Stub.getAttribute(Unknown Source)
at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getAttribute(RMIConnector.java:878)
at dk.netarkivet.common.utils.JMXUtils.getAttribute(JMXUtils.java:296)
... 7 more
I've been thinking to modify JMXHeritrixController.getCrawlJobAttribute to allow several retries after a configurable wait period to work around this problem, how about it?
Best regards,
Nicolas
Avant d'imprimer, pensez à l'environnement.
Consider the environment before printing this mail.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://ml.sbforge.org/pipermail/netarchivesuite-users/attachments/20090518/61255a62/attachment-0002.html>
More information about the NetarchiveSuite-users
mailing list