[Netarchivesuite-users] Error creating singleton of class 'dk.netarkivet.harvester.harvesting.JMXHeritrixController':

sara.aubry at bnf.fr sara.aubry at bnf.fr
Mon Feb 15 11:35:54 CET 2010


Bjarne,

Don't know if it's a bug: but it is clearlly a mixed Heritrix/NS problem.
Here is what we found out: 

Job 589 started to get many severe Heritrix errors (almost 12000) at 5:30 
:

Time:  Feb. 15, 2010 05:34:51 GMT Level:  SEVERE Message:  
Failed store of DNS Record for dns:matdeco.fr (in thread 'ToeThread #143: 
dns:matdeco.fr'; in processor 'DNS')
Exception:  
java.io.FileNotFoundException: 
/bnf/netarchivesuite/bin/MAB2/jobs/current/low/589_1266212067690/scratch/tt143http.ris 
(Too many open files)
Stacktrace: java.io.FileNotFoundException: 
/bnf/netarchivesuite/bin/MAB2/jobs/current/low/589_1266212067690/scratch/tt143http.ris 
(Too many open files)
                 at java.io.FileOutputStream.open(Native Method)
                 at java.io.FileOutputStream.(FileOutputStream.java:179)
                 at java.io.FileOutputStream.(FileOutputStream.java:70)
                 at 
org.archive.io.RecordingOutputStream.open(RecordingOutputStream.java:206)
                 at 
org.archive.io.RecordingOutputStream.open(RecordingOutputStream.java:172)
                 at 
org.archive.io.RecordingInputStream.open(RecordingInputStream.java:92)
                 at 
org.archive.util.HttpRecorder.inputWrap(HttpRecorder.java:148)
                 at 
org.archive.crawler.fetcher.FetchDNS.recordDNS(FetchDNS.java:264)
                 at 
org.archive.crawler.fetcher.FetchDNS.storeDNSRecord(FetchDNS.java:204)
                 at 
org.archive.crawler.fetcher.FetchDNS.innerProcess(FetchDNS.java:158)
                 at 
org.archive.crawler.framework.Processor.process(Processor.java:112)
                 at 
org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:302)
                 at 
org.archive.crawler.framework.ToeThread.run(ToeThread.java:151)
 ----------------------------------------------------------------- 
-----------------------------------------------------------------

Time:  
Feb. 15, 2010 05:34:51 GMT 
Level:  
SEVERE 
Message:  
Failed getHostAddress for this host: java.net.UnknownHostException: 
atlas302.bnf.fr: atlas302.bnf.fr (in thread 'ToeThread #39: 
dns:www.massecuriteprivee.fr'; in processor 'Archiver')
Exception:  
No associated exception.

 ----------------------------------------------------------------- 
-----------------------------------------------------------------

Time:  Feb. 15, 2010 05:35:03 GMT Level:  WARNING Message:  
RMI TCP Accept-0: accept loop for 
ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=46085] throws
Exception:  
java.net.SocketException: Too many open files
Stacktrace: java.net.SocketException: Too many open files
                 at java.net.PlainSocketImpl.socketAccept(Native Method)
                 at 
java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
                 at 
java.net.ServerSocket.implAccept(ServerSocket.java:453)
                 at java.net.ServerSocket.accept(ServerSocket.java:421)
                 at 
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
                 at 
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
                 at java.lang.Thread.run(Thread.java:619)

 ----------------------------------------------------------------- 
-----------------------------------------------------------------




Heritrix tried to open many work files but was limited to 1024 (we already 
had this problem with Heritrix and solved it using "ulimit -n 32768" as 
bash command line before we launch Heritrix). 
Then, NS lost JMX connexion with Heritrix, thought the HarvestController 
was available and tried to give our 361 remaining jobs (one after the 
other) to this specific machine. 

Two questions remain for us:
- is there a way and a place to run a "ulimit -n 32768" in NS before 
launching Heritrix?
- isn't NS checking that something is running on the configured port 
before it is actually launching a new Heritrix instance?

Sara 








Message de : Bjarne Andersen <bja at statsbiblioteket.dk> 
                      15/02/2010 10:54

Envoyé par : 
<netarchivesuite-users-bounces at lists.gforge.statsbiblioteket.dk>

Veuillez répondre à 
<netarchivesuite-users at lists.gforge.statsbiblioteket.dk>



Pour
"'netarchivesuite-users at lists.gforge.statsbiblioteket.dk'" 
<netarchivesuite-users at lists.gforge.statsbiblioteket.dk>
Copie

Objet
Re: [Netarchivesuite-users] Error creating singleton of class 
'dk.netarkivet.harvester.harvesting.JMXHeritrixController':



Ah. In that case you could have discovered a new bug. 

I am pretty sure that Jonas was right in his previous answer about unique 
port numbers. But the problem here could bet hat the NS-HarvestController 
is started up well and accepting new jobs - but can't start the 
heritrix-process because of portnumber problems thus failing 1 job, taking 
another, failing again and so on.

The problem could be configuration as Jonas pointed out - but I know BnF 
people are clever people, so I don't really believe in that. The problem 
could also be that there is a hanging heritrix-process (from a previous 
job) that NS-HarvestController failed to shut down (or even kill). I think 
that situation could cause the situation you experienced.

The prefered logic would be that if NS-HarvestController fails to start a 
heritrix-process it should be a little cleverer and not just taking 
another job failing again. So it the heritrix-startup fails it should most 
likely stop accepting new jobs at all and give a serious warning.

If your system is still running you could check processes on the harvester 
server to see if there is an orphan heritrix process.

Best
Bjarne

> -----Original Message-----
> From: netarchivesuite-users-
> bounces at lists.gforge.statsbiblioteket.dk [mailto:netarchivesuite-
> users-bounces at lists.gforge.statsbiblioteket.dk] On Behalf Of
> sara.aubry at bnf.fr
> Sent: Monday, February 15, 2010 10:24 AM
> To: netarchivesuite-users at lists.gforge.statsbiblioteket.dk
> Subject: Re: [Netarchivesuite-users] Error creating singleton of
> class 'dk.netarkivet.harvester.harvesting.JMXHeritrixController':
> 
> Hi Bjarne,
> 
> We are using version 3.10.1.
> And yes they all failed this morning between 6:35 and 7:07.
> 
> Sara
> 
> 
> 
> 
> 
> 
> 
> 
> Message de : Bjarne Andersen <bja at statsbiblioteket.dk>
>                       15/02/2010 10:18
> 
> Envoyé par :
> <netarchivesuite-users-bounces at lists.gforge.statsbiblioteket.dk>
> 
> Veuillez répondre à
> <netarchivesuite-users at lists.gforge.statsbiblioteket.dk>
> 
> 
> 
> Pour
> "netarchivesuite-users at lists.gforge.statsbiblioteket.dk"
> <netarchivesuite-users at lists.gforge.statsbiblioteket.dk>
> Copie
> 
> Objet
> Re: [Netarchivesuite-users] Error creating singleton of class
> 'dk.netarkivet.harvester.harvesting.JMXHeritrixController':
> 
> 
> 
> Which version of NS are you running? I think there was a previous
> bug in
> the heritrix integration that could have a harvester instance grab
> jobs
> from the queue and imediately fail them.
> Did all your jobs fail in a short time?
> Best
> bjarne
> 
> Sent fra min HTC Touch Pro
> 
> ----- Oprindelig meddelelse -----
> Fra: Jonas Lindberg Frellesen <jolf at kb.dk>
> Sendt: 15. februar 2010 10:14
> Til: netarchivesuite-users at lists.gforge.statsbiblioteket.dk
> <netarchivesuite-users at lists.gforge.statsbiblioteket.dk>
> Emne: Re: [Netarchivesuite-users] Error creating singleton of class
> 'dk.netarkivet.harvester.harvesting.JMXHeritrixController':
> 
> 
> Hi Sara
> 
> The cause of the error can be found in this line:
> Caused by: dk.netarkivet.common.exceptions.IOFailure: Port 8070
> already in
> use, or port is out of range
> 
> It means that another application on the same machine is using the
> port
> 8070, and the application you are looking at can therefore not use
> the
> same port as the other application.
> You can solve this problem by assigning an unused port to the
> application
> instead of 8070.
> 
> I cannot see exactly which port setting it is taking about, but it
> can
> only be one of the following:
> settings.common.jmx.port
> settings.common.jmx.rmiPort
> settings.harvester.harvesting.heritrix.guiPort
> settings.harvester.harvesting.heritrix.jmxPort
> 
> You can find them in the settings file for the application, or
> perhaps the
> deploy configuration file.
> All of these port have to be unique and unused by other
> applications.
> 
> 
> Best regards,
> Jonas
> 
> 
> -----Oprindelig meddelelse-----
> Fra: netarchivesuite-users-bounces at lists.gforge.statsbiblioteket.dk
> [mailto:netarchivesuite-users-
> bounces at lists.gforge.statsbiblioteket.dk] På
> vegne af sara.aubry at bnf.fr
> Sendt: 15. februar 2010 09:39
> Til: netarchivesuite-users at lists.gforge.statsbiblioteket.dk
> Emne: [Netarchivesuite-users] Error creating singleton of class
> 'dk.netarkivet.harvester.harvesting.JMXHeritrixController':
> 
> Hello Everyone,
> 
> We are still running a large test crawl at BnF.
> Everything went well last week, and just coming back from weekend,
> I have
> 361 jobs that failed due to this reason :
> Error creating singleton of class
> 'dk.netarkivet.harvester.harvesting.JMXHeritrixController':
> 
> Do you know what could have caused this error ?
> 
> Thanks in advance!
> 
> Sara
> 
> --------------------------
> 
> Here is the long stack :
> dk.netarkivet.common.exceptions.ArgumentNotValid: Error creating
> singleton
> of class
> 'dk.netarkivet.harvester.harvesting.JMXHeritrixController':
>                  at
> dk.netarkivet.common.utils.SettingsFactory.getInstance(SettingsFact
> ory.java:94)
>                  at
> dk.netarkivet.harvester.harvesting.HeritrixControllerFactory.getDef
> aultHeritrixController(HeritrixControllerFactory.java:54)
>                  at
> dk.netarkivet.harvester.harvesting.HeritrixLauncher.doCrawl(Heritri
> xLauncher.java:200)
>                  at
> dk.netarkivet.harvester.harvesting.HarvestController.runHarvest(Har
> vestController.java:236)
>                  at
> dk.netarkivet.harvester.harvesting.distribute.HarvestControllerServ
> er$HarvesterThread.run(HarvestControllerServer.java:650)
> Caused by: dk.netarkivet.common.exceptions.IOFailure: Port 8070
> already in
> use, or port is out of range
>                  at
> dk.netarkivet.common.utils.SystemUtils.checkPortNotUsed(SystemUtils
> .java:91)
>                  at
> dk.netarkivet.harvester.harvesting.JMXHeritrixController.<init>(JMX
> HeritrixController.java:237)
>                  at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>                  at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstru
> ctorAccessorImpl.java:39)
>                  at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delegatin
> gConstructorAccessorImpl.java:27)
>                  at
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>                  at
> dk.netarkivet.common.utils.SettingsFactory.getInstance(SettingsFact
> ory.java:92)
>                  ... 4 more
> Caused by: java.net.BindException: Address already in use
>                  at java.net.PlainSocketImpl.socketBind(Native
> Method)
>                  at
> java.net.PlainSocketImpl.bind(PlainSocketImpl.java:365)
>                  at
> java.net.ServerSocket.bind(ServerSocket.java:319)
>                  at
> java.net.ServerSocket.<init>(ServerSocket.java:185)
>                  at
> java.net.ServerSocket.<init>(ServerSocket.java:141)
>                  at
> dk.netarkivet.common.utils.SystemUtils.checkPortNotUsed(SystemUtils
> .java:88)
>                  ... 10 more
> 
> 
> 
> 
> 
> 
> 
> Avant d'imprimer, pensez ? l'environnement.
> Consider the environment before printing this mail.
> 
> _______________________________________________
> NetarchiveSuite-users mailing list
> NetarchiveSuite-users at lists.gforge.statsbiblioteket.dk
> https://lists.gforge.statsbiblioteket.dk/mailman/listinfo/netarchiv
> esuite-users
> 
> 
> _______________________________________________
> NetarchiveSuite-users mailing list
> NetarchiveSuite-users at lists.gforge.statsbiblioteket.dk
> https://lists.gforge.statsbiblioteket.dk/mailman/listinfo/netarchiv
> esuite-users
> 
> 
> 
> 
> 
> 
> Avant d'imprimer, pensez à l'environnement.
> Consider the environment before printing this mail.
> _______________________________________________
> NetarchiveSuite-users mailing list
> NetarchiveSuite-users at lists.gforge.statsbiblioteket.dk
> https://lists.gforge.statsbiblioteket.dk/mailman/listinfo/netarchiv
> esuite-users

_______________________________________________
NetarchiveSuite-users mailing list
NetarchiveSuite-users at lists.gforge.statsbiblioteket.dk
https://lists.gforge.statsbiblioteket.dk/mailman/listinfo/netarchivesuite-users







Avant d'imprimer, pensez à l'environnement. 
Consider the environment before printing this mail.   



More information about the NetarchiveSuite-users mailing list