[Netarchivesuite-users] Error creating singleton of class 'dk.netarkivet.harvester.harvesting.JMXHeritrixController':

Bjarne Andersen bja at statsbiblioteket.dk
Mon Feb 15 11:49:53 CET 2010


The workround for "Too many open files is described here":
http://crawler.archive.org/faq.html#toomanyopenfiles

best
Bjarne
________________________________________
Fra: netarchivesuite-users-bounces at lists.gforge.statsbiblioteket.dk [netarchivesuite-users-bounces at lists.gforge.statsbiblioteket.dk] På vegne af sara.aubry at bnf.fr [sara.aubry at bnf.fr]
Sendt: 15. februar 2010 11:35
Til: netarchivesuite-users at lists.gforge.statsbiblioteket.dk
Cc: bert.wendland at bnf.fr; PAUL.FIEVRE at bnf.fr
Emne: Re: [Netarchivesuite-users] Error creating singleton of class 'dk.netarkivet.harvester.harvesting.JMXHeritrixController':

Bjarne,

Don't know if it's a bug: but it is clearlly a mixed Heritrix/NS problem.
Here is what we found out:

Job 589 started to get many severe Heritrix errors (almost 12000) at 5:30
:

Time:  Feb. 15, 2010 05:34:51 GMT Level:  SEVERE Message:
Failed store of DNS Record for dns:matdeco.fr (in thread 'ToeThread #143:
dns:matdeco.fr'; in processor 'DNS')
Exception:
java.io.FileNotFoundException:
/bnf/netarchivesuite/bin/MAB2/jobs/current/low/589_1266212067690/scratch/tt143http.ris
(Too many open files)
Stacktrace: java.io.FileNotFoundException:
/bnf/netarchivesuite/bin/MAB2/jobs/current/low/589_1266212067690/scratch/tt143http.ris
(Too many open files)
                 at java.io.FileOutputStream.open(Native Method)
                 at java.io.FileOutputStream.(FileOutputStream.java:179)
                 at java.io.FileOutputStream.(FileOutputStream.java:70)
                 at
org.archive.io.RecordingOutputStream.open(RecordingOutputStream.java:206)
                 at
org.archive.io.RecordingOutputStream.open(RecordingOutputStream.java:172)
                 at
org.archive.io.RecordingInputStream.open(RecordingInputStream.java:92)
                 at
org.archive.util.HttpRecorder.inputWrap(HttpRecorder.java:148)
                 at
org.archive.crawler.fetcher.FetchDNS.recordDNS(FetchDNS.java:264)
                 at
org.archive.crawler.fetcher.FetchDNS.storeDNSRecord(FetchDNS.java:204)
                 at
org.archive.crawler.fetcher.FetchDNS.innerProcess(FetchDNS.java:158)
                 at
org.archive.crawler.framework.Processor.process(Processor.java:112)
                 at
org.archive.crawler.framework.ToeThread.processCrawlUri(ToeThread.java:302)
                 at
org.archive.crawler.framework.ToeThread.run(ToeThread.java:151)
 -----------------------------------------------------------------
-----------------------------------------------------------------

Time:
Feb. 15, 2010 05:34:51 GMT
Level:
SEVERE
Message:
Failed getHostAddress for this host: java.net.UnknownHostException:
atlas302.bnf.fr: atlas302.bnf.fr (in thread 'ToeThread #39:
dns:www.massecuriteprivee.fr'; in processor 'Archiver')
Exception:
No associated exception.

 -----------------------------------------------------------------
-----------------------------------------------------------------

Time:  Feb. 15, 2010 05:35:03 GMT Level:  WARNING Message:
RMI TCP Accept-0: accept loop for
ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=46085] throws
Exception:
java.net.SocketException: Too many open files
Stacktrace: java.net.SocketException: Too many open files
                 at java.net.PlainSocketImpl.socketAccept(Native Method)
                 at
java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
                 at
java.net.ServerSocket.implAccept(ServerSocket.java:453)
                 at java.net.ServerSocket.accept(ServerSocket.java:421)
                 at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
                 at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
                 at java.lang.Thread.run(Thread.java:619)

 -----------------------------------------------------------------
-----------------------------------------------------------------




Heritrix tried to open many work files but was limited to 1024 (we already
had this problem with Heritrix and solved it using "ulimit -n 32768" as
bash command line before we launch Heritrix).
Then, NS lost JMX connexion with Heritrix, thought the HarvestController
was available and tried to give our 361 remaining jobs (one after the
other) to this specific machine.

Two questions remain for us:
- is there a way and a place to run a "ulimit -n 32768" in NS before
launching Heritrix?
- isn't NS checking that something is running on the configured port
before it is actually launching a new Heritrix instance?

Sara








Message de : Bjarne Andersen <bja at statsbiblioteket.dk>
                      15/02/2010 10:54

Envoyé par :
<netarchivesuite-users-bounces at lists.gforge.statsbiblioteket.dk>

Veuillez répondre à
<netarchivesuite-users at lists.gforge.statsbiblioteket.dk>



Pour
"'netarchivesuite-users at lists.gforge.statsbiblioteket.dk'"
<netarchivesuite-users at lists.gforge.statsbiblioteket.dk>
Copie

Objet
Re: [Netarchivesuite-users] Error creating singleton of class
'dk.netarkivet.harvester.harvesting.JMXHeritrixController':



Ah. In that case you could have discovered a new bug.

I am pretty sure that Jonas was right in his previous answer about unique
port numbers. But the problem here could bet hat the NS-HarvestController
is started up well and accepting new jobs - but can't start the
heritrix-process because of portnumber problems thus failing 1 job, taking
another, failing again and so on.

The problem could be configuration as Jonas pointed out - but I know BnF
people are clever people, so I don't really believe in that. The problem
could also be that there is a hanging heritrix-process (from a previous
job) that NS-HarvestController failed to shut down (or even kill). I think
that situation could cause the situation you experienced.

The prefered logic would be that if NS-HarvestController fails to start a
heritrix-process it should be a little cleverer and not just taking
another job failing again. So it the heritrix-startup fails it should most
likely stop accepting new jobs at all and give a serious warning.

If your system is still running you could check processes on the harvester
server to see if there is an orphan heritrix process.

Best
Bjarne

> -----Original Message-----
> From: netarchivesuite-users-
> bounces at lists.gforge.statsbiblioteket.dk [mailto:netarchivesuite-
> users-bounces at lists.gforge.statsbiblioteket.dk] On Behalf Of
> sara.aubry at bnf.fr
> Sent: Monday, February 15, 2010 10:24 AM
> To: netarchivesuite-users at lists.gforge.statsbiblioteket.dk
> Subject: Re: [Netarchivesuite-users] Error creating singleton of
> class 'dk.netarkivet.harvester.harvesting.JMXHeritrixController':
>
> Hi Bjarne,
>
> We are using version 3.10.1.
> And yes they all failed this morning between 6:35 and 7:07.
>
> Sara
>
>
>
>
>
>
>
>
> Message de : Bjarne Andersen <bja at statsbiblioteket.dk>
>                       15/02/2010 10:18
>
> Envoyé par :
> <netarchivesuite-users-bounces at lists.gforge.statsbiblioteket.dk>
>
> Veuillez répondre à
> <netarchivesuite-users at lists.gforge.statsbiblioteket.dk>
>
>
>
> Pour
> "netarchivesuite-users at lists.gforge.statsbiblioteket.dk"
> <netarchivesuite-users at lists.gforge.statsbiblioteket.dk>
> Copie
>
> Objet
> Re: [Netarchivesuite-users] Error creating singleton of class
> 'dk.netarkivet.harvester.harvesting.JMXHeritrixController':
>
>
>
> Which version of NS are you running? I think there was a previous
> bug in
> the heritrix integration that could have a harvester instance grab
> jobs
> from the queue and imediately fail them.
> Did all your jobs fail in a short time?
> Best
> bjarne
>
> Sent fra min HTC Touch Pro
>
> ----- Oprindelig meddelelse -----
> Fra: Jonas Lindberg Frellesen <jolf at kb.dk>
> Sendt: 15. februar 2010 10:14
> Til: netarchivesuite-users at lists.gforge.statsbiblioteket.dk
> <netarchivesuite-users at lists.gforge.statsbiblioteket.dk>
> Emne: Re: [Netarchivesuite-users] Error creating singleton of class
> 'dk.netarkivet.harvester.harvesting.JMXHeritrixController':
>
>
> Hi Sara
>
> The cause of the error can be found in this line:
> Caused by: dk.netarkivet.common.exceptions.IOFailure: Port 8070
> already in
> use, or port is out of range
>
> It means that another application on the same machine is using the
> port
> 8070, and the application you are looking at can therefore not use
> the
> same port as the other application.
> You can solve this problem by assigning an unused port to the
> application
> instead of 8070.
>
> I cannot see exactly which port setting it is taking about, but it
> can
> only be one of the following:
> settings.common.jmx.port
> settings.common.jmx.rmiPort
> settings.harvester.harvesting.heritrix.guiPort
> settings.harvester.harvesting.heritrix.jmxPort
>
> You can find them in the settings file for the application, or
> perhaps the
> deploy configuration file.
> All of these port have to be unique and unused by other
> applications.
>
>
> Best regards,
> Jonas
>
>
> -----Oprindelig meddelelse-----
> Fra: netarchivesuite-users-bounces at lists.gforge.statsbiblioteket.dk
> [mailto:netarchivesuite-users-
> bounces at lists.gforge.statsbiblioteket.dk] På
> vegne af sara.aubry at bnf.fr
> Sendt: 15. februar 2010 09:39
> Til: netarchivesuite-users at lists.gforge.statsbiblioteket.dk
> Emne: [Netarchivesuite-users] Error creating singleton of class
> 'dk.netarkivet.harvester.harvesting.JMXHeritrixController':
>
> Hello Everyone,
>
> We are still running a large test crawl at BnF.
> Everything went well last week, and just coming back from weekend,
> I have
> 361 jobs that failed due to this reason :
> Error creating singleton of class
> 'dk.netarkivet.harvester.harvesting.JMXHeritrixController':
>
> Do you know what could have caused this error ?
>
> Thanks in advance!
>
> Sara
>
> --------------------------
>
> Here is the long stack :
> dk.netarkivet.common.exceptions.ArgumentNotValid: Error creating
> singleton
> of class
> 'dk.netarkivet.harvester.harvesting.JMXHeritrixController':
>                  at
> dk.netarkivet.common.utils.SettingsFactory.getInstance(SettingsFact
> ory.java:94)
>                  at
> dk.netarkivet.harvester.harvesting.HeritrixControllerFactory.getDef
> aultHeritrixController(HeritrixControllerFactory.java:54)
>                  at
> dk.netarkivet.harvester.harvesting.HeritrixLauncher.doCrawl(Heritri
> xLauncher.java:200)
>                  at
> dk.netarkivet.harvester.harvesting.HarvestController.runHarvest(Har
> vestController.java:236)
>                  at
> dk.netarkivet.harvester.harvesting.distribute.HarvestControllerServ
> er$HarvesterThread.run(HarvestControllerServer.java:650)
> Caused by: dk.netarkivet.common.exceptions.IOFailure: Port 8070
> already in
> use, or port is out of range
>                  at
> dk.netarkivet.common.utils.SystemUtils.checkPortNotUsed(SystemUtils
> .java:91)
>                  at
> dk.netarkivet.harvester.harvesting.JMXHeritrixController.<init>(JMX
> HeritrixController.java:237)
>                  at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>                  at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstru
> ctorAccessorImpl.java:39)
>                  at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delegatin
> gConstructorAccessorImpl.java:27)
>                  at
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>                  at
> dk.netarkivet.common.utils.SettingsFactory.getInstance(SettingsFact
> ory.java:92)
>                  ... 4 more
> Caused by: java.net.BindException: Address already in use
>                  at java.net.PlainSocketImpl.socketBind(Native
> Method)
>                  at
> java.net.PlainSocketImpl.bind(PlainSocketImpl.java:365)
>                  at
> java.net.ServerSocket.bind(ServerSocket.java:319)
>                  at
> java.net.ServerSocket.<init>(ServerSocket.java:185)
>                  at
> java.net.ServerSocket.<init>(ServerSocket.java:141)
>                  at
> dk.netarkivet.common.utils.SystemUtils.checkPortNotUsed(SystemUtils
> .java:88)
>                  ... 10 more
>
>
>
>
>
>
>
> Avant d'imprimer, pensez ? l'environnement.
> Consider the environment before printing this mail.
>
> _______________________________________________
> NetarchiveSuite-users mailing list
> NetarchiveSuite-users at lists.gforge.statsbiblioteket.dk
> https://lists.gforge.statsbiblioteket.dk/mailman/listinfo/netarchiv
> esuite-users
>
>
> _______________________________________________
> NetarchiveSuite-users mailing list
> NetarchiveSuite-users at lists.gforge.statsbiblioteket.dk
> https://lists.gforge.statsbiblioteket.dk/mailman/listinfo/netarchiv
> esuite-users
>
>
>
>
>
>
> Avant d'imprimer, pensez à l'environnement.
> Consider the environment before printing this mail.
> _______________________________________________
> NetarchiveSuite-users mailing list
> NetarchiveSuite-users at lists.gforge.statsbiblioteket.dk
> https://lists.gforge.statsbiblioteket.dk/mailman/listinfo/netarchiv
> esuite-users

_______________________________________________
NetarchiveSuite-users mailing list
NetarchiveSuite-users at lists.gforge.statsbiblioteket.dk
https://lists.gforge.statsbiblioteket.dk/mailman/listinfo/netarchivesuite-users







Avant d'imprimer, pensez à l'environnement.
Consider the environment before printing this mail.
_______________________________________________
NetarchiveSuite-users mailing list
NetarchiveSuite-users at lists.gforge.statsbiblioteket.dk
https://lists.gforge.statsbiblioteket.dk/mailman/listinfo/netarchivesuite-users




More information about the NetarchiveSuite-users mailing list