[Netarchivesuite-users] Nothing happens after starting generating dedupcrawllogindex

[Jonas Lindberg Frellesen]

Hi Andreas

You have not defined a file data transfer method.
The configuration of one such element can be seen here: 
http://netarchive.dk/suite/Configuration_Manual_devel#head-680304c34a5d7527b9c6ee04a1a15189f7a7e168

This should be defined on the physicalLocation level of your configuration file.
e.g.
	<thisPhysicalLocation name="ONB">
		<deployMachineUserName>netarchive</deployMachineUserName>
		<deployInstallDir>/home/netarchive/apps/netarchivesuite</deployInstallDir>
		<deployDatabaseDir>/home/netarchive/data/netarchivesuite/db/fullhddb</deployDatabaseDir>
		<settings>
			<common>
				<useReplicaId>ONB</useReplicaId>
                                 	<remoteFile>
					<serverName>wc06</serverName>
					<userName>myUser</userName>
					<userPassword>myPassword</userPassword>
					<port>8100</port>
				</remoteFile>
			</common>
		</settings>
....

The 'Address already in use' error was given because the settings.common.remoteFile.port has a default value of 8100, which is the same port you use for one of your applications. 


I have no experience with several harvester applications on a single machine, but I think they will conflict if they are communicating at the same time.
It should be possible to avoid this by given each harvester application a unique value for settings.common.remoteFile.port.


Best regards
Jonas


-----Oprindelig meddelelse-----
Fra: netarchivesuite-users-bounces at lists.gforge.statsbiblioteket.dk [mailto:netarchivesuite-users-bounces at lists.gforge.statsbiblioteket.dk] På vegne af aponb at gmx.at
Sendt: 19. maj 2009 11:27
Til: netarchivesuite-users at lists.gforge.statsbiblioteket.dk
Emne: [Netarchivesuite-users] Nothinghappens afterstarting generatingdedupcrawllogindex

>
> Hi Andreas.
> In your deploy configuration you haven't defined any BitarchiveApplications.
> This must be the problem: You have only defined a BitarchiveMonitorApplication for your archive replica, but not any BitarchiveApplications for that same replica.
>
> You haven't yet defined,  where to store your harvests!
>
>   
Oups! You're right. I forgot that. I put the baseFileDir for the bitarchive under the BitarchiveMonitorApplication.
Thanks for that help.
And this is the next question. I inserted the BitarchiveApplication. And now I have the following error in the HarvestControllerApplication-Logfile:

dk.netarkivet.common.exceptions.IllegalState: Reply message not ok. 
Message is: 'dk.netarkivet.common.exceptions.ArgumentNotValid: Error creating singleton of class
'dk.netarkivet.common.distribute.HTTPRemoteFile':
dk.netarkivet.common.exceptions.ArgumentNotValid: Error creating singleton of class 'dk.netarkivet.common.distribute.HTTPRemoteFile':
        at
dk.netarkivet.common.utils.SettingsFactory.getInstance(SettingsFactory.java:101)
        at
dk.netarkivet.common.distribute.RemoteFileFactory.getInstance(RemoteFileFactory.java:51)
        at
dk.netarkivet.common.distribute.RemoteFileFactory.getCopyfileInstance(RemoteFileFactory.java:67)
        at
dk.netarkivet.archive.indexserver.distribute.IndexRequestServer.doGenerateIndex(IndexRequestServer.java:170)
        at
dk.netarkivet.archive.indexserver.distribute.IndexRequestServer.access$000(IndexRequestServer.java:57)
        at
dk.netarkivet.archive.indexserver.distribute.IndexRequestServer$1.run(IndexRequestServer.java:136)
Caused by: dk.netarkivet.common.exceptions.IOFailure: Cannot start HTTPRemoteFile registry on port 8100
        at
dk.netarkivet.common.distribute.HTTPRemoteFileRegistry.startServer(HTTPRemoteFileRegistry.java:119)
        at
dk.netarkivet.common.distribute.HTTPRemoteFileRegistry.<init>(HTTPRemoteFileRegistry.java:101)
        at
dk.netarkivet.common.distribute.HTTPRemoteFileRegistry.getInstance(HTTPRemoteFileRegistry.java:138)
        at
dk.netarkivet.common.distribute.HTTPRemoteFile.getRegistry(HTTPRemoteFile.java:149)
        at
dk.netarkivet.common.distribute.HTTPRemoteFile.<init>(HTTPRemoteFile.java:105)
        at
dk.netarkivet.common.distribute.HTTPRemoteFile.getInstance(HTTPRemoteFile.java:140)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
dk.netarkivet.common.utils.SettingsFactory.getInstance(SettingsFactory.java:99)
        ... 5 more
Caused by: java.net.BindException: Address already in use
        at java.net.PlainSocketImpl.socketBind(Native Method)
        at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:359)
        at java.net.ServerSocket.bind(ServerSocket.java:319)
        at java.net.ServerSocket.<init>(ServerSocket.java:185)
        at java.net.ServerSocket.<init>(ServerSocket.java:141)
        at
org.mortbay.jetty.bio.SocketConnector.newServerSocket(SocketConnector.java:80)
        at
org.mortbay.jetty.bio.SocketConnector.open(SocketConnector.java:73)
        at
org.mortbay.jetty.AbstractConnector.doStart(AbstractConnector.java:250)
        at
org.mortbay.jetty.bio.SocketConnector.doStart(SocketConnector.java:147)
        at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:40)
        at org.mortbay.jetty.Server.doStart(Server.java:233)
        at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:40)
        at
dk.netarkivet.common.distribute.HTTPRemoteFileRegistry.startServer(HTTPRemoteFileRegistry.java:117)
        ... 15 more
' in index request for jobs
        at
dk.netarkivet.archive.indexserver.distribute.IndexRequestClient.checkMessageValid(IndexRequestClient.java:303)
        at
dk.netarkivet.archive.indexserver.distribute.IndexRequestClient.cacheData(IndexRequestClient.java:180)
        at
dk.netarkivet.archive.indexserver.distribute.IndexRequestClient.cacheData(IndexRequestClient.java:59)
        at
dk.netarkivet.archive.indexserver.FileBasedCache.cache(FileBasedCache.java:164)
        at
dk.netarkivet.archive.indexserver.FileBasedCache.getIndex(FileBasedCache.java:229)
        at
dk.netarkivet.archive.indexserver.distribute.IndexRequestClient.getIndex(IndexRequestClient.java:59)
        at
dk.netarkivet.harvester.harvesting.HarvestController.fetchDeduplicateIndex(HarvestController.java:401)
        at
dk.netarkivet.harvester.harvesting.HarvestController.writeHarvestFiles(HarvestController.java:153)
        at
dk.netarkivet.harvester.harvesting.distribute.HarvestControllerServer$HarvesterThread.run(HarvestControllerServer.java:642)

I have given every Application own Ports (my deployfile is enclosed) - so the address already in use message surprises me.
I am sure this is just a silly configuration mistake, but I can't find it. Would be nice if you have time for a quick view!

Thanks in advance
a.