[Netarchivesuite-users] Upload errors

[Kåre Fiedler Christiansen]

On Mon, 2008-06-02 at 13:51 +0200, aponb at gmx.at wrote:
> >
> > If your jobs have been reported finished (or in this case failed because of an upload error) you can use a upload commandline.
> >
> > I usually use the following (in a bash script):
> > export INSTALDIR=/path_to_your_installation.... (put the right one in yourself)
> > export CLASSPATH=$INSTALDIR/lib/dk.netarkivet.archive.jar
> > find $INSTALDIR/oldjobs/*/arcs/*.arc | xargs java dk.netarkivet.archive.tools.Upload
> > find $INSTALDIR/oldjobs/*/metadata/*.arc | xargs java dk.netarkivet.archive.tools.Upload
> >
> > The error looks a bit strange - One of the developers have to answer that one.
> >   
> 
> Thanks for your quick answer!
> The Upload Program seems to upload the data to the bitarchive. But it 
> will not reply after the upload. I am getting the following correct 
> message after starting the Upload Program:
> 
> ...
> INFO: JMSArcRepository listens for replies on channel '[Queue 
> 'PROD_COMMON_THIS_HACO_127_0_0_1_8076']'
> Uploading file 
> '/home/onbpre/netarchive/scripts/simple_harvest/oldjobs2/49_1212195698720/arcs/49-1-20080531010229-00004-webcrawler01.arc.gz'...
> 02.06.2008 13:31:48 org.slf4j.impl.JCLLoggerAdapter info
> INFO: Logging to org.slf4j.impl.JCLLoggerAdapter(org.mortbay.log) via 
> org.mortbay.log.Slf4jLog
> 02.06.2008 13:31:48 org.slf4j.impl.JCLLoggerAdapter info
> INFO: jetty-6.1.1
> 02.06.2008 13:31:48 org.slf4j.impl.JCLLoggerAdapter info
> INFO: Started SocketConnector @ 0.0.0.0:5442
> ...
> 
> Waiting till the upload is complete, but the program will not answer, 
> although the systemstate already says:
> 
> INFO: Store OK: '49-1-20080531010229-00004-webcrawler01.arc.gz'
> 
> The file is also already located on disk in the bitarachive.
> 
> I was waiting for a while and then I canceled the program with STRG-C. 
> And did restart it again. After that I am getting the following message:
> 
> 02.06.2008 13:37:31 org.slf4j.impl.JCLLoggerAdapter info
> INFO: Started SocketConnector @ 0.0.0.0:5442
> Uploading file 
> '/home/onbpre/netarchive/scripts/simple_harvest/oldjobs2/49_1212195698720/arcs/49-1-20080531010229-00004-webcrawler01.arc.gz' 
> succeeded
> All files processed, closing connection to ArcRepository
> 
> Is this problem already known? What can I do to avoid it?

It's an artifact of running everything on one machine with the same
settingsfile :-/

The reply for the store message is sent to a JMS queue, and the queue
name is generated in a way that is expected to be uniq by using the
hostname and a portnumber, plus a string identifying the queue.

In this case, it will probably share the queuename with a harvester.
That gives a 50/50 chance that you will actually receive a reply.

The solution is to use a settings-file made for this tool, with a
specialised port, and specifying the setting file using 

"-Ddk.netarkivet.settings.file=/path/to/settings-file-for-tool.xml"

or overriding the portnumber on the command line, using

"-Dsettings.common.http.port=9999" 

(where 9999 is a free port number)

I have updated feature request 291 with a link to this conversation.
https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=291

Best,
  Kåre