[Netarchivesuite-users] Upload errors

[Bjarne Andersen]

If your jobs have been reported finished (or in this case failed because of an upload error) you can use a upload commandline.

I usually use the following (in a bash script):
export INSTALDIR=/path_to_your_installation.... (put the right one in yourself)
export CLASSPATH=$INSTALDIR/lib/dk.netarkivet.archive.jar
find $INSTALDIR/oldjobs/*/arcs/*.arc | xargs java dk.netarkivet.archive.tools.Upload
find $INSTALDIR/oldjobs/*/metadata/*.arc | xargs java dk.netarkivet.archive.tools.Upload

The error looks a bit strange - One of the developers have to answer that one.

best
Bjarne


aponb at gmx.at wrote:
> I checked the jobs which run over the weekend and realized that some of 
> them failed due to an upload error.
> I've got following exception in my logs:
> 
> Error uploading arcfile '/home/onbpre/netarchive/scripts/simple_harvest/server2/49_1212195698720/arcs/49-1-20080531020618-00008-webcrawler01.arc.gz' Will be moved to '/home/onbpre/netarchive/scripts/simple_harvest/oldjobs2'
> dk.netarkivet.common.exceptions.IOFailure: Could not store 'server2/49_1212195698720/arcs/49-1-20080531020618-00008-webcrawler01.arc.gz' after 3 attempts. Giving up.
> Client-side exception occurred while storing 'server2/49_1212195698720/arcs/49-1-20080531020618-00008-webcrawler01.arc.gz' on attempt number 1 of 3.
> Error creating singleton
>     dk.netarkivet.common.utils.SettingsFactory.getInstance(SettingsFactory.java:102)
>     dk.netarkivet.common.distribute.RemoteFileFactory.getInstance(RemoteFileFactory.java:51)
>     dk.netarkivet.common.distribute.RemoteFileFactory.getDistributefileInstance(RemoteFileFactory.java:74)
>     dk.netarkivet.archive.arcrepository.distribute.StoreMessage.<init>(StoreMessage.java:55)
>     dk.netarkivet.archive.arcrepository.distribute.JMSArcRepositoryClient.store(JMSArcRepositoryClient.java:230)
>     dk.netarkivet.harvester.harvesting.HarvestController.uploadFiles(HarvestController.java:299)
>     dk.netarkivet.harvester.harvesting.HarvestController.storeFiles(HarvestController.java:244)
>     dk.netarkivet.harvester.harvesting.distribute.HarvestControllerServer.processHarvestInfoFile(HarvestControllerServer.java:550)
>     dk.netarkivet.harvester.harvesting.distribute.HarvestControllerServer.access$300(HarvestControllerServer.java:83)
>     dk.netarkivet.harvester.harvesting.distribute.HarvestControllerServer$HarvesterThread.run(HarvestControllerServer.java:647)
> Client-side exception occurred while storing 'server2/49_1212195698720/arcs/49-1-20080531020618-00008-webcrawler01.arc.gz' on attempt number 2 of 3.
> Error creating singleton
>     dk.netarkivet.common.utils.SettingsFactory.getInstance(SettingsFactory.java:102)
>     dk.netarkivet.common.distribute.RemoteFileFactory.getInstance(RemoteFileFactory.java:51)
>     dk.netarkivet.common.distribute.RemoteFileFactory.getDistributefileInstance(RemoteFileFactory.java:74)
>     dk.netarkivet.archive.arcrepository.distribute.StoreMessage.<init>(StoreMessage.java:55)
>     dk.netarkivet.archive.arcrepository.distribute.JMSArcRepositoryClient.store(JMSArcRepositoryClient.java:230)
>     dk.netarkivet.harvester.harvesting.HarvestController.uploadFiles(HarvestController.java:299)
>     dk.netarkivet.harvester.harvesting.HarvestController.storeFiles(HarvestController.java:244)
>     dk.netarkivet.harvester.harvesting.distribute.HarvestControllerServer.processHarvestInfoFile(HarvestControllerServer.java:550)
>     dk.netarkivet.harvester.harvesting.distribute.HarvestControllerServer.access$300(HarvestControllerServer.java:83)
>     dk.netarkivet.harvester.harvesting.distribute.HarvestControllerServer$HarvesterThread.run(HarvestControllerServer.java:647)
> Client-side exception occurred while storing 'server2/49_1212195698720/arcs/49-1-20080531020618-00008-webcrawler01.arc.gz' on attempt number 3 of 3.
> Error creating singleton
>     dk.netarkivet.common.utils.SettingsFactory.getInstance(SettingsFactory.java:102)
>     dk.netarkivet.common.distribute.RemoteFileFactory.getInstance(RemoteFileFactory.java:51)
>     dk.netarkivet.common.distribute.RemoteFileFactory.getDistributefileInstance(RemoteFileFactory.java:74)
>     dk.netarkivet.archive.arcrepository.distribute.StoreMessage.<init>(StoreMessage.java:55)
>     dk.netarkivet.archive.arcrepository.distribute.JMSArcRepositoryClient.store(JMSArcRepositoryClient.java:230)
>     dk.netarkivet.harvester.harvesting.HarvestController.uploadFiles(HarvestController.java:299)
>     dk.netarkivet.harvester.harvesting.HarvestController.storeFiles(HarvestController.java:244)
>     dk.netarkivet.harvester.harvesting.distribute.HarvestControllerServer.processHarvestInfoFile(HarvestControllerServer.java:550)
>     dk.netarkivet.harvester.harvesting.distribute.HarvestControllerServer.access$300(HarvestControllerServer.java:83)
>     dk.netarkivet.harvester.harvesting.distribute.HarvestControllerServer$HarvesterThread.run(HarvestControllerServer.java:647)
> 
> 
> 
> My questions are now. What went wrong and what can i do that it will not 
> happen again. And how can I transfer the files from the oldjobs 
> directory in a offical way? Or should I just copy each gz file from the 
> oldjobs2/jobid/arc directory to my bitarchive?
> 
> Thanks in advance for giving me a hint!
> 
> _______________________________________________
> NetarchiveSuite-users mailing list
> NetarchiveSuite-users at lists.gforge.statsbiblioteket.dk
> https://lists.gforge.statsbiblioteket.dk/mailman/listinfo/netarchivesuite-users

-- 
Bjarne Andersen
Driftsleder - netarkivet.dk

Statsbiblioteket
Universitetsparken
8000 Århus C
Tlf. 89462165 - Mobil 25662353
CVR/SE 10100682 - EAN 5798000791084
http://netarkivet.dk
-------------- next part --------------
A non-text attachment was scrubbed...
Name: netarkivet.vcf
Type: text/x-vcard
Size: 312 bytes
Desc: not available
URL: <http://ml.sbforge.org/pipermail/netarchivesuite-users/attachments/20080602/3de10afb/attachment-0002.vcf>