[Netarchivesuite-devel] Failed jobs and Jobtimeouttime parameter

[Søren Vejrup Carlsen]

Hi Sara.
The purpose of this parameter is to move the jobs from status STARTED to status FAILED
When we have waited for a week, after which we no longer expect the job to terminate. You might want to raise this number to 4*10080 (i.e. a month).

This was implemented in FR 1014 No good way to mark a non-reported-stopped job as FAILED or DONE.
The method dk.netarkivet.harvester.scheduler.HarvestScheduler.stopTimeoutJobs called from the scheduleJobs method in the same class handles this task.

The data from the harvesters will always be processed, but the status of the job will still be failed!

The class responsible for the processing of the feedback from the harvesters is the dk.netarkivet.harvester.scheduler.HarvestSchedulerMonitorServer and specifically the method:  processCrawlStatusMessage(CrawlStatusMessage cmsg)

Best regards

Søren

-----Oprindelig meddelelse-----
Fra: netarchivesuite-devel-bounces at lists.gforge.statsbiblioteket.dk [mailto:netarchivesuite-devel-bounces at lists.gforge.statsbiblioteket.dk] På vegne af sara.aubry at bnf.fr
Sendt: 4. juni 2010 19:00
Til: netarchivesuite-devel at lists.gforge.statsbiblioteket.dk
Cc: sara.aubry at info.unicaen.fr
Emne: [Netarchivesuite-devel] Failed jobs and Jobtimeouttime parameter

Hello everyone,

A week after the start of our second stage, our first 35 jobs were marked as Failed in the database although they are still up and running on the crawlers.

Nicolas went through the code and found a Jobtimeouttime parameter which is set on 10080 minutes.
What is the purpose of this parameter ?

He also read that it should not keep the HarvestController from updating the job status to Finished and the job details (start and end time, bytes and documents harvested for each domain...) as soon as the job is finished.
Can you confirmed this? We don't want to have inconsistent information in the job details. 

Thanks!

Sara





Avant d'imprimer, pensez ? l'environnement.