[Netarchivesuite-devel] Failed jobs and Jobtimeouttime parameter

sara.aubry at bnf.fr sara.aubry at bnf.fr
Tue Jun 8 09:18:04 CEST 2010


Hi Søren,

Thanks for your answer.
What if we manually change the status in the DB from failed back to 
started again?
Is the communication going to be ok again?
Or is the broker storing some information about failed marked jobs? 

Sara










Message de : Søren Vejrup Carlsen <svc at kb.dk> 
                      07/06/2010 10:55

Envoyé par : 
<netarchivesuite-devel-bounces at lists.gforge.statsbiblioteket.dk>

Veuillez répondre à 
<netarchivesuite-devel at lists.gforge.statsbiblioteket.dk>



Pour
"netarchivesuite-devel at lists.gforge.statsbiblioteket.dk" 
<netarchivesuite-devel at lists.gforge.statsbiblioteket.dk>
Copie
"sara.aubry at info.unicaen.fr" <sara.aubry at info.unicaen.fr>
Objet
Re: [Netarchivesuite-devel] Failed jobs and Jobtimeouttime parameter



Hi Sara.
The purpose of this parameter is to move the jobs from status STARTED to 
status FAILED
When we have waited for a week, after which we no longer expect the job to 
terminate. You might want to raise this number to 4*10080 (i.e. a month).

This was implemented in FR 1014 No good way to mark a non-reported-stopped 
job as FAILED or DONE.
The method 
dk.netarkivet.harvester.scheduler.HarvestScheduler.stopTimeoutJobs called 
from the scheduleJobs method in the same class handles this task.

The data from the harvesters will always be processed, but the status of 
the job will still be failed!

The class responsible for the processing of the feedback from the 
harvesters is the 
dk.netarkivet.harvester.scheduler.HarvestSchedulerMonitorServer and 
specifically the method:  processCrawlStatusMessage(CrawlStatusMessage 
cmsg)

Best regards

Søren

-----Oprindelig meddelelse-----
Fra: netarchivesuite-devel-bounces at lists.gforge.statsbiblioteket.dk 
[mailto:netarchivesuite-devel-bounces at lists.gforge.statsbiblioteket.dk] På 
vegne af sara.aubry at bnf.fr
Sendt: 4. juni 2010 19:00
Til: netarchivesuite-devel at lists.gforge.statsbiblioteket.dk
Cc: sara.aubry at info.unicaen.fr
Emne: [Netarchivesuite-devel] Failed jobs and Jobtimeouttime parameter

Hello everyone,

A week after the start of our second stage, our first 35 jobs were marked 
as Failed in the database although they are still up and running on the 
crawlers.

Nicolas went through the code and found a Jobtimeouttime parameter which 
is set on 10080 minutes.
What is the purpose of this parameter ?

He also read that it should not keep the HarvestController from updating 
the job status to Finished and the job details (start and end time, bytes 
and documents harvested for each domain...) as soon as the job is 
finished.
Can you confirmed this? We don't want to have inconsistent information in 
the job details. 

Thanks!

Sara





Avant d'imprimer, pensez ? l'environnement. 

_______________________________________________
Netarchivesuite-devel mailing list
Netarchivesuite-devel at lists.gforge.statsbiblioteket.dk
https://lists.gforge.statsbiblioteket.dk/mailman/listinfo/netarchivesuite-devel






Avant d'imprimer, pensez à l'environnement.   



More information about the Netarchivesuite-devel mailing list