[Netarchivesuite-users] Two concurrent jobs

[Bjarne Andersen]

You need to instantiate more harvester-controllers. One server can easily run more than one instance.

In netarchive.dk we have 5 servers - each running:
  - 3 HIGHPRIORITY instances
  - 1 LOWPRIORITY instances

The HIGHPRIORITY is for selective jobs and the LOWPRIORITY is for snap shot jobs (larger crawls)

If you do not intend to do snap shot harvesting I see no problem why you should not be able to run maybe 5 HIGHPRIORITY instances on the 
same server - this should allow for you to harvest 5 jobs simultaneously

best
-- 
Bjarne Andersen
Daily Manager - netarchive.dk

State & University Library
Universitetsparken
DK-8000 Aarhus C
T: +45 89462165 - C: +45 25662353
CVR/SE 10100682 - EAN 5798000791084
http://netarchive.dk

Peter Moser wrote:
> Hello!
> 
> I have another question. I have two harvest definitions running (both 
> for selective crawling). One definition contains a lot of 
> configurations, which take more than 6 hours to harvest. The definition 
> contains just one configuration with one url which is fetched in two 
> minutes. I want to harvest the first definition daily and the second 
> defnition every hour. The problem now ist that a job for the second 
> definition will postponed till the job of the first definition is 
> finished. What can I do that jobs for the 2nd definition will crawled 
> every hour, no matter how long the daily crawl of the first definition 
> will last?
> 
> _______________________________________________
> NetarchiveSuite-users mailing list
> NetarchiveSuite-users at lists.gforge.statsbiblioteket.dk
> https://lists.gforge.statsbiblioteket.dk/mailman/listinfo/netarchivesuite-users



-------------- next part --------------
A non-text attachment was scrubbed...
Name: netarkivet.vcf
Type: text/x-vcard
Size: 312 bytes
Desc: not available
URL: <http://ml.sbforge.org/pipermail/netarchivesuite-users/attachments/20080530/3f6f23b6/attachment-0002.vcf>