<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hello Peter</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
I have heard numbers from 1000 to 400000+. There have been also approaches to differentiate sites by size and put less bigger or more smaller sites to one job.<br>
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
When we were about to start our first broad crawl 4 years ago, we had a big unawareness. Finally we ended up with number 1 and so far we are happy. Now we are doing about 100.000<span id="��"></span> Heritrix jobs a year and mostly one seed in each job (plus
redirecters that we detect beforehand). We have 1G size limit for job but ca 90-95% of jobs end up earlier.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I don't know if Netarchivesuite is meant to work like this (we are using a self developed management system).<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<span id="ms-rterangepaste-start"></span>
<div>-- <br>
Peeter Rahuvarm<br>
National Library of Estonia<br>
</div>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>Saatja:</b> NetarchiveSuite-users <netarchivesuite-users-bounces@ml.sbforge.org> Peter Svanberg <Peter.Svanberg@kb.se> nimel<br>
<b>Saadetud:</b> reede, 24. jaanuar 2020 10:14<br>
<b>Adressaat:</b> netarchivesuite-users@ml.sbforge.org <netarchivesuite-users@ml.sbforge.org><br>
<b>Teema:</b> [Netarchivesuite-users] Number of domains/job recommendation?</font>
<div> </div>
</div>
<div dir="auto">Another short question: what do you think is a good level (*) on the number of domains per job on a broad crawl?
<div><br>
</div>
<div>We started a crawl being unaware of that we had a strange value on maxTotalSize. That lead to allmost 13000 jobs with on average 37 domains in each – not so good. (But it made us learn how to stop a broad crawl.)<br>
<br>
What should we choose?<br>
<div dir="ltr">
<div><br>
</div>
<div>Regards,</div>
<div><br>
</div>
<div>Peter Svanberg</div>
<div>National library of Sweden</div>
<div><br>
</div>
<div>(*) I would have liked to use the very Swedish word "lagom" here ... </div>
<div><a href="https://en.wikipedia.org/wiki/Lagom">https://en.wikipedia.org/wiki/Lagom</a></div>
<div><br>
</div>
</div>
</div>
</div>
</body>
</html>