<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body dir="auto">
<div dir="ltr"><span></span></div>
<div dir="ltr">
<div dir="ltr"><span></span></div>
<div dir="ltr">
<div dir="ltr"><span></span></div>
<div dir="ltr">
<div dir="ltr"><span></span></div>
<div dir="ltr">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.E-postmall17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1"><font face="Calibri, sans-serif"><span style="font-size: 14.666666984558105px;">We currently use the NAS defaults for the following parameters:</span></font><br>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB"># How many multiples of last fetch elapsed time to wait before recontacting<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB"># same server. ;
</span><span style="color:red">Heritrix </span><span lang="EN-GB" style="color:red">default 5.0</span><span lang="EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB">disposition.delayFactor=1.0<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB"># Never wait more than this long, regardless of multiple;
<span style="color:red">Heritrix default 30000</span><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB">disposition.maxDelayMs=1000<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB"># Always wait this long after one completion before recontacting same<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB"># server, regardless of multiple;
<span style="color:red">Heritrix default 3000</span><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB">disposition.minDelayMs=300<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB"># Maximum per-host bandwidth usage;
<span style="color:red">Heritrix default 0 </span>(no limit)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:65.2pt">disposition.maxPerHostBandwidthUsageKbSec=500<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p>As you can see this differs from Heritrix's default values. How where they chosen?</o:p></p>
<p class="MsoNormal"><o:p><br>
</o:p></p>
<p class="MsoNormal"><o:p>The NAS defaults sometimes lead to more than two calls per second on the same server (checked in logs; 888 calls in 419 seconds in one case).</o:p></p>
<p class="MsoNormal"><o:p><br>
</o:p></p>
<p class="MsoNormal">Concerning</p>
<p class="MsoNormal"><o:p style="background-color: rgba(255, 255, 255, 0);"><font face="UICTFontTextStyleBody" size="3"> </font></o:p></p>
<p class="MsoNormal" style="margin-left: 65.2pt;"><font face="UICTFontTextStyleBody" size="3"><span style="background-color: rgba(255, 255, 255, 0);">parallelQueues=50 ; </span></font><span style="background-color: rgba(255, 255, 255, 0); font-family: UICTFontTextStyleBody; font-size: medium;">NAS
default 1</span></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"><o:p></o:p></p>
<p class="MsoNormal">we differ much from NAS (and also Heritrix?) default. In the template files from NAS it only says "TODO evaluate this default" (for the value 1).</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal">What values do you use? Have someone done any testing with different values? Have you been criticized by site owners for using too much webserver or network resources? What’s the pros an cons with many parallel queues?</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal">(Maybe something for todays <font face="UICTFontTextStyleBody" size="3"><span style="background-color: rgba(255, 255, 255, 0);">NetarchiveSuite tele-conference?)</span></font></p>
<p class="MsoNormal"><o:p><br>
</o:p></p>
<p class="MsoNormal"><span style="background-color: rgba(255, 255, 255, 0); font-family: UICTFontTextStyleBody; font-size: medium;">Best regards,</span></p>
<div><span style="background-color: rgba(255, 255, 255, 0);"><span lang="EN-GB">-----<br>
<br>
</span><span lang="EN-GB">Peter Svanberg</span><span lang="EN-GB"><br>
</span><span lang="EN-GB">Technical officer<br>
Digital Collections Department, Newspapers, Radio and Television Division</span><span lang="EN-GB"> <br>
<br>
</span><span lang="EN-GB">National Library of Sweden</span><span lang="EN-GB"><br>
</span><span lang="EN-GB"><a href="x-apple-data-detectors://1/1" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="address" x-apple-data-detectors-result="1/1" style="text-decoration-color: rgba(31, 73, 125, 0.258824);"></a><a href="x-apple-data-detectors://1/1" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="address" x-apple-data-detectors-result="1/1" style="text-decoration-color: rgba(0, 122, 255, 0.258824);">PO
Box 5039</a> <br>
<a href="x-apple-data-detectors://1/1" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="address" x-apple-data-detectors-result="1/1" style="text-decoration-color: rgba(0, 0, 0, 0.258824);">SE-104 51 Stockholm</a><br>
Visits: <a href="x-apple-data-detectors://2" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="address" x-apple-data-detectors-result="2" style="text-decoration-color: rgba(31, 73, 125, 0.258824);"></a><a href="x-apple-data-detectors://2" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="address" x-apple-data-detectors-result="2" style="text-decoration-color: rgba(0, 122, 255, 0.258824);">Karlavägen
100, Stockholm </a><br>
<a href="x-apple-data-detectors://2" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="address" x-apple-data-detectors-result="2" style="text-decoration-color: rgba(0, 0, 0, 0.258824);">Phone</a>: +46 10 709 </span><span lang="EN-GB">32 78</span><span lang="EN-GB"><br>
<br>
</span><span lang="EN-GB">E-mail</span><span lang="EN-GB">: </span><span lang="EN-GB"><a href="mailto:peter.svanberg@kb.se" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="3">peter.svanberg@kb.se</a></span><span lang="EN-GB"><br>
</span><span lang="EN-GB">Web</span><span lang="EN-GB">: </span><span lang="EN-GB"><a href="http://www.kb.se/" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="4">www.kb.se</a></span><span lang="EN-GB"><br>
</span></span></div>
<div style="-webkit-text-size-adjust: auto;"><br>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>