<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:UICTFontTextStyleBody;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.E-postmall18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.E-postmall19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="SV" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">No reaction on this subject?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">I would also like to know how you handle hops parameters:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">org.archive.modules.deciderules.TooManyHopsDecideRule<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">maxHops NAS default 20, we use 5<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">org.archive.modules.deciderules.TransclusionDecideRule<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">maxTransHops NAS default 2, we use 5<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">maxSpeculativeHops NAS default 1, which we use<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">(I studied the Heritrix source code to document exactly what these parameters mean – I can send that if you want.)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:SV">-----<br>
<br>
</span><span lang="EN-GB" style="font-family:"Arial",sans-serif;color:black;mso-fareast-language:SV">Peter Svanberg</span><span lang="EN-GB" style="color:#1F497D;mso-fareast-language:SV"><br>
</span><span lang="EN-GB" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:SV"><br>
</span><span lang="EN-GB" style="font-family:"Arial",sans-serif;color:#1F497D;mso-fareast-language:SV">National Library of Sweden</span><span lang="EN-GB" style="color:#1F497D;mso-fareast-language:SV"><br>
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#1F497D;mso-fareast-language:SV">Phone: +46 10 709
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:SV">32 78</span><span lang="EN-GB" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:SV"><br>
<br>
</span><span lang="EN-GB" style="color:#1F497D;mso-fareast-language:SV"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-GB" style="mso-fareast-language:SV">Från:</span></b><span lang="EN-GB" style="mso-fareast-language:SV"> NetarchiveSuite-users <netarchivesuite-users-bounces@ml.sbforge.org>
<b>För </b>Peter Svanberg<br>
<b>Skickat:</b> den 9 april 2019 09:51<br>
<b>Till:</b> netarchivesuite-users@ml.sbforge.org<br>
<b>Ämne:</b> [Netarchivesuite-users] NAS/Heritrix and its webserver and network impact – ”politeness”<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="mso-fareast-language:SV">We currently use the NAS defaults for the following parameters:</span><span style="font-size:12.0pt;font-family:"Times New Roman",serif;mso-fareast-language:SV"><o:p></o:p></span></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB"># How many multiples of last fetch elapsed time to wait before recontacting</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB"># same server. ;
</span><span style="color:red">Heritrix </span><span lang="EN-GB" style="color:red">default 5.0</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB">disposition.delayFactor=1.0</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB"># Never wait more than this long, regardless of multiple;
<span style="color:red">Heritrix default 30000</span></span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB">disposition.maxDelayMs=1000</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB"># Always wait this long after one completion before recontacting same</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB"># server, regardless of multiple;
<span style="color:red">Heritrix default 3000</span></span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB">disposition.minDelayMs=300</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB"># Maximum per-host bandwidth usage;
<span style="color:red">Heritrix default 0 </span>(no limit)</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:65.2pt">disposition.maxPerHostBandwidthUsageKbSec=500<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">As you can see this differs from Heritrix's default values. How where they chosen?<o:p></o:p></p>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<p class="MsoNormal">The NAS defaults sometimes lead to more than two calls per second on the same server (checked in logs; 888 calls in 419 seconds in one case).<o:p></o:p></p>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<p class="MsoNormal">Concerning<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:UICTFontTextStyleBody"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span style="font-size:12.0pt;font-family:UICTFontTextStyleBody">parallelQueues=50 ; </span><span style="font-size:13.5pt;font-family:UICTFontTextStyleBody">NAS default 1</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">we differ much from NAS (and also Heritrix?) default. In the template files from NAS it only says "TODO evaluate this default" (for the value 1).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">What values do you use? Have someone done any testing with different values? Have you been criticized by site owners for using too much webserver or network resources? What’s the pros an cons with many parallel queues?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">(Maybe something for todays <span style="font-size:12.0pt;font-family:UICTFontTextStyleBody">NetarchiveSuite tele-conference?)</span><o:p></o:p></p>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:UICTFontTextStyleBody">Best regards,</span><o:p></o:p></p>
<div>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:12.0pt;font-family:"Times New Roman",serif;mso-fareast-language:SV">-----<br>
<br>
Peter Svanberg<br>
Technical officer<br>
Digital Collections Department, Newspapers, Radio and Television Division <br>
<br>
National Library of Sweden<br>
<a href="x-apple-data-detectors://1/1">PO Box 5039</a> <br>
<a href="x-apple-data-detectors://1/1">SE-104 51 Stockholm</a><br>
Visits: <a href="x-apple-data-detectors://2">Karlavägen 100, Stockholm </a><br>
<a href="x-apple-data-detectors://2">Phone</a>: +46 10 709 32 78<br>
<br>
E-mail: <a href="mailto:peter.svanberg@kb.se">peter.svanberg@kb.se</a><br>
Web: <a href="http://www.kb.se/">www.kb.se</a><br>
<br>
</span><span style="font-size:12.0pt;font-family:"Times New Roman",serif;mso-fareast-language:SV"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif;mso-fareast-language:SV"><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>