<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML - förformaterad Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTML-frformateradChar
{mso-style-name:"HTML - förformaterad Char";
mso-style-priority:99;
mso-style-link:"HTML - förformaterad";
font-family:Consolas;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.E-postmall20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.E-postmall21
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.E-postmall22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.E-postmall23
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
p.FormateretHTML, li.FormateretHTML, div.FormateretHTML
{mso-style-name:"Formateret HTML";
mso-style-link:"Formateret HTML Tegn";
margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.FormateretHTMLTegn
{mso-style-name:"Formateret HTML Tegn";
mso-style-priority:99;
mso-style-link:"Formateret HTML";
font-family:"Courier New";}
span.E-postmall26
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:3.0cm 2.0cm 3.0cm 2.0cm;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="SV" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Thank you so much!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">I will look more closely at these files and compare with ours. But my first question is: why do you run with parallelQueues
= 1? We use 50!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">I have previously found the following:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Found a response from the mailing list: Setting up parallel queries in the queueSignmentPolicy
bean.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">parallel queues: the default value (and historical behavior) is '1'. If instead, N, all URIs
that previously entered the same single-sex queue will go into N related queues (via a consistent hash mapping of the path portion of the URL). Each queue is considered separately for traditional courtesy based on one-to-one connections and snooze delays-mid-downloads
- so N queues mean that N downloads might go against one site at a time. Thus, it should only be used in an overlay setting, applied to sites that can handle multiple connections well.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">------<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">https://wiki.searchtechnologies.com/index.php/Using_a_Custom_Heritrix_Configuration_File_(Aspire_2):<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:65.2pt"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Remember to be careful about how many connections you should use per hostname, as it can cause
problems with the site searched and it can be considered an attack by the web administrator.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">But that doesn't seem to be correct now? When I look at crawl.log, and sort on the field "started", the "politeness"
parameters seem to be followed.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">But parallelq maybe doesn’ t have any performance significance? I suppose Heritrix makes a lot of parallel calls and downloads
even with parallelq = 1, but everything is in the same queue? Could someone explain?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">With kind regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">-----<br>
<br>
</span><span lang="EN-GB" style="font-size:11.0pt;font-family:"Arial",sans-serif;color:black">Peter Svanberg</span><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><br>
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><br>
</span><span lang="EN-GB" style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#1F497D">National Library of Sweden</span><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><br>
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#1F497D">Phone: +46 10 709
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">32 78</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><br>
<br>
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#1F497D">E-mail</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:#1F497D">:
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">peter.svanberg@kb.se</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:black"><br>
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#1F497D">Web</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:#1F497D">:
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#1F497D">www.kb.se</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><br>
<br>
</span><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Från:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> NetarchiveSuite-users <netarchivesuite-users-bounces@ml.sbforge.org>
<b>För </b>Tue Hejlskov Larsen<br>
<b>Skickat:</b> den 26 april 2019 13:38<br>
<b>Till:</b> netarchivesuite-users@ml.sbforge.org<br>
<b>Ämne:</b> Re: [Netarchivesuite-users] Strange slow non-existing-domain behavior<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="DA" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Hello Peter<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DA" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">We had also troubles last year with timouts regarding crawlertrap regex filters which hang infinite.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">You can find all our actual timeout settings below in our selective harvester settings file and in our default_orderxml<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Best regards<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Tue<o:p></o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Here our harvester settings file:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"> :<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"> :<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="DA" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="DA" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> NetarchiveSuite-users <netarchivesuite-users-bounces@ml.sbforge.org>
<b>On Behalf Of </b>Peter Svanberg<br>
<b>Sent:</b> Friday, April 26, 2019 12:46 PM<br>
<b>To:</b> netarchivesuite-users@ml.sbforge.org<br>
<b>Subject:</b> Re: [Netarchivesuite-users] Strange slow non-existing-domain behavior<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="DA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Hmm, I realize I have two parameters having 300 second values:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">
</span><span lang="EN-GB" style="font-size:10.0pt;font-family:"Courier New";color:black">frontier.retryDelaySeconds=300<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">
</span><span lang="EN-GB" style="font-size:10.0pt;font-family:"Courier New";color:black">frontier.snoozeLongMs=300000</span><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">But I don’t see any “,2t” or “,3t” in these passages and the harvester doesn’t do anything else, so why snooze?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">And in another job I get 10 seconds pauses. And no “Details and Actions” page in GUI … (Not a good NAS day.
</span><span lang="EN-GB" style="font-size:11.0pt;font-family:Wingdings;color:#1F497D;mso-fareast-language:EN-US">L</span><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">But the weather is quite nice!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">/Peter<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Från:</span></b><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> NetarchiveSuite-users <<a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org">netarchivesuite-users-bounces@ml.sbforge.org</a>>
<b>För </b>Peter Svanberg<br>
<b>Skickat:</b> den 26 april 2019 11:02<br>
<b>Till:</b> <a href="mailto:netarchivesuite-users@ml.sbforge.org">netarchivesuite-users@ml.sbforge.org</a><br>
<b>Ämne:</b> Re: [Netarchivesuite-users] Strange slow non-existing-domain behavior<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Now I discover a simular behavior, but with 404 status, 300 second wait and no problem with the domain (quick answer with
wget). Is it the same issue, solved in 5.5?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">2019-04-26T08:14:38.108Z 404 449
</span><a href="http://adcove.se/contactform.error_changefontsize_no_size"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">http://adcove.se/contactform.error_changefontsize_no_size</span></a><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">
REX </span><a href="http://55b558c7-resources.builder.misssite.com/ea01a9c/en/translations.js?sections=w"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">http://55b558c7-resources.builder.misssite.com/ea01a9c/en/translations.js?sections=w</span></a><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">idgets,mobile,shared_views,shared_components,cookie text/html #032 20190426081438022+85 sha1:7BBNH63Q5ARINIOLBLMA6MVZODMSSUSD
</span><a href="http://www.adcove.se"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">http://www.adcove.se</span></a><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">
content-size:828<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">2019-04-26T08:19:38.243Z 404 449
</span><a href="http://adcove.se/contactform.error_changeFormTitle_no_value"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">http://adcove.se/contactform.error_changeFormTitle_no_value</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">
</span><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">REX
</span><a href="http://55b558c7-resources.builder.misssite.com/ea01a9c/en/translations.js?sections"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">http://55b558c7-resources.builder.misssite.com/ea01a9c/en/translations.js?sections</span></a><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">=widgets,mobile,shared_views,shared_components,cookie text/html #032 20190426081938163+80 sha1:7BBNH63Q5ARINIOLBLMA6MVZODMSSUSD
</span><a href="http://www.adcove.se"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">http://www.adcove.se</span></a><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">
content-size:828<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">2019-04-26T08:24:38.389Z 404 449
</span><a href="http://adcove.se/contactform.error_changegoallink_no_source"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">http://adcove.se/contactform.error_changegoallink_no_source</span></a><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">
REX </span><a href="http://55b558c7-resources.builder.misssite.com/ea01a9c/en/translations.js?sections"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">http://55b558c7-resources.builder.misssite.com/ea01a9c/en/translations.js?sections</span></a><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">=widgets,mobile,shared_views,shared_components,cookie text/html #032 20190426082438299+90 sha1:7BBNH63Q5ARINIOLBLMA6MVZODMSSUSD
</span><a href="http://www.adcove.se"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">http://www.adcove.se</span></a><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">
content-size:828<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">-----<br>
<br>
</span><span lang="EN-GB" style="font-size:11.0pt;font-family:"Arial",sans-serif;color:black">Peter Svanberg</span><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><br>
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><br>
</span><span lang="EN-GB" style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#1F497D">National Library of Sweden</span><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><br>
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#1F497D">Phone: +46 10 709
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">32 78</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><br>
<br>
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#1F497D">E-mail</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:#1F497D">:
</span><a href="mailto:peter.svanberg@kb.se"><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif">peter.svanberg@kb.se</span></a><span lang="EN-GB" style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:black"><br>
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#1F497D">Web</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:#1F497D">:
</span><a href="http://www.kb.se"><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif">www.kb.se</span></a><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Från:</span></b><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> NetarchiveSuite-users <</span><a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif">netarchivesuite-users-bounces@ml.sbforge.org</span></a><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif">>
<b>För </b>Tue Hejlskov Larsen<br>
<b>Skickat:</b> den 21 mars 2019 06:16<br>
<b>Till:</b> </span><a href="mailto:netarchivesuite-users@ml.sbforge.org"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif">netarchivesuite-users@ml.sbforge.org</span></a><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif"><br>
<b>Ämne:</b> Re: [Netarchivesuite-users] Strange slow non-existing-domain behavior<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="DA" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Hi Peter<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DA" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">We had also troubles with dns spam in 5.4.2.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Yes, it is fixed in 5.5.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Best regards<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Tue<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="DA" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="DA" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> NetarchiveSuite-users <</span><a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org"><span lang="DA" style="font-size:11.0pt;font-family:"Calibri",sans-serif">netarchivesuite-users-bounces@ml.sbforge.org</span></a><span lang="DA" style="font-size:11.0pt;font-family:"Calibri",sans-serif">>
<b>On Behalf Of </b>Peter Svanberg<br>
<b>Sent:</b> Wednesday, March 20, 2019 11:33 PM<br>
<b>To:</b> </span><a href="mailto:netarchivesuite-users@ml.sbforge.org"><span lang="DA" style="font-size:11.0pt;font-family:"Calibri",sans-serif">netarchivesuite-users@ml.sbforge.org</span></a><span lang="DA" style="font-size:11.0pt;font-family:"Calibri",sans-serif"><br>
<b>Subject:</b> [Netarchivesuite-users] Strange slow non-existing-domain behavior<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="DA"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="DA">Hello again!<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">Spurred by your previous problem-solving answers, I continue.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">Strange Heritrix behavior: Do dns lookup, which fails. Report that with an -6 line. Then 10 minutes pause. Then a new dns lookup and so on.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">What happens during the pause? Waiting for dns lookup in 600 seconds? Trying the request despite the failed lookup?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">(Maybe one of the bugs fixed in 5.5?)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">Log and template below.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">Best regards,<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-GB">-----<br>
<br>
Peter Svanberg<br>
Technical officer<br>
Digital Collections Department, Newspapers, Radio and Television Division <br>
<br>
National Library of Sweden<br>
</span><a href="x-apple-data-detectors://1/1"><span lang="EN-GB">PO Box 5039</span></a><span lang="EN-GB"> <br>
</span><a href="x-apple-data-detectors://1/1"><span lang="EN-GB">SE-104 51 Stockholm</span></a><span lang="EN-GB"><br>
Visits: </span><a href="x-apple-data-detectors://2"><span lang="EN-GB">Karlavägen 100, Stockholm </span></a><span lang="EN-GB"><br>
</span><a href="x-apple-data-detectors://2"><span lang="EN-GB">Phone</span></a><span lang="EN-GB">: +46 10 709 32 78<br>
<br>
E-mail: </span><a href="mailto:peter.svanberg@kb.se"><span lang="EN-GB">peter.svanberg@kb.se</span></a><span lang="EN-GB"><br>
Web: </span><a href="http://www.kb.se/"><span lang="EN-GB">www.kb.se</span></a><span lang="DA"><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="DA"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">crawl log:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">2019-03-20T21:48:42.119Z -6 - </span>
<a href="http://lookbackvideo7-a.akamaihd.net/"><span lang="DA">http://lookbackvideo7-a.akamaihd.net/</span></a><span lang="DA"> RRX
</span><a href="https://www.facebook.com/"><span lang="DA">https://www.facebook.com/</span></a><span lang="DA"> unknown #033 - -
</span><a href="http://www.fbcdn.net"><span lang="DA">http://www.fbcdn.net</span></a>
<span lang="DA">2t<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">2019-03-20T21:48:41.164Z -1 - dns:lookbackvideo7-</span><a href="http://a.akamaihd.net"><span lang="DA">a.akamaihd.net</span></a><span lang="DA"> RRXP
</span><a href="http://lookbackvideo7-a.akamaihd.net/"><span lang="DA">http://lookbackvideo7-a.akamaihd.net/</span></a><span lang="DA"> text/dns #047 20190320214841119+45 -
</span><a href="http://www.fbcdn.net"><span lang="DA">http://www.fbcdn.net</span></a><span lang="DA"> 3t<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">2019-03-20T21:38:41.006Z -6 - </span>
<a href="http://lookbackvideo6-a.akamaihd.net/"><span lang="DA">http://lookbackvideo6-a.akamaihd.net/</span></a><span lang="DA"> RRX
</span><a href="https://www.facebook.com/"><span lang="DA">https://www.facebook.com/</span></a><span lang="DA"> unknown #024 - -
</span><a href="http://www.fbcdn.net"><span lang="DA">http://www.fbcdn.net</span></a><span lang="DA"> 2t<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">2019-03-20T21:38:40.063Z -1 - dns:lookbackvideo6-</span><a href="http://a.akamaihd.net"><span lang="DA">a.akamaihd.net</span></a><span lang="DA"> RRXP
</span><a href="http://lookbackvideo6-a.akamaihd.net/"><span lang="DA">http://lookbackvideo6-a.akamaihd.net/</span></a><span lang="DA"> text/dns #026 20190320213840006+56 -
</span><a href="http://www.fbcdn.net"><span lang="DA">http://www.fbcdn.net</span></a><span lang="DA"> 3t<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">2019-03-20T21:28:39.896Z -6 - </span>
<a href="http://lookbackvideo5-a.akamaihd.net/"><span lang="DA">http://lookbackvideo5-a.akamaihd.net/</span></a><span lang="DA"> RRX
</span><a href="https://www.facebook.com/"><span lang="DA">https://www.facebook.com/</span></a><span lang="DA"> unknown #045 - -
</span><a href="http://www.fbcdn.net"><span lang="DA">http://www.fbcdn.net</span></a><span lang="DA"> 2t<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">2019-03-20T21:28:38.942Z -1 - dns:lookbackvideo5-</span><a href="http://a.akamaihd.net"><span lang="DA">a.akamaihd.net</span></a><span lang="DA"> RRXP
</span><a href="http://lookbackvideo5-a.a"><span lang="DA">http://lookbackvideo5-a.a</span></a><span lang="DA"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">template:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="DA">fetchDns.enabled=true<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">fetchDns.acceptNonDnsResolves=false<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">fetchDns.digestContent=true<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">fetchDns.digestAlgorithm=sha1<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">fetchHttp.enabled=true<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">fetchHttp.timeoutSeconds=1200<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">fetchHttp.soTimeoutMs=20000<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">fetchHttp.maxFetchKBSec=0<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">fetchHttp.maxLengthBytes=0<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">fetchHttp.ignoreCookies=false<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">fetchHttp.sslTrustLevel=OPEN<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">fetchHttp.defaultEncoding=UTF-8<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">fetchHttp.digestContent=true<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">fetchHttp.digestAlgorithm=sha1<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">fetchHttp.sendIfModifiedSince=true<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">fetchHttp.sendIfNoneMatch=true<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">fetchHttp.sendConnectionClose=true<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">fetchHttp.sendReferer=true<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA">fetchHttp.sendRange=false<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span lang="DA"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="DA"><o:p> </o:p></span></p>
</div>
</div>
</div>
</body>
</html>