<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.E-postmall19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="SV" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Yes, okay, I did look at such a warc file last week, forgot that now.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Thank you all for your quick answers, we’ll do tests right away!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">-----<br>
<br>
</span><span lang="EN-GB" style="font-size:11.0pt;font-family:"Arial",sans-serif;color:black">Peter Svanberg</span><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><br>
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><br>
</span><span lang="EN-GB" style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#1F497D">National Library of Sweden</span><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><br>
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#1F497D">Phone: +46 10 709
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">32 78</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><br>
<br>
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#1F497D">E-mail</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:#1F497D">:
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">peter.svanberg@kb.se</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:black"><br>
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#1F497D">Web</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:#1F497D">:
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#1F497D">www.kb.se</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><br>
<br>
</span><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Från:</span></b><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> NetarchiveSuite-users <netarchivesuite-users-bounces@ml.sbforge.org>
<b>För </b>Colin Samuel Rosenthal<br>
<b>Skickat:</b> den 18 mars 2019 13:12<br>
<b>Till:</b> netarchivesuite-users@ml.sbforge.org<br>
<b>Ämne:</b> Re: [Netarchivesuite-users] NAS/Heritrix doesn't obey byte limits<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<div id="divtagdefaultwrapper">
<p><span style="font-family:"Calibri",sans-serif;color:black">Hi Peter,<o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black">The crawl logs and all the other logs and reports are packaged up in a metadata warc-file which is uploaded to the archive along with the harvested data. When you find this, the hosts report is usually
very informative because it shows how many objects and bytes are harvested for each host.<o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black">/Colin<o:p></o:p></span></p>
<p><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
<div id="Signature">
<div name="divtagdefaultwrapper">
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">--<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Colin Rosenthal PhD<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Senior IT Consultant<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Royal Danish Library (Aarhus)<o:p></o:p></span></p>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
<div>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-family:"Calibri",sans-serif;color:black">
<hr size="2" width="98%" align="center">
</span></div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> NetarchiveSuite-users <<a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org">netarchivesuite-users-bounces@ml.sbforge.org</a>>
on behalf of Peter Svanberg <<a href="mailto:Peter.Svanberg@kb.se">Peter.Svanberg@kb.se</a>><br>
<b>Sent:</b> Monday, March 18, 2019 1:03 PM<br>
<b>To:</b> <a href="mailto:netarchivesuite-users@ml.sbforge.org">netarchivesuite-users@ml.sbforge.org</a><br>
<b>Subject:</b> Re: [Netarchivesuite-users] NAS/Heritrix doesn't obey byte limits</span><span style="font-family:"Calibri",sans-serif;color:black">
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks, Tue</span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The crawl.log file has one line for each URL which Heritrix has tried to fetch, I assume.</span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">But that file (and surrounding files) seem to disappear when the job is done, right? So you have to monitor during harvest?</span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">/Peter</span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Från:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> NetarchiveSuite-users <<a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org">netarchivesuite-users-bounces@ml.sbforge.org</a>>
<b>För </b>Tue Hejlskov Larsen<br>
<b>Skickat:</b> den 18 mars 2019 11:10<br>
<b>Till:</b> <a href="mailto:netarchivesuite-users@ml.sbforge.org">netarchivesuite-users@ml.sbforge.org</a><br>
<b>Ämne:</b> Re: [Netarchivesuite-users] NAS/Heritrix doesn't obey byte limits</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<p><span style="color:black"> <o:p></o:p></span></p>
<p><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hello Peter</span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">In the H3 crawl.log.
</span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">You can find it during the harvest f.x. here:</span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">harvester_low_8090/307071_1552863418226/heritrix3/jobs/307071_1552863418226/logs/crawl.log</span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">or in Netarchivesuite GUI under the job reports</span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">It will tell you why </span><span lang="EN-US" style="font-size:11.0pt;font-family:Wingdings;color:#1F497D">J</span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">
</span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Best regards</span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Tue</span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p><b><span lang="DA" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span lang="DA" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> NetarchiveSuite-users <<a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org">netarchivesuite-users-bounces@ml.sbforge.org</a>>
<b>On Behalf Of </b>Peter Svanberg<br>
<b>Sent:</b> Monday, March 18, 2019 11:01 AM<br>
<b>To:</b> <a href="mailto:netarchivesuite-users@ml.sbforge.org">netarchivesuite-users@ml.sbforge.org</a><br>
<b>Subject:</b> Re: [Netarchivesuite-users] NAS/Heritrix doesn't obey byte limits</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<p><span lang="DA" style="color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="DA" style="color:black">Hello Tue! </span><span style="color:black"><o:p></o:p></span></p>
<div>
<p><span lang="DA" style="color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p><span lang="DA" style="color:black">Not much, the ones that I’ve seen. Which kind of log/filename should I check? Should we change log level? How? What should I look for in the log?<br>
<br>
(I’ve got sysadmin and programming experience but I’m new with this system. So I’m eager to learn more!)</span><span style="color:black"><o:p></o:p></span></p>
<div>
<div>
<p><span lang="DA" style="color:black">/Peter</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p><span lang="DA" style="color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<div>
<p><span lang="DA" style="color:black"><br>
18 mars 2019 kl. 10:07 skrev Tue Hejlskov Larsen <<a href="mailto:tlr@kb.dk">tlr@kb.dk</a>>:</span><span style="color:black"><o:p></o:p></span></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p><span lang="DA" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hello Peter</span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="DA" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">What does your crawllogs tell you?</span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Best regards</span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Tue</span><span style="color:black"><o:p></o:p></span></p>
<p><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p><b><span lang="DA" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span lang="DA" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> NetarchiveSuite-users <<a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org">netarchivesuite-users-bounces@ml.sbforge.org</a>>
<b>On Behalf Of </b>Peter Svanberg<br>
<b>Sent:</b> Monday, March 18, 2019 12:07 AM<br>
<b>To:</b> <a href="mailto:netarchivesuite-users@ml.sbforge.org">netarchivesuite-users@ml.sbforge.org</a><br>
<b>Subject:</b> [Netarchivesuite-users] NAS/Heritrix doesn't obey byte limits</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<p><span lang="DA" style="color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<div>
<div>
<div>
<p><span lang="DA" style="color:black">Hello, NAS users and others!</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p><span lang="DA" style="color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p><span lang="DA" style="color:black">We are experiencing a very strange behavior from NAS/Heritrix (see attached Excel file, with comments):</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p><span lang="DA" style="color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p><span lang="DA" style="color:black">The harvests reports says ”Stopped due to … byte/object limit reached" at very different levels – sometimes much above (more than five times the limit), sometimes much below the limit. We fail to see any pattern in this,
it seems more or less random.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p><span lang="DA" style="color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p><span lang="DA" style="color:black">What are we doing wrong? Is it some error in the harvest template? (Attached below each table.)</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p><span lang="DA" style="color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p><span lang="DA" style="color:black">Or, if it is some kind of bug, are there workarounds?</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p><span lang="DA" style="color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p><span lang="DA" style="color:black">We would much appreciate any hints, as this is quite a problem for us, both for the on-going selective harvests and the upcoming big snapshot run!</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p><span lang="DA" style="color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p><span lang="DA" style="color:black">(We are running version 5.4.2, I hope that it doesn’t affect this problem, as we can’t upgrade now.)</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p><span lang="DA" style="color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p><span lang="DA" style="color:black">Best regards,</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p style="margin-bottom:12.0pt"><span lang="EN-GB" style="color:black">-----<br>
<br>
Peter Svanberg<br>
Technical officer<br>
Digital Collections Department, Newspapers, Radio and Television Division <br>
<br>
National Library of Sweden<br>
<span class="MsoHyperlink">PO Box 5039</span> <br>
<span class="MsoHyperlink">SE-104 51 Stockholm</span><br>
Visits: <span class="MsoHyperlink">Karlavägen 100, Stockholm </span><br>
<span class="MsoHyperlink">Phone</span>: +46 10 709 32 78<br>
<br>
E-mail: <a href="mailto:peter.svanberg@kb.se">peter.svanberg@kb.se</a><br>
Web: <a href="http://www.kb.se/">www.kb.se</a></span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p><span lang="DA" style="color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>