<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p>Hi Peter,</p>
<p><br>
</p>
<p>The crawl logs and all the other logs and reports are packaged up in a metadata warc-file which is uploaded to the archive along with the harvested data. When you find this, the hosts report is usually very informative because it shows how many objects and
bytes are harvested for each host.</p>
<p><br>
</p>
<p>/Colin</p>
<p><br>
</p>
<div id="Signature">
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<div>--</div>
<div>Colin Rosenthal PhD</div>
<div>Senior IT Consultant</div>
<div>Royal Danish Library (Aarhus)</div>
</div>
</div>
<br>
<br>
<div style="color: rgb(0, 0, 0);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> NetarchiveSuite-users <netarchivesuite-users-bounces@ml.sbforge.org> on behalf of Peter Svanberg <Peter.Svanberg@kb.se><br>
<b>Sent:</b> Monday, March 18, 2019 1:03 PM<br>
<b>To:</b> netarchivesuite-users@ml.sbforge.org<br>
<b>Subject:</b> Re: [Netarchivesuite-users] NAS/Heritrix doesn't obey byte limits</font>
<div> </div>
</div>
<div>
<div style="">
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-GB" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D">Thanks, Tue</span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-GB" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D"> </span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-GB" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D">The crawl.log file has one line for each URL which Heritrix has tried to fetch, I assume.</span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-GB" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D"> </span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-GB" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D">But that file (and surrounding files) seem to disappear when the job is done, right? So you have to monitor during harvest?</span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-GB" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D"> </span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-GB" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D">/Peter</span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-GB" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D"> </span></p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0cm 0cm 0cm">
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<b><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">Från:</span></b><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> NetarchiveSuite-users <netarchivesuite-users-bounces@ml.sbforge.org>
<b>För </b>Tue Hejlskov Larsen<br>
<b>Skickat:</b> den 18 mars 2019 11:10<br>
<b>Till:</b> netarchivesuite-users@ml.sbforge.org<br>
<b>Ämne:</b> Re: [Netarchivesuite-users] NAS/Heritrix doesn't obey byte limits</span></p>
</div>
</div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
</p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-US" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D">Hello Peter</span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-US" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D"> </span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-US" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D">In the H3 crawl.log.
</span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-US" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D"> </span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-US" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D">You can find it during the harvest f.x. here:</span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-US" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D">harvester_low_8090/307071_1552863418226/heritrix3/jobs/307071_1552863418226/logs/crawl.log</span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-US" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D">or in Netarchivesuite GUI under the job reports</span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-US" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D"> </span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-US" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D">It will tell you why </span><span lang="EN-US" style="font-size:11.0pt; font-family:Wingdings; color:#1F497D">J</span><span lang="EN-US" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D">
</span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-US" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D"> </span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-US" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D">Best regards</span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-US" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D">Tue</span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-US" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D"> </span></p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0cm 0cm 0cm">
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<b><span lang="DA" style="font-size:11.0pt; font-family:"Calibri",sans-serif">From:</span></b><span lang="DA" style="font-size:11.0pt; font-family:"Calibri",sans-serif"> NetarchiveSuite-users <<a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org" style="color: blue; text-decoration: underline;">netarchivesuite-users-bounces@ml.sbforge.org</a>>
<b>On Behalf Of </b>Peter Svanberg<br>
<b>Sent:</b> Monday, March 18, 2019 11:01 AM<br>
<b>To:</b> <a href="mailto:netarchivesuite-users@ml.sbforge.org" style="color: blue; text-decoration: underline;">
netarchivesuite-users@ml.sbforge.org</a><br>
<b>Subject:</b> Re: [Netarchivesuite-users] NAS/Heritrix doesn't obey byte limits</span></p>
</div>
</div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA"> </span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA">Hello Tue! </span></p>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA"> </span></p>
</div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA">Not much, the ones that I’ve seen. Which kind of log/filename should I check? Should we change log level? How? What should I look for in the log?<br>
<br>
(I’ve got sysadmin and programming experience but I’m new with this system. So I’m eager to learn more!)</span></p>
<div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA">/Peter</span></p>
</div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA"> </span></p>
</div>
</div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA"><br>
18 mars 2019 kl. 10:07 skrev Tue Hejlskov Larsen <<a href="mailto:tlr@kb.dk" style="color: blue; text-decoration: underline;">tlr@kb.dk</a>>:</span></p>
</div>
<blockquote style="margin-top:5.0pt; margin-bottom:5.0pt">
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D">Hello Peter</span><span lang="DA"></span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D"> </span><span lang="DA"></span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-US" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D">What does your crawllogs tell you?</span><span lang="DA"></span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-US" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D"> </span><span lang="DA"></span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-US" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D">Best regards</span><span lang="DA"></span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-US" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D">Tue</span><span lang="DA"></span></p>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-US" style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#1F497D"> </span><span lang="DA"></span></p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0cm 0cm 0cm">
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<b><span lang="DA" style="font-size:11.0pt; font-family:"Calibri",sans-serif">From:</span></b><span lang="DA" style="font-size:11.0pt; font-family:"Calibri",sans-serif"> NetarchiveSuite-users <<a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org" style="color: blue; text-decoration: underline;">netarchivesuite-users-bounces@ml.sbforge.org</a>>
<b>On Behalf Of </b>Peter Svanberg<br>
<b>Sent:</b> Monday, March 18, 2019 12:07 AM<br>
<b>To:</b> <a href="mailto:netarchivesuite-users@ml.sbforge.org" style="color: blue; text-decoration: underline;">
netarchivesuite-users@ml.sbforge.org</a><br>
<b>Subject:</b> [Netarchivesuite-users] NAS/Heritrix doesn't obey byte limits</span><span lang="DA"></span></p>
</div>
</div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA"> </span></p>
<div>
<div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA">Hello, NAS users and others!</span></p>
</div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA"> </span></p>
</div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA">We are experiencing a very strange behavior from NAS/Heritrix (see attached Excel file, with comments):</span></p>
</div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA"> </span></p>
</div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA">The harvests reports says ”Stopped due to … byte/object limit reached" at very different levels – sometimes much above (more than five times the limit), sometimes much below the limit. We fail to see any pattern in this, it seems more or less
random.</span></p>
</div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA"> </span></p>
</div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA">What are we doing wrong? Is it some error in the harvest template? (Attached below each table.)</span></p>
</div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA"> </span></p>
</div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA">Or, if it is some kind of bug, are there workarounds?</span></p>
</div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA"> </span></p>
</div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA">We would much appreciate any hints, as this is quite a problem for us, both for the on-going selective harvests and the upcoming big snapshot run!</span></p>
</div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA"> </span></p>
</div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA">(We are running version 5.4.2, I hope that it doesn’t affect this problem, as we can’t upgrade now.)</span></p>
</div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA"> </span></p>
</div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA">Best regards,</span></p>
</div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="EN-GB">-----<br>
<br>
Peter Svanberg<br>
Technical officer<br>
Digital Collections Department, Newspapers, Radio and Television Division <br>
<br>
National Library of Sweden<br>
<a href="" style="color: blue; text-decoration: underline;">PO Box 5039</a> <br>
<a href="" style="color: blue; text-decoration: underline;">SE-104 51 Stockholm</a><br>
Visits: <a href="" style="color: blue; text-decoration: underline;">Karlavägen 100, Stockholm </a><br>
<a href="" style="color: blue; text-decoration: underline;">Phone</a>: +46 10 709 32 78<br>
<br>
E-mail: <a href="mailto:peter.svanberg@kb.se" style="color: blue; text-decoration: underline;">peter.svanberg@kb.se</a><br>
Web: <a href="http://www.kb.se/" style="color: blue; text-decoration: underline;">www.kb.se</a></span><span lang="EN-GB" style="font-size:9.0pt; font-family:"Calibri",sans-serif; color:#1F497D"><br>
<br>
</span><span lang="DA"></span></p>
</div>
<div>
<p style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;">
<span lang="DA"> </span></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</body>
</html>