<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.E-postmall19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.E-postmall20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.E-postmall21
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.E-postmall22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.E-postmall23
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.E-postmall24
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.E-postmall25
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.E-postmall26
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.E-postmall27
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:971866029;
mso-list-type:hybrid;
mso-list-template-ids:757886654 67502097 67502105 67502107 67502095 67502105 67502107 67502095 67502105 67502107;}
@list l0:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:32.2pt;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1
{mso-list-id:1786970620;
mso-list-type:hybrid;
mso-list-template-ids:-2046803034 69009409 69009411 69009413 69009409 69009411 69009413 69009409 69009411 69009413;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="SV" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">Yes, for each domain the limit value MIN(the domains individual limit, harvest limit) is used, correct?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">Sql extract, before or after step 2? (I suppose “before” as you wrote in previous e-mail.)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">My interpretation of your description: You administer individual limits on each/some domain(s), decided from earlier crawls, to avoid run-away. So in step 2 the harvest limit is 16 GByte but some
domains have lover domain limits.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l1 level1 lfo3"><![if !supportLists]><span lang="EN-GB" style="font-family:Symbol;color:#1F497D"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-GB" style="color:#1F497D">I suppose you then have algorithms for the limit increase/decrease/move-to-mega and implement those changes directly in the database via a script/program?<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l1 level1 lfo3"><![if !supportLists]><span lang="EN-GB" style="font-family:Symbol;color:#1F497D"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-GB" style="color:#1F497D">How much room for growth do you use?<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l1 level1 lfo3"><![if !supportLists]><span lang="EN-GB" style="font-family:Symbol;color:#1F497D"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-GB" style="color:#1F497D">How do you handle domains which did reach their individual limit? Analyse why?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">Thankful regards<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">Peter<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-GB" style="mso-fareast-language:SV">Från:</span></b><span lang="EN-GB" style="mso-fareast-language:SV"> NetarchiveSuite-users <netarchivesuite-users-bounces@ml.sbforge.org>
<b>För </b>Tue Hejlskov Larsen<br>
<b>Skickat:</b> den 26 juni 2</span><span style="mso-fareast-language:SV">019 20:47<br>
<b>Till:</b> netarchivesuite-users@ml.sbforge.org<br>
<b>Ämne:</b> Re: [Netarchivesuite-users] Your URI/sec and KB/sec figures?; Individual new limits<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">You have 2 levels of bytelimits – on the domain and harvest definition level.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">So when you e.g. run a broad crawl step 1 with a
<b>harvest</b> “byte limit” level on 50 MB it means that even though the <b>domain</b> byte limit is on 200 GB for a domain it will stop after 50 MB and if the domain byte limit is on 5 MB it will stop after 5 MB and the status will be set to “byte limit
reached”. But if the domain completes within the harvest 50 MB limit it will be set to complete.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">We do the sql extract after step 2 to find all the domains which hit the “byte limit” on 2, 4, 6, 8, 10, 12, 14, 16 GB and deside which ones we will increase (or lower) the domain byte limit for.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">We decide also which huge domains (> 16 GB) to move to the selective broad harvest definitions “mega_big_sites” or other selective broad harvest crawls with special harvest setup.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">It’s a way to avoid that jobs runs “away” in crawler traps or harvesting the whole www by a mistake and to increase the broadcrawl harvesting in a controlled way.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Best regards<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Tue<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="DA" style="mso-fareast-language:DA">From:</span></b><span lang="DA" style="mso-fareast-language:DA"> NetarchiveSuite-users <<a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org">netarchivesuite-users-bounces@ml.sbforge.org</a>>
<b>On Behalf Of </b>Peter Svanberg<br>
<b>Sent:</b> Wednesday, June 26, 2019 7:09 PM<br>
<b>To:</b> <a href="mailto:netarchivesuite-users@ml.sbforge.org">netarchivesuite-users@ml.sbforge.org</a><br>
<b>Subject:</b> Re: [Netarchivesuite-users] Your URI/sec and KB/sec figures?; Individual new limits<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="DA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">Thanks for you patience!</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">My aim with URL statistics was to get some hints on what speed is needed (leading to request for more resources from the IT department).</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">(I split in several follow up e-mails with different subjects.)</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">Why do you have to do sql handling to individually set new max limits? What is the difference between that and running step 2 with a new global limit?</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">Regards!</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">Peter</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="mso-fareast-language:SV">Från:</span></b><span style="mso-fareast-language:SV"> NetarchiveSuite-users <<a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org">netarchivesuite-users-bounces@ml.sbforge.org</a>>
<b>För </b>Tue Hejlskov Larsen<br>
<b>Skickat:</b> den 24 juni 2019 21:26<br>
<b>Till:</b> <a href="mailto:netarchivesuite-users@ml.sbforge.org">netarchivesuite-users@ml.sbforge.org</a><br>
<b>Ämne:</b> Re: [Netarchivesuite-users] Your URI/sec and KB/sec figures?</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">See my TLR comments below</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="DA" style="mso-fareast-language:DA">From:</span></b><span lang="DA" style="mso-fareast-language:DA"> NetarchiveSuite-users <<a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org">netarchivesuite-users-bounces@ml.sbforge.org</a>>
<b>On Behalf Of </b>Peter Svanberg<br>
<b>Sent:</b> Monday, June 24, 2019 5:05 PM<br>
<b>To:</b> <a href="mailto:netarchivesuite-users@ml.sbforge.org">netarchivesuite-users@ml.sbforge.org</a><br>
<b>Subject:</b> Re: [Netarchivesuite-users] Your URI/sec and KB/sec figures?</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><span lang="DA"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">I continue with my curiosity, I hope it’s OK.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">You mean about 70 TByte fetched in about 100–120 days? (Or was the selective “mega” included in 70?)</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">TLR>>> yes we run both broadcrawl step 1 and step 2 and selective broad crawl jobs in parallel, because we have 2 dedicated harvester farms in AAR and CPH.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">And 20 TByte is what ends up being to stored in the archive?
</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">TLR>>> yes</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">Approximately how many URI:s does this correspond to – before and after deduplication?</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">TLR >>> We are talking million/billion urls, - just to mention - we have 1.1 billion 5003 “byte limit reached” in step 2. All urls are recorded in the crawllogs also the deduplicated are annotated
there. </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">So you need to specify which type of return code urls you want numbers for
</span><span lang="EN-GB" style="font-family:Wingdings;color:#1F497D">J</span><span lang="EN-GB" style="color:#1F497D">. We do some daily statistics for some of the return codes. It is really huge numbers!</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">TLR>>>Deduplication gives appr. 40-50 % and gz 40-50 %. </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">We have been advised to do the broad crawl in several steps with increasing max thresholds, is that what you do in 2 steps? With what thresholds different (and what levels)?</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">TLR>>> yes we have following steps:</span><o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:32.2pt;text-indent:-18.0pt;mso-list:l0 level1 lfo2">
<![if !supportLists]><span style="mso-list:Ignore">1)<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]><span lang="EN-GB" style="color:#1F497D">Step 1: 50 MB byte limit : all domains in the jobs database : duration 1-2 weeks</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">We do some sqlextraction from the jobdatabase and increase the max bytelimit per domain for some 10- 20.000 domains before each new step 2 broad crawl.</span><o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:32.2pt;text-indent:-18.0pt;mso-list:l0 level1 lfo2">
<![if !supportLists]><span style="mso-list:Ignore">2)<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]><span lang="EN-GB" style="color:#1F497D">Step 2: 16 GB byte limit: all domains which have hit 50 MB limit: duration 1-2 months.</span><o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:32.2pt;text-indent:-18.0pt;mso-list:l0 level1 lfo2">
<![if !supportLists]><span style="mso-list:Ignore">3)<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]><span lang="EN-GB" style="color:#1F497D">And we run about 3-4 big selective broad crawl harvests in parallel - with different (huge) domains - taken out of the step 2 broad crawl. They run 3 – 6 weeks each and each harvest creates
about 10 -20 jobs running in parallel in AAR together with the normal daily selective harvests.
</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">So we are using the most of our harvester capacity in long periods during the “broad crawl”.
</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">
</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">And with reference to the subject line … what is your typical URI/sec and KB/sec figures in a single job?</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">TLR>>> I have not looked into that ( we have between 50-90 different jobs /day) , because we are using the NAS std. setup ( you have got a copy of that earlier).
</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">The main problem was earlier - domains which blocked/throttled us and next that our capacity agreements with the biggest webhotels where too low. The biggest one ( a .be company) have about 50-75
% of all .dk domains.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">After we have increased our max concurrent requests agreements with them to 40MB/sec for our harvester ip ranges in AAR and CPH and upgraded to NAS 5.5 we have no big performance issues anymore.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-GB" style="color:#1F497D">Med venlig hilsen</span><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:SV"><br>
<br>
</span><span style="font-family:"Arial",sans-serif;color:black;mso-fareast-language:SV">Peter Svanberg</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="mso-fareast-language:SV">Från:</span></b><span style="mso-fareast-language:SV"> NetarchiveSuite-users <<a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org">netarchivesuite-users-bounces@ml.sbforge.org</a>>
<b>För </b>Tue Hejlskov Larsen<br>
<b>Skickat:</b> den 24 juni 2019 15:16<br>
<b>Till:</b> <a href="mailto:netarchivesuite-users@ml.sbforge.org">netarchivesuite-users@ml.sbforge.org</a><br>
<b>Ämne:</b> Re: [Netarchivesuite-users] Your URI/sec and KB/sec figures?</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">The 70 TB is based on NAS GUI/crawllog numbers – and before deduplication and gz – about 20 TB gz uploaded.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">“A broadcrawl” runs about 2 - 2 1/2 months - and we do some job follow up during step 2 ( this part takes about 1 ½ month) and the selective broad crawl job “mega big sites” (runs for a month or
more and here we use another queue assign policy and much lower delays and harvest only domains which can take a huge number of crawling requests!) </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Best regards</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Tue</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="DA" style="mso-fareast-language:DA">From:</span></b><span lang="DA" style="mso-fareast-language:DA"> NetarchiveSuite-users <<a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org">netarchivesuite-users-bounces@ml.sbforge.org</a>>
<b>On Behalf Of </b>Peter Svanberg<br>
<b>Sent:</b> Monday, June 24, 2019 2:54 PM<br>
<b>To:</b> <a href="mailto:netarchivesuite-users@ml.sbforge.org">netarchivesuite-users@ml.sbforge.org</a><br>
<b>Subject:</b> Re: [Netarchivesuite-users] Your URI/sec and KB/sec figures?</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><span lang="DA"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">Thank you Tue, this is very interesting information!</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">About 70 TB in how many days?</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">You emphasize “harvested” – do you aim at that more data is downloaded but not archived (sorted out duplicates/irrelevant?)?</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">I’ll return when I have gathered corresponding info on our environment.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">Regards,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:SV">-----<br>
<br>
</span><span lang="EN-GB" style="font-family:"Arial",sans-serif;color:black;mso-fareast-language:SV">Peter Svanberg</span><span lang="EN-GB" style="color:#1F497D;mso-fareast-language:SV"><br>
</span><span lang="EN-GB" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:SV"><br>
</span><span lang="EN-GB" style="font-family:"Arial",sans-serif;color:#1F497D;mso-fareast-language:SV">National Library of Sweden</span><span lang="EN-GB" style="color:#1F497D;mso-fareast-language:SV"><br>
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#1F497D;mso-fareast-language:SV">Phone: +46 10 709
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:SV">32 78</span><span lang="EN-GB" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:SV"><br>
<br>
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#1F497D;mso-fareast-language:SV">E-mail</span><span lang="EN-GB" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:SV">:
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:SV"><a href="mailto:peter.svanberg@kb.se">peter.svanberg@kb.se</a></span><span lang="EN-GB" style="font-size:9.0pt;color:black;mso-fareast-language:SV"><br>
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#1F497D;mso-fareast-language:SV">Web</span><span lang="EN-GB" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:SV">:
</span><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#1F497D;mso-fareast-language:SV"><a href="http://www.kb.se">www.kb.se</a></span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="mso-fareast-language:SV">Från:</span></b><span style="mso-fareast-language:SV"> NetarchiveSuite-users <<a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org">netarchivesuite-users-bounces@ml.sbforge.org</a>>
<b>För </b>Tue Hejlskov Larsen<br>
<b>Skickat:</b> den 24 juni 2019 12:22<br>
<b>Till:</b> <a href="mailto:netarchivesuite-users@ml.sbforge.org">netarchivesuite-users@ml.sbforge.org</a><br>
<b>Ämne:</b> Re: [Netarchivesuite-users] Your URI/sec and KB/sec figures?</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Hi Peter</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">We have currently only minor performance issues during harvesting. We have almost finished with our 2. broadcrawl this year – it will end up between 60-70 TB
<i>harvested</i> pages.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Our harvesting capacity is 90-100 Heritrix harvesters including some virtual Umbra harvesters…</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">We are using physical servers for the broadcrawl harvesters and virtual servers for selective harvesters.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">The 5 physical servers have each:</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">32 G MEM, 24 CPU’s, 4 TB local storage</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">The 5 Virtual servers using NFS:</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">20 G RAM, 8 CPU’s and 3 TB NFS storage</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">On each server we have between 8-10 Heritrix instances running – withdrawn the Umbra harvesters which only have one per server.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Between the harvester and the www we have a firewall and throttling firewall agreements with about 5 webhotels, because they blocked/throttled our harvesters.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Best regards</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Tue</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="DA" style="mso-fareast-language:DA">From:</span></b><span lang="DA" style="mso-fareast-language:DA"> NetarchiveSuite-users <<a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org">netarchivesuite-users-bounces@ml.sbforge.org</a>>
<b>On Behalf Of </b>Peter Svanberg<br>
<b>Sent:</b> Monday, June 24, 2019 11:39 AM<br>
<b>To:</b> <a href="mailto:netarchivesuite-users@ml.sbforge.org">netarchivesuite-users@ml.sbforge.org</a><br>
<b>Subject:</b> [Netarchivesuite-users] Your URI/sec and KB/sec figures?</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><span lang="DA"> </span><o:p></o:p></p>
<p class="MsoNormal">Hello!<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB">I discovered a Heritrix mailinglist(*). Amongst some interesting tips on making the crawl faster, I also read some speed figures far from what we ever get. So I ask you: what do you get as speed values?</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB">Our latest 19 selective harvests have the following figures (from crawl-report.txt in the jobs metadata WARC file):</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB">URIs/sec: slowest job 0,83; fastest job 9,8; average 5,11</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB">KB/sec: slowest 34; fastest 863; average 313</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB">(I realize that this besides NAS/Heritrix configuration depends much on hardware, memory, disk I/O, network capacity etc. but don’t know which such figures that are most relevant to add to this comparison. Suggestions?)</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB">* <a href="https://groups.yahoo.com/neo/groups/archive-crawler/conversations/messages">
https://groups.yahoo.com/neo/groups/archive-crawler/conversations/messages</a></span><o:p></o:p></p>
</div>
</body>
</html>