<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:Consolas;
mso-fareast-language:EN-US;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="en-DK" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hi Peter<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Our broad crawl is based on all domains in the NAS harvest dbs - dead or alive. We are not forced to postprocess or clean a specific broadcrawl domain list before a broadcrawl to limit the number of processed domains.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">The total domain list in the database is updated before each broad crawl with the new “currently paid” domains
<a href="https://punktum.dk/en">from the danish TLD provider</a> - since 2005. <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Every domain in the harvest dbs get updated or "touched" 4 times per year - because domains appears and disappears very frequently in DK (30-40K over 3 months).
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">We are not aware of any problems with the way NAS do the updates.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">?Timelimit and time spent on job follow up.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">You can chose to decrease specific H3 template timeouts for your defaultorder template, use harvest timelimits or bytelimit a crawl step to avoid hanging jobs ( and make traffic limit agreements with the hosting centers,
too). <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">In DK every broadcrawl harvester instance is configured to only have 50 toethreads queues per job. A broad crawl job with a long seeds list is harvested in parallel within the toethreads 50 limit.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">If you set the toethreads limit higher you can process more seeds in parallel but you need to have HW (CPU,RAM, IO etc) and database power because of the harvesting reporting and
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">perhaps also fewer/one harvester instance(s) per server.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">It is weighted balance depending on your HW setup.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">In the broadcrawl we only follow up on the big step 2 jobs and a few big selective domain harvests containing only domains which hit the 50MB limit in step 1 -<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">e.g. sometimes we take very big domains out of the step 2 and put them into the focused selective crawls with more fast harvest setup using the selective pool of harvesters.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">The job follow up is hard and time consuming but “pay back” in QA of our biggest part of the harvested content – and gives a good understanding of our harvesting quality – and what are currently the most important harvesting
problems and new crawler traps. <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">If you don’t have the time for that, you can use time limits to get rid of hanging jobs and finish a broad crawl on time. It can be an appropriate choice.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Best regards<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Tue<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DK">
<o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="mso-fareast-language:#2000">From:</span></b><span lang="EN-US" style="mso-fareast-language:#2000"> NetarchiveSuite-users <netarchivesuite-users-bounces@ml.sbforge.org>
<b>On Behalf Of </b>Peter Svanberg<br>
<b>Sent:</b> Friday, 20 October 2023 18.46<br>
<b>To:</b> netarchivesuite-users@ml.sbforge.org<br>
<b>Subject:</b> Re: [Netarchivesuite-users] Timelimit – usage and NAS problems<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="en-DK"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">Yes, the harvesting stops when the timelimit is reached -- what else did you expect? :-)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">The purpose of setting a timelimit is, I suppose, to stop very slow or tracked/looping harvests. Maybe you meant that you don't have any need for that? Or did you expect some other timelimit behaviour?<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">And as I said (and maybe you alluded to) there is a problem with false-reporting timelimit for already completed domains. Can be fixed in the database, but you have to gather info from the metadata WARC files first,
to know what jobs was actually stopped. I have Python scripts. But this could be fixed quite easily in NAS.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">Thanks for an update of your current schedule and figures!<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">Side track: We also observed that NAS reports "Domain completed" even if the domain doesn't exist (DNS error). Maybe that case should be a separate stop reason?<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">-----<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">Peter Svanberg, Sweden<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="SV">-----Ursprungligt meddelande-----<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="SV">Från: NetarchiveSuite-users <</span><span lang="en-DK"><a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org"><span lang="SV">netarchivesuite-users-bounces@ml.sbforge.org</span></a></span><span lang="SV">> För
Tue Hejlskov Larsen<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="SV">Skickat: den 3 oktober 2023 17:13<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="SV">Till: </span><span lang="en-DK"><a href="mailto:netarchivesuite-users@ml.sbforge.org"><span lang="SV">netarchivesuite-users@ml.sbforge.org</span></a></span><span lang="SV"><o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">Ämne: Re: [Netarchivesuite-users] Timelimit – usage and NAS problems<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">If we use the timelimit for a broadcrawl job it cuts the domain seed queue crawlling - eventhough it is not finished harvesting of all the domain's in the job. You can see it in Heritrix seeds or host reports. We are
using a harvesting policy different from BNF's.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">So if we used timelimit for a job we will loose a lot of content especially in step 2. We have tried it - it was stopped after a couple of days. We have jobs in step 2 which runs for more weeks because of the size
of the domains.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">We have up to 10K domains in each job (domains are grouped e.g. by order templates hops etc.) in our 2 broad crawl steps.
<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">Each step has an overall maxbyte per domain (even though the maxbyte can be set higher on the domain level) : 50MB for step 1 and 16 G for step 2. In step 1 all domains in the jobdbs (about 3 mio) are crawled even
though they are inactive or active. Only the domains which hit the 50MB limit are included in step 2 with an overall 16G maxbyte limit per domain. Lower bytelimits on domain level have higher priority than the overall step limit. The 2 steps schedules about
500-800 jobs. Step 1 runs about 10 days on 110 crawlers in parallel without job curating and harvest about 12-15 TB. Step 2 runs on the same number of harvesters for about 6 weeks with job curating and harvest about 80-100 TB before dedup and compress. A broadcrawl
is done 4 times per year with "maintenance windows" - between 2 weeks to 1 month per quarter.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">Best regards<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">Tue <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">-----Original Message-----<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">From: NetarchiveSuite-users <</span><span lang="en-DK"><a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org"><span lang="EN-GB">netarchivesuite-users-bounces@ml.sbforge.org</span></a></span><span lang="EN-GB">>
On Behalf Of Peter Svanberg<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">Sent: Tuesday, 3 October 2023 15.35<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">To: 'netarchivesuite-users@ml.sbforge.org' <</span><span lang="en-DK"><a href="mailto:netarchivesuite-users@ml.sbforge.org"><span lang="EN-GB">netarchivesuite-users@ml.sbforge.org</span></a></span><span lang="EN-GB">><o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">Subject: Re: [Netarchivesuite-users] Timelimit – usage and NAS problems<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">(Continuation after Zoom meeting:)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">So BNF use timelimit but is not influenced by the NAS insufficiency and Denmark does not use it. Anyone else use it?<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">And what did you mean, Tue, about the queue? At the time limit all harvesters are stopped and what was harvested so far is saved in the WARC and the job is DONE -- or? What was the problem? (Besides that some domains
are falsely reported as timelimit stopped. But you can correct this in the database, which we did in one case.)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="SV">---<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="SV">Peter Svanberg, Sweden<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="SV"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="SV"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="SV">-----Ursprungligt meddelande-----<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="SV">Från: NetarchiveSuite-users <</span><span lang="en-DK"><a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org"><span lang="SV">netarchivesuite-users-bounces@ml.sbforge.org</span></a></span><span lang="SV">> För
Peter Svanberg<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="SV">Skickat: den 2 oktober 2023 13:37<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="SV">Till: </span><span lang="en-DK"><a href="mailto:netarchivesuite-users@ml.sbforge.org"><span lang="SV">netarchivesuite-users@ml.sbforge.org</span></a></span><span lang="SV"><o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">Ämne: [Netarchivesuite-users] Timelimit – usage and NAS problems<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">Two things about timelimits.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">1) When and how do you use timelimits in harvesting? It’s another way to limit the jobs. I suppose that stops mainly slow host – that maybe has figures for politeness in robots.txt, if you allow that to influence.
Or host with many small objects, each delaying.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">2) NAS has limitations in handling jobs stopped by timelimit. It checks for mentions of ”timelimit” on the last line in some Heritrix report and then reports timelimit for all domains which has not allready been stopped
by data or object limits. Hence the statistics gets wrong. In our current broad crawl (pass 3) just 11 % of the domains were not ready when the jobs where timelimit stopped. Also, if there is another pass, all those falsely timelimit-reported domains is unnecessarily
harvetested again.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">This can be corrected in two ways:<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">A) NAS could look in the hosts-report files ”remaining” column to check which domains are stopped or not.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">B) We could suggest/fix Heritrix to add a line in the log with a new Heritrix code when the queue for a domain gets empty. And then easily use that in NAS, as with objects and data limit codes.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">I appreciate answers and comments.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">Peter Svanberg<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">Sweden<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">_______________________________________________<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">NetarchiveSuite-users mailing list<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="en-DK"><a href="mailto:NetarchiveSuite-users@ml.sbforge.org"><span lang="EN-GB">NetarchiveSuite-users@ml.sbforge.org</span></a></span><span lang="EN-GB"><o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="en-DK"><a href="https://ml.sbforge.org/mailman/listinfo/netarchivesuite-users"><span lang="EN-GB">https://ml.sbforge.org/mailman/listinfo/netarchivesuite-users</span></a></span><span lang="EN-GB"><o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">_______________________________________________<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">NetarchiveSuite-users mailing list<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="en-DK"><a href="mailto:NetarchiveSuite-users@ml.sbforge.org"><span lang="EN-GB">NetarchiveSuite-users@ml.sbforge.org</span></a></span><span lang="EN-GB"><o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="en-DK"><a href="https://ml.sbforge.org/mailman/listinfo/netarchivesuite-users"><span lang="EN-GB">https://ml.sbforge.org/mailman/listinfo/netarchivesuite-users</span></a></span><span lang="EN-GB"><o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">_______________________________________________<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-GB">NetarchiveSuite-users mailing list<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="en-DK"><a href="mailto:NetarchiveSuite-users@ml.sbforge.org"><span lang="EN-GB">NetarchiveSuite-users@ml.sbforge.org</span></a></span><span lang="EN-GB"><o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="en-DK"><a href="https://ml.sbforge.org/mailman/listinfo/netarchivesuite-users"><span lang="EN-GB">https://ml.sbforge.org/mailman/listinfo/netarchivesuite-users</span></a></span><span lang="EN-GB"><o:p></o:p></span></p>
</div>
</body>
</html>