<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
tt
{mso-style-priority:99;
font-family:"Courier New";}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.E-postmall20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.apple-style-span
{mso-style-name:apple-style-span;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="SV" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Hi Sara,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">The phrase ”revisit target files” was an ill-chosen way of referring to “the WARC files indirectly (via index) referenced
from revisit records”.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">All those files must be accessible for a process reading a certain WARC file. You can’t take out separate WARC files (e.g.
for research) unless you “reduplicate” them. (Are there tools for doing that?)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Well, we must consider these different aspects. Thank you for info!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Regards,
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Peter Svanberg<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Från:</span></b><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> NetarchiveSuite-users <netarchivesuite-users-bounces@ml.sbforge.org>
<b>För </b>sara.aubry@bnf.fr<br>
<b>Skickat:</b> den 14 januari 2020 12:36<br>
<b>Till:</b> netarchivesuite-users@ml.sbforge.org<br>
<b>Ämne:</b> Re: [Netarchivesuite-users] Questions about deduplication (and reduplication)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:10.0pt;font-family:"Arial",sans-serif">Hi Peter,</span><span lang="EN-GB"><br>
<br>
</span><span lang="EN-GB" style="font-size:10.0pt;font-family:"Arial",sans-serif">If you set deduplication to true in NAS harvesting settings and at profile level, then Heritrix will create revisit records (not revisit files) in the harvesting workflow, so
along with other WARC request, response and metadata records.</span><span lang="EN-GB"><br>
</span><span lang="EN-GB" style="font-size:10.0pt;font-family:"Arial",sans-serif">Each time the crawler tries to fetch a binary web component, it lookups in the lucene duplicates index and if there, it will mark it in the crawl log and create a complete WARC
revisit record. </span><span lang="EN-GB"><br>
<br>
</span><tt><span lang="EN-GB">2020-01-14T11:15:47.302Z 200 2065 </span></tt><a href="https://img.lemde.fr/2015/10/01/0/123/3253/2169/110/74/60/0/a55eb3e_25814-1pls9ni.jpg"><tt><span lang="EN-GB">https://img.lemde.fr/2015/10/01/0/123/3253/2169/110/74/60/0/a55eb3e_25814-1pls9ni.jpg</span></tt></a><tt><span lang="EN-GB">LE
</span></tt><a href="https://www.lemonde.fr/services/"><tt><span lang="EN-GB">https://www.lemonde.fr/services/</span></tt></a><tt><span lang="EN-GB">image/jpeg #118 20200114111547158+32 sha1:WPRRSOTVFZNNIDJVPHMT5LDDNDGIMPRR
</span></tt><a href="https://www.lemonde.fr/afrique/"><tt><span lang="EN-GB">https://www.lemonde.fr/afrique/</span></tt></a><tt><b><span lang="EN-GB">duplicate:"BnF-32274-28-20191212105654-00003-ciblee_2019_fogg120.bnf.fr.warc.gz,295532254,20191212110808000</span></b></tt><tt><span lang="EN-GB">",content-size:2579</span></tt><span lang="EN-GB"><br>
<br>
</span><span lang="EN-GB" style="font-size:10.0pt;font-family:"Arial",sans-serif">OpenWayback CDX indexer creates CDX lines for these records that OpenWayback playbacks very well.</span><span lang="EN-GB"><br>
</span><span lang="EN-GB" style="font-size:10.0pt;font-family:"Arial",sans-serif">I imagine th pywb also plays them without any problem.</span><span lang="EN-GB"><br>
<br>
</span><span lang="EN-GB" style="font-size:10.0pt;font-family:"Arial",sans-serif">The oldest revist can be very very old if the file hasn't changed and is still being crawled.</span><span lang="EN-GB"><br>
</span><span lang="EN-GB" style="font-size:10.0pt;font-family:"Arial",sans-serif">I don't know how old is our oldest (not before late 2016, since it came with NAS 5.2 :</span><span lang="EN-GB"><br>
</span><a href="https://sbforge.org/display/NAS/NetarchiveSuite+5.2.x+Release+Notes"><span lang="EN-GB" style="font-size:10.0pt;font-family:"Arial",sans-serif">https://sbforge.org/display/NAS/NetarchiveSuite+5.2.x+Release+Notes</span></a><span lang="EN-GB" style="font-size:10.0pt;font-family:"Arial",sans-serif">)</span><span lang="EN-GB"><br>
</span><span lang="EN-GB" style="font-size:10.0pt;font-family:"Arial",sans-serif"> </span><span lang="EN-GB"><br>
</span><span lang="EN-GB" style="font-size:10.0pt;font-family:"Arial",sans-serif">Regarding space saving, we have precise numbers :</span><span lang="EN-GB"><br>
</span><span lang="EN-GB" style="font-size:10.0pt;font-family:"Arial",sans-serif">For our 2019 focused crawls, we harvested 107,49TB of uncompressed data and didn't harvest 34,77TB we "saved" from deduplication (i.e. 24,5%).</span><span lang="EN-GB"><br>
</span><span lang="EN-GB" style="font-size:10.0pt;font-family:"Arial",sans-serif">For our 2019 broad crawl, we harvested 234,64TB of uncompressed data and didn't harvest 66,53TB we "saved" from deduplication (i.e 22%). </span><span lang="EN-GB"><br>
</span><span lang="EN-GB" style="font-size:10.0pt;font-family:"Arial",sans-serif">So at our scale, deduplication saves a fourth of our storage, which is huge !</span><span lang="EN-GB"><br>
<br>
</span><span lang="EN-GB" style="font-size:10.0pt;font-family:"Arial",sans-serif">Sara</span><span lang="EN-GB"><br>
<br>
<br>
<br>
<br>
<br>
<br>
</span><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">De : </span><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif">"Peter Svanberg" <</span><a href="mailto:Peter.Svanberg@kb.se"><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif">Peter.Svanberg@kb.se</span></a><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif">></span><span lang="EN-GB"><br>
</span><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">A : </span><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif">"</span><a href="mailto:netarchivesuite-users@ml.sbforge.org"><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif">netarchivesuite-users@ml.sbforge.org</span></a><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif">"
<</span><a href="mailto:netarchivesuite-users@ml.sbforge.org"><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif">netarchivesuite-users@ml.sbforge.org</span></a><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif">></span><span lang="EN-GB"><br>
</span><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">Date : </span><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif">13/01/2020 23:05</span><span lang="EN-GB"><br>
</span><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">Objet : </span><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif">Re: [Netarchivesuite-users] Questions about deduplication (and reduplication)</span><span lang="EN-GB"><br>
</span><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">Envoyé par : </span><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif">"NetarchiveSuite-users" <</span><a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org"><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif">netarchivesuite-users-bounces@ml.sbforge.org</span></a><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif">></span><span lang="EN-GB"><o:p></o:p></span></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" noshade="" style="color:#A0A0A0" align="center">
</div>
<p class="MsoNormal"><span lang="EN-GB"><br>
<br>
<br>
Thanks, Sara!<br>
<br>
So, when reduplicating, e.g. at Wayback or Pyweb usage, all potential revisit target files must be reachable – not a problem? Kristinn mentioned that generating indexes (of content) can take much longer as it have to look up in url indexes and open a lot of
files. Something you (or others) have experienced?<br>
<br>
Do you have any idea of how old the oldest revisit target to recent warc files could be? Five, maybe ten years, then?<br>
<br>
And I add a fifth question:<br>
<br>
5) How much space do you save – just approximately.<br>
<br>
Peter<br>
<br>
<br>
13 jan. 2020 kl. 17:47 skrev "</span><a href="mailto:sara.aubry@bnf.fr"><span lang="EN-GB">sara.aubry@bnf.fr</span></a><span lang="EN-GB">" <</span><a href="mailto:sara.aubry@bnf.fr"><span lang="EN-GB">sara.aubry@bnf.fr</span></a><span lang="EN-GB">>:<br>
<br>
</span><span style="font-family:"Tahoma",sans-serif"></span><span lang="EN-GB" style="font-size:10.0pt;font-family:"Arial",sans-serif">Hi Peter,</span><span lang="EN-GB"><br>
</span><span lang="EN-GB" style="font-size:10.0pt;font-family:"Arial",sans-serif"><br>
For BnF,</span><span lang="EN-GB" style="font-family:"Calibri",sans-serif"><br>
1) yes<br>
2) you probably mean focused crawls: yes <br>
3) URL<br>
4) Only when we have a major change in the crawler or the data format. Which means, the least possible.<br>
Because it really save a lot of space, and also because we don't care about intervals between WARC files: that's why WARC revisit records were made for.<br>
Deduplication also sometimes incidentally restarts when the previous capture of a harvest is not finished (either at crawl stage or post-processing stage) or crashed.</span><span lang="EN-GB"><br>
</span><span lang="EN-GB" style="font-family:"Calibri",sans-serif"><br>
Best,</span><span lang="EN-GB"><br>
</span><span lang="EN-GB" style="font-family:"Calibri",sans-serif"><br>
Sara</span><span lang="EN-GB"><br>
<br>
<br>
<br>
</span><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F"><br>
De : </span><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif">"Peter Svanberg" <</span><a href="mailto:Peter.Svanberg@kb.se"><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif">Peter.Svanberg@kb.se</span></a><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif">><span style="color:#5F5F5F"><br>
A : </span>"</span><a href="mailto:netarchivesuite-users@ml.sbforge.org"><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif">netarchivesuite-users@ml.sbforge.org</span></a><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif">"
<</span><a href="mailto:netarchivesuite-users@ml.sbforge.org"><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif">netarchivesuite-users@ml.sbforge.org</span></a><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif">><span style="color:#5F5F5F"><br>
Date : </span>13/01/2020 17:31<span style="color:#5F5F5F"><br>
Objet : </span>[Netarchivesuite-users] Questions about deduplication (and reduplication)<span style="color:#5F5F5F"><br>
Envoyé par : </span>"NetarchiveSuite-users" <</span><a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org"><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif">netarchivesuite-users-bounces@ml.sbforge.org</span></a><span lang="EN-GB" style="font-size:7.5pt;font-family:"Arial",sans-serif">></span><span lang="EN-GB"><o:p></o:p></span></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" noshade="" style="color:#A0A0A0" align="center">
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-GB"><br>
<br>
</span><span lang="EN-GB" style="font-family:"Calibri",sans-serif"><br>
Hello!<br>
<br>
I’m trying to understand how NAS and Heritrix handles deduplication, which lead to an internal discussion about the overall pros and cons of ditto. I then found Kristinn Sigurðsson’s interesting web archiving blog articles. He has written about de- and reduplication:
</span><a href="https://kris-sigur.blogspot.com/2015/01/the-downside-of-web-archive.html"><span lang="EN-GB" style="font-family:"Calibri",sans-serif;color:#0082BF">https://kris-sigur.blogspot.com/2015/01/the-downside-of-web-archive.html</span></a><span lang="EN-GB" style="font-family:"Calibri",sans-serif"><br>
<br>
Some short questions about the deduplication in NAS (is.hi.bok.deduplicator.DeDuplicator) that I would appreciate quick answers on (from all NAS user sites):<br>
<br>
1) Do you use deduplication for snapshot harvests (broad crawls)?<br>
2) Do you use deduplication for snapshot harvests?<br>
3) Which matching method do you use – DIGEST or URL?<br>
4) Do you “restart” the deduplication at intervals? How long intervals?<br>
<br>
By (4) I mean you do a harvest with no deduplication, limiting the number of dependencies between WARC files. (Somewhat like total and incremental backups.) Maybe you just do deduplication between the 2–3 steps in a broad crawl? Or between the last X broad
crawls?<br>
<br>
Regards, </span><span lang="EN-GB" style="font-family:"Arial",sans-serif"><br>
-----<br>
<br>
Peter Svanberg<br>
<br>
National Library of Sweden<br>
Phone: +46 10 709 32 78<br>
<br>
E-mail</span><span lang="EN-GB" style="font-family:"Calibri",sans-serif">: </span>
<a href="mailto:peter.svanberg@kb.se"><span lang="EN-GB" style="font-family:"Arial",sans-serif">peter.svanberg@kb.se</span></a><span lang="EN-GB" style="font-family:"Arial",sans-serif"><br>
Web</span><span lang="EN-GB" style="font-family:"Calibri",sans-serif">: </span><a href="www.kb.se"><span lang="EN-GB" style="font-family:"Arial",sans-serif">www.kb.se</span></a><span lang="EN-GB"><br>
</span><span lang="EN-GB" style="font-family:"Calibri",sans-serif"><br>
<br>
</span><tt><span lang="EN-GB" style="font-size:10.0pt">_______________________________________________</span></tt><span lang="EN-GB" style="font-size:10.0pt;font-family:"Courier New""><br>
<tt>NetarchiveSuite-users mailing list</tt><br>
</span><a href="mailto:NetarchiveSuite-users@ml.sbforge.org"><span lang="EN-GB" style="font-size:10.0pt;font-family:"Courier New"">NetarchiveSuite-users@ml.sbforge.org</span></a><u><span lang="EN-GB" style="color:blue"><br>
</span></u><a href="https://ml.sbforge.org/mailman/listinfo/netarchivesuite-users"><tt><span lang="EN-GB" style="font-size:10.0pt">https://ml.sbforge.org/mailman/listinfo/netarchivesuite-users</span></tt></a><span lang="EN-GB"><o:p></o:p></span></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center">
</div>
<p><span style="font-family:"Arial",sans-serif">Exposition </span><a href="https://www.bnf.fr/fr/agenda/tolkien-voyage-en-terre-du-milieu"><b><i><span style="font-family:"Arial",sans-serif">Tolkien, voyage en Terre du Milieu</span></i></b></a><span style="font-family:"Arial",sans-serif">-
du 22 octobre 2019 au 16 février 2020 - BnF - François-Mitterrand</span><o:p></o:p></p>
<p><b><span lang="EN-GB" style="font-family:"Arial",sans-serif;color:green">Avant d'imprimer, pensez à l'environnement.</span></b><span lang="EN-GB"><o:p></o:p></span></p>
<p><span lang="EN-GB">_______________________________________________<br>
NetarchiveSuite-users mailing list<br>
</span><a href="mailto:NetarchiveSuite-users@ml.sbforge.org"><span lang="EN-GB">NetarchiveSuite-users@ml.sbforge.org</span></a><span lang="EN-GB"><br>
</span><a href="https://ml.sbforge.org/mailman/listinfo/netarchivesuite-users"><span lang="EN-GB">https://ml.sbforge.org/mailman/listinfo/netarchivesuite-users</span></a><tt><span lang="EN-GB" style="font-size:10.0pt">_______________________________________________</span></tt><span lang="EN-GB" style="font-size:10.0pt;font-family:"Courier New""><br>
<tt>NetarchiveSuite-users mailing list</tt><br>
</span><a href="mailto:NetarchiveSuite-users@ml.sbforge.org"><span lang="EN-GB" style="font-size:10.0pt;font-family:"Courier New"">NetarchiveSuite-users@ml.sbforge.org</span></a><span lang="EN-GB" style="font-size:10.0pt;font-family:"Courier New""><br>
</span><a href="https://ml.sbforge.org/mailman/listinfo/netarchivesuite-users"><tt><span lang="EN-GB" style="font-size:10.0pt">https://ml.sbforge.org/mailman/listinfo/netarchivesuite-users</span></tt></a><span lang="EN-GB"><o:p></o:p></span></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-family:"Arial",sans-serif">
<hr size="2" width="100%" align="center">
</span></div>
<p><span style="font-family:"Arial",sans-serif">Exposition </span><a href="https://www.bnf.fr/fr/agenda/tolkien-voyage-en-terre-du-milieu"><b><i><span style="font-family:"Arial",sans-serif">Tolkien, voyage en Terre du Milieu</span></i></b></a><span style="font-family:"Arial",sans-serif">
- du 22 octobre 2019 au 16 février 2020 - BnF - François-Mitterrand<o:p></o:p></span></p>
<p><strong><span lang="EN-GB" style="font-family:"Arial",sans-serif;color:green">Avant d'imprimer, pensez à l'environnement.</span></strong><span lang="EN-GB" style="font-family:"Arial",sans-serif;color:green"><o:p></o:p></span></p>
</div>
</body>
</html>