<font size=2 face="sans-serif">Hello Peter,</font><br><br><font size=2 face="sans-serif">Your conclusion is right: Yes, we do
have False in (2) and BnfHarvestReport in (3). And we never used positive
values for (1) Maxobjects.</font><br><br><font size=2 face="sans-serif">Regards,</font><br><font size=2 face="sans-serif"> Bert</font><br><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">De :
</font><font size=1 face="sans-serif">"Peter Svanberg"
<Peter.Svanberg@kb.se></font><br><font size=1 color=#5f5f5f face="sans-serif">A :
</font><font size=1 face="sans-serif">"netarchivesuite-users@ml.sbforge.org"
<netarchivesuite-users@ml.sbforge.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date :
</font><font size=1 face="sans-serif">31/08/2022 17:11</font><br><font size=1 color=#5f5f5f face="sans-serif">Objet :
</font><font size=1 face="sans-serif">Re: [Netarchivesuite-users]
Limit both number of bytes and number of objects per domain</font><br><font size=1 color=#5f5f5f face="sans-serif">Envoyé par :
</font><font size=1 face="sans-serif">"NetarchiveSuite-users"
<netarchivesuite-users-bounces@ml.sbforge.org></font><br><hr noshade><br><br><br><font size=3 color=#004080 face="Calibri">Hi Sara,</font><br><font size=3 color=#004080 face="Calibri"> </font><br><font size=3 color=#004080 face="Calibri">This was interesting! Lost
of questions:</font><br><font size=3 color=#004080 face="Calibri"> </font><br><font size=3 color=#004080 face="Calibri">1) Have
you used positive values on objects max?</font><br><font size=3 color=#004080 face="Calibri">2) Have
you changed </font><font size=3 color=#2f2f2f face="Consolas">settings.harvester.scheduler.jobGen.objectLimitIsSetByQuotaEnforcer
</font><font size=3 color=#004080 face="Calibri">or is it </font><font size=3 color=#2f2f2f face="Consolas">true</font><font size=3 color=#004080 face="Calibri">?</font><br><font size=3 color=#004080 face="Calibri">3) What
is your value on </font><font size=3 face="Calibri"><i>settings.harvester.harvesting.harvestReport.class</i></font><font size=3 color=#004080 face="Calibri">? </font><font size=3 color=#2f2f2f face="Consolas">BnfHarvestReport</font><font size=3 face="Times New Roman">or </font><font size=3 color=#2f2f2f face="Consolas">LegacyHarvestReport</font><br><font size=3 color=#004080 face="Calibri"> </font><br><font size=3 color=#004080 face="Calibri">The template examples in
NAS have both frontier and quotaenforcer, but with this comment.</font><br><font size=3 color=#004080 face="Calibri"> </font><br><font size=3 color=#004080 face="Calibri">## Can be used instead of
the QuotaEnforcer module. In this case the following line should look </font><br><font size=3 color=#004080 face="Calibri">## like: frontier.queueTotalBudget=%{FRONTIER_QUEUE_TOTAL_BUDGET_PLACEHOLDER}</font><br><font size=3 color=#004080 face="Calibri">## instead of: frontier.queueTotalBudget=</font><br><font size=3 color=#004080 face="Calibri"> </font><br><font size=3 color=#004080 face="Calibri">frontier.queueTotalBudget=%{FRONTIER_QUEUE_TOTAL_BUDGET_PLACEHOLDER}</font><br><font size=3 color=#004080 face="Calibri"> </font><br><font size=3 color=#004080 face="Calibri">(Somewhat later:) Now I see,
your statement makes me understand the handling in </font><font size=3 face="Calibri"><i>configureQuotaEnforcer():</i></font><font size=3 color=#004080 face="Calibri">it makes it possible to have both, by setting the value of the one that
shouldn’t be used to infinity.</font><br><font size=3 color=#004080 face="Calibri"> </font><br><font size=3 color=#004080 face="Calibri">But you must have False in
(2) and </font><font size=3 color=#2f2f2f face="Consolas">BnfHarvestReport</font><font size=3 color=#004080 face="Calibri">in (3) above, or else I’m puzzled again. </font><font size=3 color=#004080 face="Wingdings">J</font><br><font size=3 color=#004080 face="Calibri"> </font><br><font size=3 face="Arial">-----<br>Peter</font><font size=3 color=#004080 face="Calibri"><br></font><br><font size=3 color=#004080 face="Calibri"> </font><br><font size=3 face="Calibri"><b>Från:</b> NetarchiveSuite-users <</font><a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org"><font size=3 color=blue face="Calibri"><u>netarchivesuite-users-bounces@ml.sbforge.org</u></font></a><font size=3 face="Calibri">>
<b>För </b></font><a href=mailto:sara.aubry@bnf.fr><font size=3 color=blue face="Calibri"><u>sara.aubry@bnf.fr</u></font></a><font size=3 face="Calibri"><b><br>Skickat:</b> den 30 augusti 2022 14:02<b><br>Till:</b> </font><a href="mailto:netarchivesuite-users@ml.sbforge.org"><font size=3 color=blue face="Calibri"><u>netarchivesuite-users@ml.sbforge.org</u></font></a><font size=3 face="Calibri"><b><br>Ämne:</b> Re: [Netarchivesuite-users] Limit both number of bytes and number
of objects per domain</font><br><font size=3 face="Times New Roman"> </font><br><font size=3 face="Arial">Hi Peter,</font><font size=3 face="Times New Roman"><br></font><font size=3 face="Arial"><br>I can't technically answer your question but QuotaEnforcer and </font><font size=3 face="Courier New">queueTotalBudget
</font><font size=3 face="Arial">are two different processors and have
not been implemented in Heritrix to work together.</font><font size=3 face="Times New Roman"><br></font><font size=3 face="Arial"><br>At BnF, we are using </font><font size=3 face="Courier New">queueTotalBudget</font><font size=3 face="Arial">to
manage queues by number of URLs.<br>Here is what we have in our profiles :</font><font size=3 face="Times New Roman"><br></font><font size=3 face="Courier New"><br> <!-- FRONTIER (START)<br> Record of all URIs discovered and queued-for-collection<br> --><br> <bean id="frontier" class="org.archive.crawler.frontier.BdbFrontier"><br> <property name="maxRetries" value="10"
/><br> <property name="retryDelaySeconds" value="60"
/><br> <property name="recoveryLogEnabled"
value="false" /><br> <property name="balanceReplenishAmount"
value="1000" /><br> <property name="errorPenaltyAmount"
value="1" /><br> <!-- NETARCHIVESUITE Placeholder FRONTIER_QUEUE_TOTAL_BUDGET_PLACEHOLDER
--><br> <property name="queueTotalBudget" value="%{FRONTIER_QUEUE_TOTAL_BUDGET_PLACEHOLDER}"
/><br> <property name="snoozeLongMs" value="300000"
/><br> <property name="extract404s" value="false"
/><br> </bean><br> <!-- FRONTIER (END) --></font><font size=3 face="Times New Roman"><br></font><font size=3 face="Arial"><br>And we have no place holder for the quotaEnforcer.</font><font size=3 face="Times New Roman"><br></font><font size=3 face="Arial"><br>Best,</font><font size=3 face="Times New Roman"><br></font><font size=3 face="Arial"><br>Sara</font><font size=3 face="Times New Roman"><br><br><br><br></font><font size=3 color=#5f5f5f face="Arial"><br>De : </font><font size=3 face="Arial">"Peter
Svanberg" <</font><a href=mailto:Peter.Svanberg@kb.se><font size=3 color=blue face="Arial"><u>Peter.Svanberg@kb.se</u></font></a><font size=3 face="Arial">></font><font size=3 color=#5f5f5f face="Arial"><br>A : </font><font size=3 face="Arial">"</font><a href="mailto:netarchivesuite-users@ml.sbforge.org"><font size=3 color=blue face="Arial"><u>netarchivesuite-users@ml.sbforge.org</u></font></a><font size=3 face="Arial">"
<</font><a href="mailto:netarchivesuite-users@ml.sbforge.org"><font size=3 color=blue face="Arial"><u>netarchivesuite-users@ml.sbforge.org</u></font></a><font size=3 face="Arial">></font><font size=3 color=#5f5f5f face="Arial"><br>Date : </font><font size=3 face="Arial">30/08/2022
13:41</font><font size=3 color=#5f5f5f face="Arial"><br>Objet : </font><font size=3 face="Arial">Re:
[Netarchivesuite-users] Limit both number of bytes and number of objects
per domain</font><font size=3 color=#5f5f5f face="Arial"><br>Envoyé par : </font><font size=3 face="Arial">"NetarchiveSuite-users"
<</font><a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org"><font size=3 color=blue face="Arial"><u>netarchivesuite-users-bounces@ml.sbforge.org</u></font></a><font size=3 face="Arial">></font><div align=center><hr noshade></div><br><font size=3 face="Times New Roman"><br><br></font><font size=3 color=#004080 face="Calibri"><br>Sorry, I mixed it up, alt. 3 edited below. So I suppose now that alt. 3
is true. And that the value of <i>frontier.queueTotalBudget</i> is irrelevant
if you use <i>quotaenforcer</i>, i.e. if <i><ref bean="quotaenforcer"/></i> is among the fetchProcessors.processors. True?<br> <br>But there is a rumour that you should decide between byte and object limit
– true or false?<br> <br>Regards,</font><font size=3 face="Arial"><br>-----<br>Peter Svanberg</font><font size=3 face="Times New Roman"><br></font><font size=3 color=#004080 face="Calibri"><br> </font><font size=3 face="Calibri"><b><br>Från:</b> NetarchiveSuite-users <</font><a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org"><font size=3 color=blue face="Calibri"><u>netarchivesuite-users-bounces@ml.sbforge.org</u></font></a><font size=3 face="Calibri">>
<b>För </b>Peter Svanberg<b><br>Skickat:</b> den 29 augusti 2022 14:20<b><br>Till:</b> </font><a href="mailto:netarchivesuite-users@ml.sbforge.org"><font size=3 color=blue face="Calibri"><u>netarchivesuite-users@ml.sbforge.org</u></font></a><font size=3 face="Calibri"><b><br>Ämne:</b> [Netarchivesuite-users] Limit both number of bytes and number
of objects per domain<br> <br>Could someone please explain this handling?<br> <br>In a snapshot we want to limit both number of bytes and number of objects
per domain. If you give positive values in GUI for new snapshot harvest,
what is recommended?<br> <br>1. You should not. Why not?<br>2. You must change <i>settings.harvester.scheduler.jobGen.objectLimitIsSetByQuotaEnforcer</i>to
false and change<i><br>settings.harvester.harvesting.harvestReport.class</i> to <i>dk.netarkivet.harvester.harvesting.report.BnfHarvestReport</i>(which
doesn’t assume annotations in crawl log).<br>3. You can keep <i>settings.harvester.scheduler.jobGen.objectLimitIsSetByQuotaEnforcer</i>as
true and it works …? Even though </font><font size=3 color=#004080 face="Calibri">FRONTIER_QUEUE_TOTAL_BUDGET_PLACEHOLDER
(and hence frontier.queueTotalBudget) is set to infinity?</font><font size=3 face="Calibri"><i><strike>QUOTA_ENFORCER_GROUP_MAX_FETCH_SUCCES_PLACEHOLDER</strike></i><strike>in
template (and hence <i>quotaenforcer.groupMaxFetchSuccesses</strike></i><strike>)
is set to infinity</strike> (in <i>configureQuotaEnforcer()</i>)?<br> <br>Regards,<br> <br> </font><p><table width=394 style="border-collapse:collapse;"><tr height=8><td width=113 style="border-style:solid;border-color:#000000;border-width:0px 3px 0px 0px;padding:0px 0px;"><td width=278 style="border-style:solid;border-color:#000000;border-width:0px 0px 0px 3px;padding:0px 0px;"><font size=3 face="Arial"><b>Peter
Svanberg<br>Technical officer </b><br>Aquisitions and Metadata Department<br>Film, Games, Sheet Music and Web Unit<br> <b><br>National Library of Sweden</b><br>PO Box 5039, SE-102 41 Stockholm<br>Visits: Karlavägen 96, Stockholm<br>+46 10-709 32 78</font><font size=3 color=blue face="Times New Roman"><u><br></u></font><a href=mailto:Peter.Svanberg@kb.se><font size=3 color=#0082bf face="Arial"><u>Peter.Svanberg@kb.se</u></font></a><font size=3 color=blue face="Times New Roman"><u><br></u></font><a href=https://www.kb.se/><font size=3 color=blue face="Arial"><u>www.kb.se</u></font></a></table><p><font size=3 face="Calibri"><br> <br> </font><font size=3 face="Courier New">_______________________________________________<br>NetarchiveSuite-users mailing list</font><font size=3 color=blue face="Times New Roman"><u><br></u></font><a href="mailto:NetarchiveSuite-users@ml.sbforge.org"><font size=3 color=blue face="Courier New"><u>NetarchiveSuite-users@ml.sbforge.org</u></font></a><font size=3 color=blue face="Times New Roman"><u><br></u></font><a href="https://ml.sbforge.org/mailman/listinfo/netarchivesuite-users"><font size=3 color=blue face="Courier New"><u>https://ml.sbforge.org/mailman/listinfo/netarchivesuite-users</u></font></a><div align=center><hr></div><p><font size=3 face="Arial">Samedi 17 et dimanche 18 septembre 2022 :
<b>la BnF fête la réouverture du site Richelieu</b>, après douze ans de
travaux de rénovation et de modernisation, avec </font><a href="https://www.bnf.fr/fr/agenda/richelieu-le-reveil-reouverture"><font size=3 color=blue face="Arial"><b><u>un
parcours de visite</u></b></font></a><font size=3 face="Arial"> en compagnie
d’artistes et comédiens l'après-midi, et </font><a href="https://www.bnf.fr/fr/agenda/richelieu-le-reveil-performances-et-lectures"><font size=3 color=blue face="Arial"><b><u>des
événements et performances</u></b></font></a><font size=3 face="Arial">la soirée. </font><p><font size=3 color=#008000 face="Arial"><b>Avant d'imprimer, pensez
à l'environnement.</b></font><tt><font size=2>_______________________________________________<br>NetarchiveSuite-users mailing list<br>NetarchiveSuite-users@ml.sbforge.org<br></font></tt><a href="https://ml.sbforge.org/mailman/listinfo/netarchivesuite-users"><tt><font size=2>https://ml.sbforge.org/mailman/listinfo/netarchivesuite-users</font></tt></a><tt><font size=2><br></font></tt><p><font face="sans-serif"><hr />
<p>Samedi 17 et dimanche 18 septembre 2022 : <strong>la BnF fête la réouverture du site Richelieu</strong>, après douze ans de travaux de rénovation et de modernisation, avec <strong><a href="https://www.bnf.fr/fr/agenda/richelieu-le-reveil-reouverture">un parcours de visite</a></strong> en compagnie d’artistes et comédiens l'après-midi, et <strong><a href="https://www.bnf.fr/fr/agenda/richelieu-le-reveil-performances-et-lectures">des événements et performances</a></strong> la soirée.
<p style="color:#008000"><strong>Avant d'imprimer, pensez à l'environnement.</strong></p></font>