<font size=2 face="Arial">Hi Peter,</font><br><br><font size=2 color=#004080 face="Arial">1) Have
you used positive values on objects max?</font><br><font size=2 face="Arial">We used a set of positive values in "Maximum
number of objects" defined in configurations (usually 50,000, 100,000
or 150,000). </font><br><br><font size=2 color=#004080 face="Arial">2) Have
you changed </font><font size=2 color=#2f2f2f face="Arial">settings.harvester.scheduler.jobGen.objectLimitIsSetByQuotaEnforcer
</font><font size=2 color=#004080 face="Arial">or is it </font><font size=2 color=#2f2f2f face="Arial">true</font><font size=2 color=#004080 face="Arial">?</font><br><font size=2 face="Arial">objectLimitIsSetByQuotaEnforcer is set to
false</font><br><br><font size=2 color=#004080 face="Arial">3) What
is your value on </font><font size=2 face="Arial"><i>settings.harvester.harvesting.harvestReport.class</i></font><font size=2 color=#004080 face="Arial">? </font><font size=2 color=#2f2f2f face="Arial">BnfHarvestReport</font><font size=2 face="Arial">or </font><font size=2 color=#2f2f2f face="Arial">LegacyHarvestReport</font><br><font size=2 face="Arial">settings.harvester.harvesting.harvestReport.class
is set to BnfHarvestReport</font><br><br><font size=2 face="Arial">Best,</font><br><br><font size=2 face="Arial">Sara</font><br><br><br><br><font size=1 color=#5f5f5f face="Agency FB">De :
</font><font size=1 face="Agency FB">"Peter Svanberg"
<Peter.Svanberg@kb.se></font><br><font size=1 color=#5f5f5f face="Agency FB">A :
</font><font size=1 face="Agency FB">"netarchivesuite-users@ml.sbforge.org"
<netarchivesuite-users@ml.sbforge.org></font><br><font size=1 color=#5f5f5f face="Agency FB">Date :
</font><font size=1 face="Agency FB">31/08/2022 17:11</font><br><font size=1 color=#5f5f5f face="Agency FB">Objet :
</font><font size=1 face="Agency FB">Re: [Netarchivesuite-users]
Limit both number of bytes and number of objects per domain</font><br><font size=1 color=#5f5f5f face="Agency FB">Envoyé par :
</font><font size=1 face="Agency FB">"NetarchiveSuite-users"
<netarchivesuite-users-bounces@ml.sbforge.org></font><br><hr noshade><br><br><br><font size=3 color=#004080 face="Agency FB">Hi Sara,</font><br><font size=3 color=#004080 face="Agency FB"> </font><br><font size=3 color=#004080 face="Agency FB">This was interesting! Lost
of questions:</font><br><font size=3 color=#004080 face="Agency FB"> </font><br><font size=3 color=#004080 face="Agency FB">1) Have
you used positive values on objects max?</font><br><font size=3 color=#004080 face="Agency FB">2) Have
you changed </font><font size=3 color=#2f2f2f face="Agency FB">settings.harvester.scheduler.jobGen.objectLimitIsSetByQuotaEnforcer
</font><font size=3 color=#004080 face="Agency FB">or is it </font><font size=3 color=#2f2f2f face="Agency FB">true</font><font size=3 color=#004080 face="Agency FB">?</font><br><font size=3 color=#004080 face="Agency FB">3) What
is your value on </font><font size=3 face="Agency FB"><i>settings.harvester.harvesting.harvestReport.class</i></font><font size=3 color=#004080 face="Agency FB">? </font><font size=3 color=#2f2f2f face="Agency FB">BnfHarvestReport</font><font size=3 face="Agency FB">or </font><font size=3 color=#2f2f2f face="Agency FB">LegacyHarvestReport</font><br><font size=3 color=#004080 face="Agency FB"> </font><br><font size=3 color=#004080 face="Agency FB">The template examples in
NAS have both frontier and quotaenforcer, but with this comment.</font><br><font size=3 color=#004080 face="Agency FB"> </font><br><font size=3 color=#004080 face="Agency FB">## Can be used instead
of the QuotaEnforcer module. In this case the following line should look
</font><br><font size=3 color=#004080 face="Agency FB">## like: frontier.queueTotalBudget=%{FRONTIER_QUEUE_TOTAL_BUDGET_PLACEHOLDER}</font><br><font size=3 color=#004080 face="Agency FB">## instead of: frontier.queueTotalBudget=</font><br><font size=3 color=#004080 face="Agency FB"> </font><br><font size=3 color=#004080 face="Agency FB">frontier.queueTotalBudget=%{FRONTIER_QUEUE_TOTAL_BUDGET_PLACEHOLDER}</font><br><font size=3 color=#004080 face="Agency FB"> </font><br><font size=3 color=#004080 face="Agency FB">(Somewhat later:) Now I
see, your statement makes me understand the handling in </font><font size=3 face="Agency FB"><i>configureQuotaEnforcer():</i></font><font size=3 color=#004080 face="Agency FB">it makes it possible to have both, by setting the value of the one that
shouldn’t be used to infinity.</font><br><font size=3 color=#004080 face="Agency FB"> </font><br><font size=3 color=#004080 face="Agency FB">But you must have False
in (2) and </font><font size=3 color=#2f2f2f face="Agency FB">BnfHarvestReport</font><font size=3 color=#004080 face="Agency FB">in (3) above, or else I’m puzzled again. J</font><br><font size=3 color=#004080 face="Agency FB"> </font><br><font size=3 face="Agency FB">-----<br>Peter</font><font size=3 color=#004080 face="Agency FB"><br></font><br><font size=3 color=#004080 face="Agency FB"> </font><br><font size=3 face="Agency FB"><b>Från:</b> NetarchiveSuite-users <</font><a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org"><font size=3 color=blue face="Agency FB"><u>netarchivesuite-users-bounces@ml.sbforge.org</u></font></a><font size=3 face="Agency FB">>
<b>För </b></font><a href=mailto:sara.aubry@bnf.fr><font size=3 color=blue face="Agency FB"><u>sara.aubry@bnf.fr</u></font></a><font size=3 face="Agency FB"><b><br>Skickat:</b> den 30 augusti 2022 14:02<b><br>Till:</b> </font><a href="mailto:netarchivesuite-users@ml.sbforge.org"><font size=3 color=blue face="Agency FB"><u>netarchivesuite-users@ml.sbforge.org</u></font></a><font size=3 face="Agency FB"><b><br>Ämne:</b> Re: [Netarchivesuite-users] Limit both number of bytes and number
of objects per domain</font><br><font size=3 face="Agency FB"> </font><br><font size=3 face="Agency FB">Hi Peter,<br><br>I can't technically answer your question but QuotaEnforcer and queueTotalBudget
are two different processors and have not been implemented in Heritrix
to work together.<br><br>At BnF, we are using queueTotalBudgetto manage queues by number of
URLs.<br>Here is what we have in our profiles :<br><br> <!-- FRONTIER (START)<br> Record of all URIs discovered and queued-for-collection<br> --><br> <bean id="frontier" class="org.archive.crawler.frontier.BdbFrontier"><br> <property name="maxRetries" value="10"
/><br> <property name="retryDelaySeconds" value="60"
/><br> <property name="recoveryLogEnabled"
value="false" /><br> <property name="balanceReplenishAmount"
value="1000" /><br> <property name="errorPenaltyAmount"
value="1" /><br> <!-- NETARCHIVESUITE Placeholder FRONTIER_QUEUE_TOTAL_BUDGET_PLACEHOLDER
--><br> <property name="queueTotalBudget" value="%{FRONTIER_QUEUE_TOTAL_BUDGET_PLACEHOLDER}"
/><br> <property name="snoozeLongMs" value="300000"
/><br> <property name="extract404s" value="false"
/><br> </bean><br> <!-- FRONTIER (END) --><br><br>And we have no place holder for the quotaEnforcer.<br><br>Best,<br><br>Sara<br><br><br><br></font><font size=3 color=#5f5f5f face="Agency FB"><br>De : </font><font size=3 face="Agency FB">"Peter
Svanberg" <</font><a href=mailto:Peter.Svanberg@kb.se><font size=3 color=blue face="Agency FB"><u>Peter.Svanberg@kb.se</u></font></a><font size=3 face="Agency FB">></font><font size=3 color=#5f5f5f face="Agency FB"><br>A : </font><font size=3 face="Agency FB">"</font><a href="mailto:netarchivesuite-users@ml.sbforge.org"><font size=3 color=blue face="Agency FB"><u>netarchivesuite-users@ml.sbforge.org</u></font></a><font size=3 face="Agency FB">"
<</font><a href="mailto:netarchivesuite-users@ml.sbforge.org"><font size=3 color=blue face="Agency FB"><u>netarchivesuite-users@ml.sbforge.org</u></font></a><font size=3 face="Agency FB">></font><font size=3 color=#5f5f5f face="Agency FB"><br>Date : </font><font size=3 face="Agency FB">30/08/2022
13:41</font><font size=3 color=#5f5f5f face="Agency FB"><br>Objet : </font><font size=3 face="Agency FB">Re:
[Netarchivesuite-users] Limit both number of bytes and number of objects
per domain</font><font size=3 color=#5f5f5f face="Agency FB"><br>Envoyé par : </font><font size=3 face="Agency FB">"NetarchiveSuite-users"
<</font><a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org"><font size=3 color=blue face="Agency FB"><u>netarchivesuite-users-bounces@ml.sbforge.org</u></font></a><font size=3 face="Agency FB">></font><div align=center><hr noshade></div><br><font size=3 face="Agency FB"><br><br></font><font size=3 color=#004080 face="Agency FB"><br>Sorry, I mixed it up, alt. 3 edited below. So I suppose now that alt. 3
is true. And that the value of <i>frontier.queueTotalBudget</i> is irrelevant
if you use <i>quotaenforcer</i>, i.e. if <i><ref bean="quotaenforcer"/></i> is among the fetchProcessors.processors. True?<br> <br>But there is a rumour that you should decide between byte and object limit
– true or false?<br> <br>Regards,</font><font size=3 face="Agency FB"><br>-----<br>Peter Svanberg<br></font><font size=3 color=#004080 face="Agency FB"><br> </font><font size=3 face="Agency FB"><b><br>Från:</b> NetarchiveSuite-users <</font><a href="mailto:netarchivesuite-users-bounces@ml.sbforge.org"><font size=3 color=blue face="Agency FB"><u>netarchivesuite-users-bounces@ml.sbforge.org</u></font></a><font size=3 face="Agency FB">>
<b>För </b>Peter Svanberg<b><br>Skickat:</b> den 29 augusti 2022 14:20<b><br>Till:</b> </font><a href="mailto:netarchivesuite-users@ml.sbforge.org"><font size=3 color=blue face="Agency FB"><u>netarchivesuite-users@ml.sbforge.org</u></font></a><font size=3 face="Agency FB"><b><br>Ämne:</b> [Netarchivesuite-users] Limit both number of bytes and number
of objects per domain<br> <br>Could someone please explain this handling?<br> <br>In a snapshot we want to limit both number of bytes and number of objects
per domain. If you give positive values in GUI for new snapshot harvest,
what is recommended?<br> <br>1. You should not. Why not?<br>2. You must change <i>settings.harvester.scheduler.jobGen.objectLimitIsSetByQuotaEnforcer</i>to
false and change<i><br>settings.harvester.harvesting.harvestReport.class</i> to <i>dk.netarkivet.harvester.harvesting.report.BnfHarvestReport</i>(which
doesn’t assume annotations in crawl log).<br>3. You can keep <i>settings.harvester.scheduler.jobGen.objectLimitIsSetByQuotaEnforcer</i>as
true and it works …? Even though </font><font size=3 color=#004080 face="Agency FB">FRONTIER_QUEUE_TOTAL_BUDGET_PLACEHOLDER
(and hence frontier.queueTotalBudget) is set to infinity?</font><font size=3 face="Agency FB"><i><strike>QUOTA_ENFORCER_GROUP_MAX_FETCH_SUCCES_PLACEHOLDER</strike></i><strike>in
template (and hence <i>quotaenforcer.groupMaxFetchSuccesses</strike></i><strike>)
is set to infinity</strike> (in <i>configureQuotaEnforcer()</i>)?<br> <br>Regards,<br> <br> </font><p><table width=394 style="border-collapse:collapse;"><tr height=8><td width=113 style="border-style:solid;border-color:#000000;border-width:0px 3px 0px 0px;padding:0px 0px;"><td width=278 style="border-style:solid;border-color:#000000;border-width:0px 0px 0px 3px;padding:0px 0px;"><font size=3 face="Agency FB"><b>Peter
Svanberg<br>Technical officer </b><br>Aquisitions and Metadata Department<br>Film, Games, Sheet Music and Web Unit<br> <b><br>National Library of Sweden</b><br>PO Box 5039, SE-102 41 Stockholm<br>Visits: Karlavägen 96, Stockholm<br>+46 10-709 32 78</font><font size=3 color=blue face="Agency FB"><u><br></u></font><a href=mailto:Peter.Svanberg@kb.se><font size=3 color=#0082bf face="Agency FB"><u>Peter.Svanberg@kb.se</u></font></a><font size=3 color=blue face="Agency FB"><u><br></u></font><a href=https://www.kb.se/><font size=3 color=blue face="Agency FB"><u>www.kb.se</u></font></a></table><p><font size=3 face="Agency FB"><br> <br> _______________________________________________<br>NetarchiveSuite-users mailing list</font><font size=3 color=blue face="Agency FB"><u><br></u></font><a href="mailto:NetarchiveSuite-users@ml.sbforge.org"><font size=3 color=blue face="Agency FB"><u>NetarchiveSuite-users@ml.sbforge.org</u></font></a><font size=3 color=blue face="Agency FB"><u><br></u></font><a href="https://ml.sbforge.org/mailman/listinfo/netarchivesuite-users"><font size=3 color=blue face="Agency FB"><u>https://ml.sbforge.org/mailman/listinfo/netarchivesuite-users</u></font></a><div align=center><hr></div><p><font size=3 face="Agency FB">Samedi 17 et dimanche 18 septembre 2022
: <b>la BnF fête la réouverture du site Richelieu</b>, après douze ans
de travaux de rénovation et de modernisation, avec </font><a href="https://www.bnf.fr/fr/agenda/richelieu-le-reveil-reouverture"><font size=3 color=blue face="Agency FB"><b><u>un
parcours de visite</u></b></font></a><font size=3 face="Agency FB"> en compagnie
d’artistes et comédiens l'après-midi, et </font><a href="https://www.bnf.fr/fr/agenda/richelieu-le-reveil-performances-et-lectures"><font size=3 color=blue face="Agency FB"><b><u>des
événements et performances</u></b></font></a><font size=3 face="Agency FB">la soirée. </font><p><font size=3 color=#008000 face="Agency FB"><b>Avant d'imprimer, pensez
à l'environnement.</b></font><font size=2 face="Agency FB">_______________________________________________<br>NetarchiveSuite-users mailing list<br>NetarchiveSuite-users@ml.sbforge.org<br></font><a href="https://ml.sbforge.org/mailman/listinfo/netarchivesuite-users"><font size=2 face="Agency FB">https://ml.sbforge.org/mailman/listinfo/netarchivesuite-users</font></a><font size=2 face="Agency FB"><br></font><p><font face="sans-serif"><hr />
<p>Samedi 17 et dimanche 18 septembre 2022 : <strong>la BnF fête la réouverture du site Richelieu</strong>, après douze ans de travaux de rénovation et de modernisation, avec <strong><a href="https://www.bnf.fr/fr/agenda/richelieu-le-reveil-reouverture">un parcours de visite</a></strong> en compagnie d’artistes et comédiens l'après-midi, et <strong><a href="https://www.bnf.fr/fr/agenda/richelieu-le-reveil-performances-et-lectures">des événements et performances</a></strong> la soirée.
<p style="color:#008000"><strong>Avant d'imprimer, pensez à l'environnement.</strong></p></font>