[Netarchivesuite-users] Limit both number of bytes and number of objects per domain

[Peter Svanberg]

Sorry, I mixed it up, alt. 3 edited below. So I suppose now that alt. 3 is true. And that the value of frontier.queueTotalBudget is irrelevant if you use quotaenforcer, i.e. if <ref bean="quotaenforcer"/>  is among the fetchProcessors.processors. True?

But there is a rumour that you should decide between byte and object limit - true or false?

Regards,
-----
Peter Svanberg


Från: NetarchiveSuite-users <netarchivesuite-users-bounces at ml.sbforge.org> För Peter Svanberg
Skickat: den 29 augusti 2022 14:20
Till: netarchivesuite-users at ml.sbforge.org
Ämne: [Netarchivesuite-users] Limit both number of bytes and number of objects per domain

Could someone please explain this handling?

In a snapshot we want to limit both number of bytes and number of objects per domain. If you give positive values in GUI for new snapshot harvest, what is recommended?


1.       You should not. Why not?

2.       You must change settings.harvester.scheduler.jobGen.objectLimitIsSetByQuotaEnforcer to false and change
settings.harvester.harvesting.harvestReport.class to dk.netarkivet.harvester.harvesting.report.BnfHarvestReport (which doesn't assume annotations in crawl log).

3.       You can keep settings.harvester.scheduler.jobGen.objectLimitIsSetByQuotaEnforcer as true and it works ...? Even though FRONTIER_QUEUE_TOTAL_BUDGET_PLACEHOLDER (and hence frontier.queueTotalBudget) is set to infinity?QUOTA_ENFORCER_GROUP_MAX_FETCH_SUCCES_PLACEHOLDER in template (and hence quotaenforcer.groupMaxFetchSuccesses) is set to infinity (in configureQuotaEnforcer())?

Regards,


[KB Logo]<https://www.kb.se/>

Peter Svanberg
Technical officer
Aquisitions and Metadata Department
Film, Games, Sheet Music and Web Unit

National Library of Sweden
PO Box 5039, SE-102 41 Stockholm
Visits: Karlavägen 96, Stockholm
+46 10-709 32 78
Peter.Svanberg at kb.se<mailto:Peter.Svanberg at kb.se>
www.kb.se<https://www.kb.se/>



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://ml.sbforge.org/pipermail/netarchivesuite-users/attachments/20220830/4b8504f1/attachment.html>