<font size=2 face="sans-serif">Hi everyone,</font><br><br><font size=2 face="sans-serif">Just a quick note to let you know
that we have launched a broad crawl test with 5.3.1 at the end of last
week.</font><br><font size=2 face="sans-serif">And everything went smooth: we generated
872 jobs, ran 20 of them using 10 crawlers, job status are consistent</font><br><font size=2 face="sans-serif">and there is nothing wrong with the
broker.</font><br><br><font size=2 face="sans-serif">We have the following configuration:</font><br><font size=2 face="sans-serif">- CentOS 7.3 (which seems to be
similar to Red Hat 4.8)</font><br><font size=2 face="sans-serif">- Java(TM) SE Runtime Environment (build
1.8.0_40-b25) 64-Bit</font><br><font size=2 face="sans-serif">- OpenMQ (MessageQueue5.1)</font><br><br><br><font size=2 face="sans-serif">Maybe more important, we are using this
configuration on the scheduler.</font><br><font size=2 face="sans-serif">
<scheduler></font><br><font size=2 face="sans-serif">
<jobtimeouttime>31536000</jobtimeouttime></font><br><font size=2 face="sans-serif">
<jobgenerationperiode>60</jobgenerationperiode></font><br><font size=2 face="sans-serif">
<jobGen></font><br><font size=2 face="sans-serif">
<class>dk.netarkivet.harvester.scheduler.jobgen.FixedDomainConfigurationCountJobGenerator</class></font><br><font size=2 face="sans-serif">
<objectLimitIsSetByQuotaEnforcer>false</objectLimitIsSetByQuotaEnforcer></font><br><font size=2 face="sans-serif">
<domainConfigSubsetSize>5000</domainConfigSubsetSize></font><br><font size=2 face="sans-serif">
<config></font><br><font size=2 face="sans-serif">
<fixedDomainCountSnapshot>5000</fixedDomainCountSnapshot></font><br><font size=2 face="sans-serif">
<fixedDomainCountFocused>500</fixedDomainCountFocused></font><br><font size=2 face="sans-serif">
<excludeDomainsWithZeroBudget>true</excludeDomainsWithZeroBudget></font><br><font size=2 face="sans-serif">
<postponeUnregisteredChannel>false</postponeUnregisteredChannel></font><br><font size=2 face="sans-serif">
</config></font><br><font size=2 face="sans-serif">
</jobGen></font><br><font size=2 face="sans-serif">
</scheduler></font><br><br><font size=2 face="sans-serif">If I remember well, at KB and ONB, you
are using a different job generator that tries to make homogenous jobs
sizes based</font><br><font size=2 face="sans-serif">on the previous harvest. The one we
are using is making jobs taking the domains in alphabetical order.</font><br><br><font size=2 face="sans-serif">Hope this help,</font><br><br><font size=2 face="sans-serif">Sara</font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">De :
</font><font size=1 face="sans-serif"><aponb@gmx.at></font><br><font size=1 color=#5f5f5f face="sans-serif">A :
</font><font size=1 face="sans-serif"><netarchivesuite-devel@ml.sbforge.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date :
</font><font size=1 face="sans-serif">29/06/2017 11:13</font><br><font size=1 color=#5f5f5f face="sans-serif">Objet :
</font><font size=1 face="sans-serif">Re: [Netarchivesuite-devel]
Multiple jobs submitted simultaneously under 5.3.1</font><br><font size=1 color=#5f5f5f face="sans-serif">Envoyé par :
</font><font size=1 face="sans-serif">Netarchivesuite-devel
<netarchivesuite-devel-bounces@ml.sbforge.org></font><br><hr noshade><br><br><br><font size=3>Hi Sara,<br><br>I forgot to mention that the problems were coming up with our daily crawls.
The intention was to deploy 5.3.1, waiting for some daily crawls, before
starting the broad crawl.<br><br>Thanks for your settings and for telling how your broad crawl will work!<br></font><br><font size=2 face="Arial">Hi Andreas,</font><font size=3><br></font><font size=2 face="Arial"><br>Are your problems coming up because you just launched a broad crawl?</font><font size=3><br></font><font size=2 face="Arial"><br>At BnF, we are still running 5.3.0 with default settings on these parameters:<br>settings.harvester.harvesting.sendReadyInterval on 30s <br>settings.harvester.harvesting.sendReadyDelay on 1000ms</font><font size=3><br></font><font size=2 face="Arial"><br>We are currently testing 5.3.1 on very small crawls (working well)<br>and we will start bigger crawls next week. I'll let you know<br>how it goes.</font><font size=3><br></font><font size=2 face="Arial"><br>Sara </font><font size=3><br><br><br><br></font><font size=1 color=#5f5f5f face="sans-serif"><br>De : </font><a href=mailto:aponb@gmx.at><font size=1 color=blue face="sans-serif"><u><aponb@gmx.at></u></font></a><font size=1 color=#5f5f5f face="sans-serif"><br>A : </font><a href="mailto:netarchivesuite-devel@ml.sbforge.org"><font size=1 color=blue face="sans-serif"><u><netarchivesuite-devel@ml.sbforge.org></u></font></a><font size=1 color=#5f5f5f face="sans-serif"><br>Date : </font><font size=1 face="sans-serif">28/06/2017
11:43</font><font size=1 color=#5f5f5f face="sans-serif"><br>Objet : </font><font size=1 face="sans-serif">[Netarchivesuite-devel]
Multiple jobs submitted simultaneously under 5.3.1</font><font size=1 color=#5f5f5f face="sans-serif"><br>Envoyé par : </font><font size=1 face="sans-serif">Netarchivesuite-devel
</font><a href="mailto:netarchivesuite-devel-bounces@ml.sbforge.org"><font size=1 color=blue face="sans-serif"><u><netarchivesuite-devel-bounces@ml.sbforge.org></u></font></a><font size=3><br></font><hr noshade><font size=3><br><br></font><tt><font size=2><br>If was running Nas Version on 5.3.1 in production and did get a huge <br>number of jobs with the same Configurations submitted. This must be the
<br>behavior of </font></tt><a href="https://sbforge.org/jira/browse/NAS-2614"><tt><font size=2 color=blue><u>https://sbforge.org/jira/browse/NAS-2614</u></font></tt></a><tt><font size=2>which
was fixed for <br>Version 5.3.1 - the strange thing is, that I had not any problems in <br>Version 5.3.0.<br><br>Is anyone experiencing the same issue?<br>As suggested I set settings.harvester.harvesting.sendReadyInterval to <br>300 and I am using settings.harvester.harvesting.sendReadyDelay with <br>value 300<br><br>Also the HarvestJobManagerApplication dies with OutOfMemory Exception,
<br>even when started with parameter -Xmx4096m<br><br>20:28:11.823 ERROR d.n.c.lifecycle.PeriodicTaskExecutor - Task threw <br>exception: java.lang.OutOfMemoryError: Java heap space<br>java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: <br>Java heap space<br> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
<br>~[na:1.8.0_77]<br> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
<br>~[na:1.8.0_77]<br> at <br>dk.netarkivet.common.lifecycle.PeriodicTaskExecutor.checkExecution(PeriodicTaskExecutor.java:171)
<br>[common-core-5.3.1.jar:UNKNOWN_REVISION]<br> at <br>dk.netarkivet.common.lifecycle.PeriodicTaskExecutor.access$500(PeriodicTaskExecutor.java:47)
<br>[common-core-5.3.1.jar:UNKNOWN_REVISION]<br> at <br>dk.netarkivet.common.lifecycle.PeriodicTaskExecutor$1.run(PeriodicTaskExecutor.java:152)
<br>[common-core-5.3.1.jar:UNKNOWN_REVISION]<br><br>Do you have any thoughts on this?<br>Regards<br>a.<br><br>_______________________________________________<br>Netarchivesuite-devel mailing list</font></tt><tt><font size=2 color=blue><u><br></u></font></tt><a href="mailto:Netarchivesuite-devel@ml.sbforge.org"><tt><font size=2 color=blue><u>Netarchivesuite-devel@ml.sbforge.org</u></font></tt></a><font size=3 color=blue><u><br></u></font><a href="https://ml.sbforge.org/mailman/listinfo/netarchivesuite-devel"><tt><font size=2 color=blue><u>https://ml.sbforge.org/mailman/listinfo/netarchivesuite-devel</u></font></tt></a><font size=3><br></font><font size=3 face="sans-serif"><br></font><hr><p><font size=3 face="sans-serif">Expositions :</font><font size=3 color=blue face="sans-serif"><b><i><u><br></u></i></b></font><a href=http://www.bnf.fr/fr/evenements_et_culture/anx_expositions/f.monde_topor.html><font size=3 color=blue face="sans-serif"><b><i><u>Le
monde selon Topor</u></i></b></font></a><font size=3 face="sans-serif">- jusqu'au 16 juillet 2017 - BnF - François-Mitterrand</font><font size=3 color=blue face="sans-serif"><b><i><u><br></u></i></b></font><a href=http://www.bnf.fr/fr/evenements_et_culture/anx_expositions/f.bibliotheque_la_nuit.html><font size=3 color=blue face="sans-serif"><b><i><u>La
bibliothèque, la nuit – Bibliothèques mythiques en réalité virtuelle </u></i></b></font></a><font size=3 face="sans-serif">-
jusqu'au 13 août 2017 - BnF - François-Mitterrand</font><p><font size=3 color=#008000 face="sans-serif"><b>Avant d'imprimer, pensez
à l'environnement.</b></font><p><font size=3><br></font><br><tt><font size=3>_______________________________________________<br>Netarchivesuite-devel mailing list<br></font></tt><a href="mailto:Netarchivesuite-devel@ml.sbforge.org"><tt><font size=3 color=blue><u>Netarchivesuite-devel@ml.sbforge.org</u></font></tt></a><tt><font size=3><br></font></tt><a href="https://ml.sbforge.org/mailman/listinfo/netarchivesuite-devel"><tt><font size=3 color=blue><u>https://ml.sbforge.org/mailman/listinfo/netarchivesuite-devel</u></font></tt></a><tt><font size=3><br></font></tt><p><tt><font size=2>_______________________________________________<br>Netarchivesuite-devel mailing list<br>Netarchivesuite-devel@ml.sbforge.org<br></font></tt><a href="https://ml.sbforge.org/mailman/listinfo/netarchivesuite-devel"><tt><font size=2>https://ml.sbforge.org/mailman/listinfo/netarchivesuite-devel</font></tt></a><tt><font size=2><br></font></tt><p><font face="sans-serif"><hr />
<p>Expositions :<br />
<strong><em><a href="http://www.bnf.fr/fr/evenements_et_culture/anx_expositions/f.monde_topor.html">Le monde selon Topor</a></em></strong> - jusqu'au 16 juillet 2017 - BnF - François-Mitterrand<br />
<strong><em><a href="http://www.bnf.fr/fr/evenements_et_culture/anx_expositions/f.bibliotheque_la_nuit.html">La bibliothèque, la nuit – Bibliothèques mythiques en réalité virtuelle </a></em></strong> - jusqu'au 13 août 2017 - BnF - François-Mitterrand</p>
<p style="color:#008000"><strong>Avant d'imprimer, pensez à l'environnement.</strong></p></font>