[Netarchivesuite-users] Inconsistency in the count of Bytes and Documents Harvested

[Bjarne Andersen]

In my opinion this is a bug. It should be either with or without duplicates.

OR - maybe a better solution would be to report figures for both numbers and bytes - twice including and excluding duplicates - that would give 4 figures:
 - number of bytes downloaded from the web-servers
 - number of objects downloaded from the web-servers
 - number of bytes stored in the archive
 - number of objects stored in the archive

That would allow the interface also to present these two figures (calculated from the 4 above)
 - number of bytes discarded through deduplication
 - number of objects discarded through deduplication

If one wanted to do some calculation on the two latter (deduplication) maybe they should be stored in the history-database as well

best
Bjarne

________________________________________
Fra: netarchivesuite-users-bounces at lists.gforge.statsbiblioteket.dk [netarchivesuite-users-bounces at lists.gforge.statsbiblioteket.dk] På vegne af sara.aubry at bnf.fr [sara.aubry at bnf.fr]
Sendt: 16. februar 2011 11:25
Til: netarchivesuite-users at lists.gforge.statsbiblioteket.dk
Emne: [Netarchivesuite-users] Inconsistency in the count of Bytes and   Documents Harvested

Hello everyone,

While running tests on the last features Nicolas has developped,  we have
noticed that the counts of
Bytes Harvested and Documents Harvested from the crawl.log that appear on
the page "Details for Job XXX" are not consistent:

- Documents Harvested = the number of URL (i.e. lines in the crawl.log)
matching the domain pattern, including duplicates,

- Bytes Harvested matches = the total of content-size for all URL matching
the domain pattern, excluding duplicates.

Has this been done on purpose? Or is a bug?
If it is a bug, should we fix it?

Sara



Avant d'imprimer, pensez ? l'environnement.