[Netarchivesuite-users] DNS preparation, domain state active?

clara.wiatrowski at bnf.fr clara.wiatrowski at bnf.fr
Tue Nov 2 18:20:26 CET 2021


Hi Peter, 
It's me. According to BnF an active domain is a domain with a valid 
config.
When there is no DNS response we set the default config associated to the 
domain like this one : 





Max number of objects / bytes (depended on your crawl settings) is set to 
zero (and we had a comment).
So NAS won't take the domain.

Do you need more infos ?

Best,
Clara Wiatrowski



De :    "Peter Svanberg" <Peter.Svanberg at kb.se>
A :     "'netarchivesuite-users at ml.sbforge.org'" 
<netarchivesuite-users at ml.sbforge.org>
Date :  02/11/2021 17:11
Objet : [Netarchivesuite-users] DNS preparation, domain state active?
Envoyé par :    "NetarchiveSuite-users" 
<netarchivesuite-users-bounces at ml.sbforge.org>



Someone (Tue?) talked about a domain having some status, “active” and 
others. I can’t find that, please explain.
 
/Peter
 
 
 
Från: Peter Svanberg 
Skickat: den 2 november 2021 09:26
Till: 'netarchivesuite-users at ml.sbforge.org' 
<netarchivesuite-users at ml.sbforge.org>
Ämne: DNS in NAS/Heritrik
 
Hello!
 
In preparing for a new broad crawl we gather info on new domain names. 
Some issues around that.
 
1) I have a memory of reading about some of you had done some scripts 
concerning domains and seeds. What was that about? What around this does 
NAS/Heritrix not handle on its own?
 
2) Well, one thing we’ve found is that NAS assumes that a domain X answers 
on URL http://www.X, but that is not true. We found hundreds of domains 
which have no www.X host but answers on http://X. Maybe this should be 
changed in some way in NAS?
 
Med vänlig hälsning
 
 


Peter Svanberg
Teknisk handläggare
Insamling och metadata
Insamling 1
 
Kungliga biblioteket
Box 5039, 102 41 Stockholm
Besöksadress: Karlavägen 96, Stockholm
+46 10 709 32 78
Peter.Svanberg at kb.se
www.kb.se
 _______________________________________________
NetarchiveSuite-users mailing list
NetarchiveSuite-users at ml.sbforge.org
https://ml.sbforge.org/mailman/listinfo/netarchivesuite-users


Découvrez toute la programmation culturelle de la rentrée à la BnF 
Pass BnF lecture/culture  : bibliothèques, expositions, conférences, concerts en illimité pour 15 € / an  –  Acheter en ligne Avant d'imprimer, pensez à l'environnement. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://ml.sbforge.org/pipermail/netarchivesuite-users/attachments/20211102/6fbc4e60/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 12822 bytes
Desc: not available
URL: <https://ml.sbforge.org/pipermail/netarchivesuite-users/attachments/20211102/6fbc4e60/attachment-0001.gif>


More information about the NetarchiveSuite-users mailing list