[Netarchivesuite-users] Lots of -50 status codes

[Colin Samuel Rosenthal]

I know we got rid of a lot -50 codes once we fixed our queue assignment policy to always queue dns lookups in the same queue as urls for the corresponding domain. But all that should be fixed in 5.5 . Do you see any problems with dns lookups?


cheers,

Colin


--
Colin Rosenthal PhD
Senior IT Consultant
Royal Danish Library (Aarhus)
________________________________
From: NetarchiveSuite-users <netarchivesuite-users-bounces at ml.sbforge.org> on behalf of Peter Svanberg <Peter.Svanberg at kb.se>
Sent: Friday, June 14, 2019 11:56:42 AM
To: netarchivesuite-users at ml.sbforge.org
Subject: [Netarchivesuite-users] Lots of -50 status codes

Hello all!

I get quite a lot of -50 status lines in my crawl.log (test snapshot runs, NetarchiveSuite 5.5). I haven’t seen any pattern in which kind of URL:s that gets this – sometimes it is URL:s which redirects to entrance/top level page (should be banned!), sometimes it’s quite ordinary URL:s, often to images, I think. And manual fetching later always works.

One pattern is that it seems as if it either

·         all requests to a certain host are -50, or

·         first a series of 200 and then ditto -50
so they are not intermixed. That could imply some problem – or automatic blocking after a while? – on the host.

What are your experiences?

-----

Peter Svanberg

National Library of Sweden
Phone: +46 10 709 32 78

E-mail: peter.svanberg at kb.se
Web: www.kb.se



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://ml.sbforge.org/pipermail/netarchivesuite-users/attachments/20190614/f72e6959/attachment.html>