[Netarchivesuite-devel] [archive-crawler] crawler-commons project

[Søren Vejrup Carlsen]

FYI
________________________________________
Fra: archive-crawler at yahoogroups.com [archive-crawler at yahoogroups.com] På vegne af stack [stack at duboce.net]
Sendt: 20. november 2009 17:48
Til: archive-crawler at yahoogroups.com
Emne: [archive-crawler] crawler-commons project

Hey crawlers:

I was Apachecon in Oakland in early November and was present during a meeting of a few of the open source crawler projects (Ken Krugle for Bixo, Andrzej for Nutch, and the Apache Droids fellow via Skype from Spain).  The topic was what can be done to facilitate sharing of common resources and code amongst the open source crawlers; e.g. robots.txt parsing, url parse, canonicalization, page similarity, etc.  There was no Heritrix representative so I'm writing the list because my guess is that you fellas are probably interested in what was discussed and the in particular, the meetings' outcome.

Here are meetings from the meeting: http://wiki.apache.org/nutch/ApacheConUs2009MeetUp.

Andrzej went ahead and started a crawler-commons project over on google code: http://code.google.com/p/crawler-commons/

Write Ken Krugle or the crawler-commons project if interested in adding commonalities or participating (or me offline if you'd like to know more about the meeting).

Thanks,
St.Ack

P.S. Gordon I tried sending you this stuff offlist but my mail must be stuck in your spam filters.  Go easy.







__._,_.___


Your email settings: Individual Email|Traditional
Change settings via the Web<http://groups.yahoo.com/group/archive-crawler/join;_ylc=X3oDMTJmNGYxZmZoBF9TAzk3NDc2NTkwBGdycElkAzg3NTk4NjcEZ3Jwc3BJZAMxNzA1MDA0OTI0BHNlYwNmdHIEc2xrA3N0bmdzBHN0aW1lAzEyNTg3MzU3MDE-> (Yahoo! ID required)
Change settings via email: Switch delivery to Daily Digest<mailto:archive-crawler-digest at yahoogroups.com?subject=Email%20Delivery:%20Digest> | Switch to Fully Featured<mailto:archive-crawler-fullfeatured at yahoogroups.com?subject=Change%20Delivery%20Format:%20Fully%20Featured>
Visit Your Group <http://groups.yahoo.com/group/archive-crawler;_ylc=X3oDMTJkNW9jcHBwBF9TAzk3NDc2NTkwBGdycElkAzg3NTk4NjcEZ3Jwc3BJZAMxNzA1MDA0OTI0BHNlYwNmdHIEc2xrA2hwZgRzdGltZQMxMjU4NzM1NzAx> | Yahoo! Groups Terms of Use <http://docs.yahoo.com/info/terms/> | Unsubscribe <mailto:archive-crawler-unsubscribe at yahoogroups.com?subject=Unsubscribe>

__,_._,___