We are proud to announce that we finally have a release candidate for crawler v3.x.
Over the next few weeks we will be rolling the crawler out to more networks as we test it with various IRC daemons.
The whole project has been revamped since the last time I talked about the new crawler, and the new system appears to be highly stable.
- The crawler now only stores information about servers that it connects to, and stores an MOTD with them. Once the latest version of the website is rolled out you will be able to see the MOTD of individual servers on a network.
- The crawler tries to use SSL and falls back to a plain text connection.
- The crawl servers push out messages to IRC about networks being crawled.
- The crawl server gives staff more useful error messages to help debug connection issues.
- The new crawler supports networks that, for some reason, the old crawler didn’t support very well.
- The new crawl servers can easily be scaled out to an unlimited number of slaves.
Once the new crawler is completely in place, the hostnames and IP addresses of the crawl servers will change.
You will be made aware of that change via email if you have signed up for our General Mailing List. You can also keep track of changes on our exempts page.