Crawler v3.1707181 RC1

We are proud to announce that we finally have a release candidate for crawler v3.x.
Over the next few weeks we will be rolling the crawler out to more networks as we test it with various IRC daemons.

Crawler Changes

The whole project has been revamped since the last time I talked about the new crawler, and the new system appears to be highly stable.

  • The crawler now only stores information about servers that it connects to, and stores an MOTD with them. Once the latest version of the website is rolled out you will be able to see the MOTD of individual servers on a network.
  • The crawler tries to use SSL and falls back to a plain text connection.
  • The crawl servers push out messages to IRC about networks being crawled.
  • The crawl server gives staff more useful error messages to help debug connection issues.
  • The new crawler supports networks that, for some reason, the old crawler didn’t support very well.
  • The new crawl servers can easily be scaled out to an unlimited number of slaves.

IP/Hostname Changes

Once the new crawler is completely in place, the hostnames and IP addresses of the crawl servers will change.
You will be made aware of that change via email if you have signed up for our General Mailing List. You can also keep track of changes on our exempts page.

xnite

I am the founder, and lead developer, of IRC-Source. I started IRC-Source in May of 2014 because, although I don't use it as much any more, I still have a deep interest in IRC. I also carry a deep interest in web development, and wanted to write something that would be challenging, and allow me to bring my passions together in a way that would be interesting for others. I run a network called BuddyIM (irc.buddy.im), it's pretty cool, you should chill there.

More Posts - Website - Twitter - LinkedIn - Reddit