IRC-Source Crawler 3.x Progress

IRC-Source Crawler 3.x is currently successfully updating InspIRCd based networks. We will be testing the crawler on various IRCd’s once we have tested the stability of the new crawler. The crawler is designed to scale up quickly with it’s Master/Slave system, so we are planning on using 2 crawl servers at first to see how things go. The new crawler, and scaling, is necessary because the current crawler uses all CPU and memory when crawling which could lead to some of the errors that we have been having with indexing certain networks.

The new crawler is designed to check networks that need to be crawled more often. This should ensure more accurate stats for networks, and overall. There is currently no ETA on when the crawler will come into it’s beta stages and become available as an option, however we will notify you via Twitter and our blog when this happens.
If you are currently interested in trying out the new crawler we are interested in networks with various types of IRCds that can aid in testing. Please contact us at [email protected] if you are interested in trying out the crawler in it’s alpha/development stage.

— IRC-Source Staff

xnite

I am the founder, and lead developer, of IRC-Source. I started IRC-Source in May of 2014 because, although I don't use it as much any more, I still have a deep interest in IRC. I also carry a deep interest in web development, and wanted to write something that would be challenging, and allow me to bring my passions together in a way that would be interesting for others. I run a network called BuddyIM (irc.buddy.im), it's pretty cool, you should chill there.

More Posts - Website - Twitter - LinkedIn - Reddit