On January 8th the crawler was unable to connect to some networks because it was blacklisted by EFnet RBL. This also may have lead to being unable to connect to some, primarily InspIRCd based, networks for several days after. Upon discovery of these facts I contacted the EFnet RBL team about the issue, asking why we were blacklisted and how it could be prevented going forward, even though the IP had already appeared to be removed from the blacklist. As it would have it, we weren’t actually blacklisted at all, here’s what Gavin at EFnetRBL had to say:
Unfortunately, the domain expired for a day or so, and godaddy’s wildcard dns caused some servers to incorrectly ban users.
There’s a couple ways to protect against this
1) bopm config has a “ban_unknown” option. a proper config should have this set to “no”, and specify which responses should be banned (ie – 127.0.0.*). only servers that had ban_unknown=yes would have banned based on the wildcard.
2) whichever servers you connect to, request kline/dline exempt for your ip. this way even if your bot gets banned for whatever reason, it can still connect.
Please let this incident serve as a reminder to exempt our crawler hostname in your IRCd configurations.