Expected Downtime

We are expecting downtime for all web, and crawl services sometime between the window of 6AM – 2PM UTC-0 on Friday, August 18th 2017.
During this downtime we will be performing maintenance on our database and web infrastructure which is expected to last at least 1 hour while we get everything migrated and reconfigured.

All systems go – July 31st

As of July 31st we will be putting the new crawl servers into effect. The new servers will have the hostnames yin.ircs.me and yang.ircs.me.
The new addresses for the crawl servers are:

yin.ircs.me
49.56.142.157
2607:5300:201:3100::13dd

yang.ircs.me
149.56.142.163
2607:5300:201:3100::299d

Please update the exempts in your IRCd configurations to include these IP addresses before July 31st to avoid any disruption in crawling your network.

New Features

  1. The new crawler contains a master-slave system for added redundancy in crawls.
  2. The new crawler grabs up MOTD for each individual server, and will be accompanied by a website update.
  3. The new crawler allows staff more control over the index and status of a network’s index.
  4. The new crawler is able to handle networks which previously the crawler struggled with.
  5. The new crawler is much more fault tolerant.
  6. The new crawler is able to crawl at lightning speeds!

 

IRC-Source Crawler 3.x Progress

IRC-Source Crawler 3.x is currently successfully updating InspIRCd based networks. We will be testing the crawler on various IRCd’s once we have tested the stability of the new crawler. The crawler is designed to scale up quickly with it’s Master/Slave system, so we are planning on using 2 crawl servers at first to see how things go. The new crawler, and scaling, is necessary because the current crawler uses all CPU and memory when crawling which could lead to some of the errors that we have been having with indexing certain networks.

Continue reading “IRC-Source Crawler 3.x Progress”

Happy Birthday to IRC-Source

The first ever recorded line of history in the IRC-Source database is 2014-05-27 23:57:48 UTC-6. We consider this date to be the birthday of IRC-Source. In recognition of IRC-Source turning 3, here is a brief history of IRC-Source, and some of our goals.

In 2016 2 new members were added to the IRC-Source team, James Lu and Ken Spencer.

Continue reading “Happy Birthday to IRC-Source”

Missing Crawler Data Post Mortem

Incident Details

Last night we tried to only crawl networks once per hour and wound up with missing crawler data. Usually we crawl at the beginning of the hour, then again 30 minutes later to crawl missed networks.
Due to a slight time difference between the crawl server and the database server some networks were being indexed at the end of the hour.
We crawl networks backwards from the last time each time. Meaning if Freenode was first one hour, then it will be last the next, so on and so forth. This caused data on large networks to be crawled at the end of the hour every other hour.

Conclusion / Fix

We have implemented an ntp client daemon on both servers, but just in case of future time desyncs we start indexing 1 minute after the hour. We will be monitoring the situation, and if this becomes a problem we will set it back.

May Updates

Community Updates

A fix for the forums has been implemented which now allows for any member to post to the forums.
Please remember that forums are still in alpha stages, and many features may be missing or incomplete.

Forums

The forums activity has been pretty low recently, we could really use your help to get things going even if you just come on to post an introduction for yourself and/or your network. Every last post helps get our community moving along.

Graphs

New graphs have been added, and some changed to display information a little bit better.
We now have 7 day history, last 30 days, and more. You will also notice that the home page displays all statistics over the past 24 hours.
You may embed these ne graphs into your website to show off your statistics to your visitors.

Development Updates

 Bug Fixes

  • Forums have been fixed so members can now create posts and comment.
  • Login now allows email OR username, unless migrating your account from our old WordPress website.
  • CAPTCHA issues have been resolved by creating a custom CAPTCHA solution
  • Various other bug fixes and text updates

Other Improvements

  • Search pages start off clean
  • Faster sessiona handling + persistent sessions across each web server.
  • IRCCloud links have been added to channel and network pages
  • Patrons get an ad-free role allowing them to bypass advertisements
  • Changed markup to allow linking usernames by using /u/username for example.
  • Improved website by loading form responses with jquery instead of using nasty redirects.
  • Various SEO improvements to increase search visability.
  • Various performance optimizations.

Into the future

We are currently working on an API for IRC-Source, and will release more details once it’s finished.
The forums and moderation panels still need some work, and I’ve mostly been working on the backend (Admin Panel) lately. Unfortunately our advertiser (Google Adsense) has laid out some ground rules for us to abide by in order to keep ads up, which currently fund the site. If you have even $1 a month to spare, please become a patron.

Goals made, promises kept

After making our $20 goal we made the promise to split the crawler up onto multiple servers to reduce crawl load. We also made the promise to enable IPv6 support on the crawl bot, which we have done. We also got a new crawl server, check exempts for up to date exempt blocks.

 

New Content Policies

Unfortunately due to restrictions by our ads provider,the only funding this website currently has, new policies had to be put into place to avoid losing that funding.
Without ads this site can’t stay alive, and to be quite honest it doesn’t even make enough to pay the $120/mo server bills. So to ensure the survival of IRC-Source new content policies had to be put into place and can be found here.

To be fully transparent here, we don’t really want to do this but we have to. So here is how we have/are going to handle this issue:

  • Our new content policies only align with Google’s content policies for Adsense. We have not added anything extra.
  • Channels that violate the content policies will be hidden from our website, but we will keep networks out of it.
  • If a network is absolutely dedicated to something that violates our content policies, such as networks dedicated to content of adult nature (sex, porn, etc.) then the network may become suspended as a last resort measure.
    • We will do everything we possibly can to keep an entire network from being hidden from the site but in some cases it may be unavoidable.
  • We will continue to work with Google to take down pages that violate their content policies.
  • We will continue to try to raise funding for IRC-Source through Patreon, where Patrons receive benefits such as an ad-free experience and special user rank on the forums/profile.
    • Once $150/mo is earned through Patreon we will disable ads for every visitor site-wide. IRC-Source is non-profit, we are just interested in paying the server bills.
  • A new report button will be put into place to help let us know where there might be content in violation of the new policies.

Once again, we hate to do this but we do not have a choice. It’s either this or we lose funding for the website. Alternatively we could find an ad provider that doesn’t require us to censor our content, but unfortunately those advertisers tend to have low quality advertisements that are not suitable for IRC-Source.

♥ IRC-Source Staff