Mail problems solved

For an unknown period of time the email system on the IRC-Source website was not working. We now have a new email provider and emails will be sent out as normal once again.
For accounts awaiting verification, I have gone ahead and manually verified them. If you were waiting on a password reset, or any other communications from the website, please try again.
If you are still not receiving emails please check your spam filter, or join the IRC-Source help channel on irc.AltSociety.co in #IRC-Source.

IRC-Source will breath new life!

When I started this project, I had a vision for what an IRC search engine should be. That vision has not yet been met. Through my adult life I have been struggling with bipolar disorder (extreme mood swings, depression and manic). My disorder has caused me to neglect this, and many other projects that I’ve started throughout my adult life. I have started getting treatment for my disorder, and I feel better than ever before! I feel like my vision can finally be realized.

The future of IRC-Source

I am rebuilding much of the code from the ground up, starting with the crawler bot. I won’t bother wasting your time with the could-be’s and the dreams I hope to achieve. Instead I will tell you about the realistic goals that I have in mind for the upcoming weeks. Last night, only 3 days into my treatment, I started working on the new crawler bot. The current production version of the crawler is an ugly mess written in a combination of PHP, PERL and Bash scripting. The new crawler that I have begun working on is a pure PERL implementation which I hope will be faster, and much more effective. Due to unforeseen circumstances, many of the requests to maintain network information have gone unanswered. A friend of mine tried to help out, but failed to reach out to many of the people who have requested this access. The new crawler, upon doing it’s normal crawls, will reach out to people who have made these requests, as well as future requests during it’s normal crawl operations. In other words, I hope to have this system automated in the very near future.

A new scoring system will be implemented, and it will take into account the following factors when scoring a network:

  • User count
  • Count of indexed channels (channels that have more than 2 people in them)
  • IRC Operator count, as well as the balance between IRC Operators and users.
  • IRCv3 Compliance
  • A small fraction of the score will be based on whether or not certain information about the network is available, which in most cases will require someone to maintain its information.

I would like to avoid giving specific details on exactly how networks are scored, as that would make it easier for people to manipulate the scoring system in their favor. Scores will be based on statistical information collected over the previous 1 week period, and re-calculated after each crawl. I believe this will give a more accurate depiction of each network.

I have recently modified the behavior of the graphs for each network, and instead of showing average counts they now show the max counts for each period. The yearly graphs now show each network’s stats by week instead of the month.

Once the crawler is completed, which should be by the end of 2015, featured networks will be added to the home page. IRC-Source will feature a network for each month, week, day and hour.
If all goes as intended, I plan an update to record statistics for channels meeting a certain criteria. Channels will need to have at least 20 users in them, and visible in the /list at the time of crawl, to be tracked. At this time there are no plans to add maintainers for channels, but this could change in the future. From the list of tracked channels, I hope to add featured channels just as planned for featured networks.

A new ticket system will be put into place to help better assist you when you need help from IRC-Source staff. I have some experience with the system I plan on putting into place. I believe it should be sufficient for making sure that help requests are handled properly, and within a timely manner (unlike in the past). This also means that the forums will be removed from IRC-Source (nobody uses them outside of the help forum anyway).

I know I have not kept up on many of my plans previously, but I feel like I can finally concentrate on this project again. I know that my goals are realistic, and I feel like my vision will be realized and I hope that you will all really enjoy it. Here’s to a better year ahead of us, and thank you to those who have seen this through and stuck by my side throughout my time on this project.

Cheers mates! ~ xnite

Development Log – Week of 2015-05-03

In case you haven’t noticed, I haven’t been keeping up on the development logs. I realize that many of the members here like to be informed of what is new and what is in the works, and I think it’s very important to keep a line of communication open during the development process.

Obstacles

Right now I am performing a big migration to a new VM that I spun up for IRC-Source. A problem that I have run into is that the current crawler is not running efficiently, and I don’t want this problem to  persist on a new server. I had intended on phasing it out in favor of a new crawler anyway, so the sooner the better.

Currently working on…

New crawler bot

The current crawler is too basic, and single threaded, there are too many scripts involved in the process (crawl list builder, crawler bot, parser). The new crawler will be faster, more efficient, and scalable.

Wishful thinking

The new crawler daemon will handle crawling, parsing, and sending data to an API on the website. The new crawler will have the ability to thread out, crawling more networks, faster. The goal is to have the entire indexing process done in under 5 minutes per run, but pretty much anything will be better than current run time.

Not so wishful thinking after all

I’m not far from completing the new crawler, there are a few things to polish off before it’s ready for testing on the production database though. I have opened the floor for beta testing of the new crawler when it is ready. The goal date for beta testing is May 25th, and full on integration by June 30th.

Development Roadmap – Q2:2015

It took me a while to put together this quarter’s Development Roadmap without giving away too many details or secrets about some of the upcoming work in this project.
I can now discuss a little bit about the crawler bot, and something new going on with the rating system. I want you all to know that I am very alive, and this project is still active.

Q1 Setbacks

Last month I got everything stuck into a bit of a migration phase, so I’m still moving some stuff over to a new server. One of the problems that exists on the previous server is that the crawl bot isn’t running as efficiently as it could be, so I’m really pushing to get version 2.x of the crawler done before I complete the migration process once and for all. So on top of keeping up on support requests, keeping up on development and keeping up on maintenance of the servers, things have been moving along slower than I had hoped. Some things expected to be done last quarter were not done, and for that I do apologize.

Q2 Expectations

I have some big plans this quarter, and I think that the important bits are going to work out on schedule this time, given that there is still two months left and quite a bit of the work has been started. Around Quarter 3 I’ll probably be working out the rough edges and starting on the public API.

While I’m working on the new crawler bot I will also need to rework some of the regular expressions and this may cause some of the data parsing to break for some networks. If, after the migration to the new crawler, you notice that data does not look correct please notify me via the support forum so that I can get it fixed.

Short Term Goals

These are goals that are planned to be wrapped up by the end of Q2:2015 (Q2=April 01-June 30): (Items that are crossed out have been completed)

  1. Network Administration Panel
    1. Manage users who have access to administrate networks
    2. Grant users the ability to “claim” networks securely by a process which connects to the network and verifies IRCOp access.
  2. Network Listings
    1. Featured Networks
    2. Categorize & Tag Networks
    3. Browse or Search by categories & tags
    4. Identify which listings are SFW and which ones are NSFW
    5. Split network page to pages?
  3. Improved Scoring System
    1. Removing cap on maximum score
    2. Less data driven, more people driven
  4. Crawler 2.x
    1. Crawler threads out to crawl up to 100 networks at the same time.
      1. Crawling process is expected to be under 15 minutes, where current is about 45-50 minutes.
      2. In the future crawler may attempt to stay connected to the network to avoid flooding connections.
    2. The crawler is being entirely re-written in PERL to interact with a new web API for crawling.
      1. Between the web API and threading multiple servers could be used in the future to index all networks within as little as 2 minutes.
    3. Crawler will daemonize
      1. Crawler will always check if networks need to be indexed and stay on top of the task instead of checking every 30 minutes.
      2. In the future crawler bots could try to always remain connected to avoid flooding connections.
    4. During testing period you may see a test crawl bot 2.x connecting from [email protected], feel free to add an exception or ban the testing host name including the ident.

Long Term Goals

These are the current long term goals that are planned for completion, hopefully, by the end of 2015.

  1. Administration & Moderation Panels (Q2:2015 – Q3:2015)
    1. Allow volunteer staff to help trigger network verification process and provide the tools necessary for them to help get support requests moved along quicker.
      1. Support requests can be done quicker.
      2. Gives members an opportunity to get their foot in the door involving themselves in the process
    2. Need to work out volunteer agreement before I can start accepting volunteers.
  2. Channel Listings
    1. Channel Pages w/ Graphs for channels where there is an established maintainer and contact information is provided.
    2. Featured Channels
    3. Categorize & Tag Channels
    4. Identify which listings are SFW and which ones are NSFW
  3. Public API (Q3:2015?)
    1. View Network Information
    2. View Channel Information
    3. View Public user profile information
  4. Improved crawl bots – (Imported from unfinished tasks in previous quarter, please see Crawler 2.x in short term goals)
    1. Identify Janus linked networks where possible & automatically add Janus flag to network record

Development Delays

My apologies, I have not been very active in development over the past week as I have had a family emergency arise.
I will be out of town this week, and I am sorry for any inconvenience this may cause.
In my free time, however, I have been working on some of the smaller tasks such as working out kinks in the crawler bot, such as the bot not properly detecting when it has failed to connect or accidentally marking it self as having been banned from a network where it has not been.
I will also still be checking support requests while I am out of town, however less frequently than I typically would while at home.

I hope you will all understand, and as always keep checking back for new updates.

Development Log – Week of 2015-02-22

Previous weeks development got off to a very slow start with a few minor mishaps & downtime, but I’m excited to kick this week off by announcing a few new updates:

Firstly, there is now a Support Forum where you can make support requests and request access to maintain networks. This will remain online, along with the Off-Topic forum if anyone actually cares to post there. I have also added paging support to the network browse/search page which should speed up page load times on mobile devices and over all improve the visual experience.

Some of the things that I had anticipated did not get done last week, and are now being moved over to this weeks agenda. Featured networks is going to be a big focus area, as well as continuing drafting out different ideas for an automated verification process for maintainers that provides both convenience and security (two things that don’t typically compliment each other).
Flags and categories is still going to be a huge focus considering that paging is now completed, and their implementation should now be simplified.

I don’t see all of this taking too long, if there is time at the end of the week I will try to fit in some of the cosmetic changes that have been requested before hitting some of the heavier development next week.

Outage Report – 2015/02/18

We experienced a brief outage this morning where the MySQL database server was unreachable for about 7 hours, in which time the crawlers also experienced down time.
The issue has been resolved, and I am currently working on migrating the database server over to the local network.

If you have any questions or concerns related to this outage,  please leave your feedback in the comments below.

Development Log – Week of 2015-02-15

Over the previous week I had been shuffling to reproduce the features of the previous version of this website as quickly, and efficiently as possible. WordPress provides a strong basis for my work, and migrating the old code over to plugin’s for WordPress has been much easier and faster than originally anticipated. Because of this, over the last week, I have been able to work on the User Account pages, as well as network maintenance. Things that I once thought would take until about the end of March may only take another week or two, and after a 25.5 hour spree of binge coding, I decided to give it a rest for the weekend.

Development picks back up on Monday, and I hope to improve listings with flags & categories. The new search is good for finding networks with keywords, but I think it’s important to categorize networks and allow people to browse by category. The flags system will contain items which I think you should know about a network. These flags will include whether or not the network has Janus links, or if it’s an Adults only network, “XDCC Friendly” networks will also be marked as such.

Over the past week I think manually verifying networks has not been working out so well, so I’m excited to try and finally roll out an automated process this week, and I will of course be keeping a close eye on it to make sure everything goes smoothly.

Some people have requested that I allow networks to start uploading images as part of their network pages, since this is a purely cosmetic difference this isn’t top priority however I will place it into the road map for future updates.

Paging of network listings has been another widely requested feature, this is something which will be implemented, hopefully, this week.

Featured networks is something that I hope to work on this week, and I would like to feature networks by the hour, day, week and month at random. First, however, my goal is to make sure that networks are properly identified as 18+ so that those don’t get mixed in with featured networks as they aren’t suitable for all audiences.

As a note to those of you who are joining the support channel: If you don’t receive an answer right away, I’m sorry I may not be around all the time. Please be patient and I’ll try to handle all requests in the order that they are received.

You can always leave a comment on these development posts and I’ll read them and try to respond if you have any feature requests or suggestions. Your feedback is always appreciated, and I will try to implement feature requests by demand.

Development Roadmap – Q1:2015

Introduction

This road map covers current short-term & long-term goals for the development of IRC-Source! This document may be expanded upon as we continue development on IRC-Source, so I’m going to try and keep it plain and simple, and cover what I’m working on right now, what has been done since the last “stable” update, and where I hope to head things in the upcoming months.

All new back-end framework

In case you haven’t poked around & already noticed, the entire website is now powered by WordPress, and I have ported over code from the current IRC-Source! code base as WordPress plugins.
I finally made this decision for a few different reasons, the primary being that I wanted a solid, well backed, framework for handling my sites pages and posts without re-inventing the wheel so many times I could make my own big-rig. Switching to WordPress, which I have already been using for my other sites 5+ years, allows me to focus more on the code that actually makes IRC-Source what it is and less on things like user authentication, paging, and a theme engine.
Being a single developer can be very stressful, so by dumping the extra work load I have finally been given the motivation and encouragement that I needed to get this project back on it’s feet.

Good News / Bad News

The migration from the Stand-alone project to a WordPress based project means there are obviously going to be some drawbacks, and of course quite a few benefits.
The greatest benefits are: not having to rewrite an entire code base to emulate most of the stuff that WordPress already does for us, and this means that development will be much faster going forward.
The drawback is that, for now, most of the sites features have not been re-implemented into the current code base. To make up for this, I have implemented the, long awaited for, network submissions page, and you will now be able to add (but unfortunately not manage yet) your IRC network to our listings. This process, of course, is in it’s testing phase and if you have any trouble adding your network please contact me via our web chat.

Short Term Goals

These are goals that are planned to be wrapped up by the end of Q1:2015 (Q1=January 01-March 31): (Items that are crossed out have been completed)

  1. Restore all the features of previous versions of the web site including, but not limited to:
    1. Network Listings
    2. Channel listings
    3. Featured networks
    4. Server lists
    5. Networks & Channel Search
  2. User Control Panel
    1. Manage profile & account information
    2. Manage networks
  3. Network Administration Panel
    1. Edit / Update network information
    2. Manage users who have access to administrate networks
    3. Grant users the ability to “claim” networks securely by a process which connects to the network and verifies IRCOp access.
  4. Improved Listings
    1. Categorize & Tag Networks
    2. Categorize & Tag Channels
    3. Browse or Search by categories & tags
    4. Identify which listings are SFW and which ones are NSFW

Long Term Goals

These are the current long term goals that are planned for completion, hopefully, by the end of 2015.

  1.  Public API
    1. View Network Information
    2. View Channel Information
    3. View Public user profile information
  2. Improved crawl bots
    1. Speed up index parsing process
    2. Speed up total indexing process (currently 15 minutes to index all 111 listed networks).
    3. Identify Janus linked networks where possible & automatically add Janus flag to network record

Feel free to offer feed back, or feature requests below in the comments & happy channel surfing.

End of Year Analytics

Starting in late May of 2014 IRC-Source! has picked up quite a bit of traffic. Throughout 2015 it is my goal to improve the project by laying out a brand new frame work and continuing efforts toward development on the website.