/r/irc, what's the best way of logging tens of thousands of IRC Channels? reddit / irc ---------------------------------------------------------------------- darknyan · 2012-11-25 02:24 Recently, I was interested in making a small database of public communications for a class, and I wanted to log about a couple thousand channels into any format. What would be the best way of doing so? ---------------------------------------------------------------------- NoGreatReason · 2012-11-25 02:54 Cool idea and I'm interested to see what others suggest. How do you log even a few irc channels? Juat Mirc or any basic client? ---------------------------------------------------------------------- tobsn · 2012-11-25 03:09 servers limit you to join ~20-40 channels per client. you would need one IP per client. either you get one or multiple dedicated/big nodes servers with tons of IPv6 IPs on them (you won't get many IPv4 at any big hoster without handing over some serious cash, and ircds can handle v6 for 14-16 years now), or you get tons of single instances on a cloud network like ec2. latter will be very expensive. your best bet is ipv6 on one or multiple big machines at a serious hoster like softlayer, which happily supplies your with a quadzillion v6 addresses. for the client you should use a barebone irc lib and then run it in something that uses low memory and CPU (like writing it in cpp and compiling it with CPU flags etc. to get it very very tiny and efficient. next you also want to stuff the data into something that can be easy clustered and scaled without problems because you will accumulate a lot of data. mongodb is very easy scalable (yes, haha, mongo is "webscale"), if you try it out you'll see. you might be able to write a small proxy to which the irc clients write to and itself queues and batch commits to the database, as a handler in the middle to get rid of write tasks/wait to the DB on the irc client. that's my thought on it. just keep in mind if you go with the cheap version you might spend some money on a big server and traffic might get a problem at some point but it's def. doable. generally, I have to say I hate this kind of service because I tend to find my nick all over google blurting out random garbage in random irc channels for the world to read, and I never gave the consent to use it as content for someone's IRC spider. ---------------------------------------------------------------------- darknyan · 2012-11-25 03:14 It's not going to be public, as far as the database is concerned, but in terms of IPv6, my current VPS provider can provide thousands. If it was public, I would provide the logs to the Tribeforth foundation so it could be used in semantic debating. ---------------------------------------------------------------------- tobsn · 2012-11-25 03:17 well even if not public, I did not give you access ;) anyway, do ipv6, next problem will be bandwidth over hundreds of clients. :) ---------------------------------------------------------------------- frumious · 2012-11-25 03:18 Write or adapt an existing IRC bot. If you have any programming ability doing this is fairly trivial. Note, there is what I believe to be a bot doing just this king of driftnet collection on snoonet and it's been kicked from some of the channels where the humans don't appreciate this kind of monitoring. ---------------------------------------------------------------------- darknyan · 2012-11-25 03:23 I don't mind having my bot kicked from irc channels when they don't want to be logged. It's a right of privacy and the right of the channel mod to do so. ---------------------------------------------------------------------- faghat · 2012-11-25 03:34 irc.shakeababy.net (link for Rizon) allows up to 500 channels I think, and you can connect 3 times to each Rizon link, so that's 1,500 channels covered on one IP. ---------------------------------------------------------------------- tobsn · 2012-11-25 03:49 yeah but do we know his network? (I might read over it) ---------------------------------------------------------------------- faghat · 2012-11-25 04:24 No, I assumed he just meant random channels (and thus, networks). In which case, even if he didn't go to Rizon (are there any other large networks that have a high channel limit btw?), I think it would be less hassle than setting up a crapload of connections to the network that he regularly goes on. ---------------------------------------------------------------------- tobsn · 2012-11-25 04:32 I just assumed he already has one big network selected to get a lot of content. quakenet/freenode/efnet/ircnet and I think they're all limited ---------------------------------------------------------------------- faghat · 2012-11-25 04:40 Perhaps. In any case, I was just offering a solution to one of his possible issues. ---------------------------------------------------------------------- TehZomB · 2012-11-25 05:32 building on the ipv6 thing, many ipv6 i:lines are done via the /48 or /64 block... so you'd either need multiple blocks or you'd need to ask for an i:line. ---------------------------------------------------------------------- tobsn · 2012-11-25 05:36 yeah I was actually thinking about it but ignored it in hope I'm wrong. ;) ---------------------------------------------------------------------- TehZomB · 2012-11-25 05:39 haha. be careful, i got an entire subnet k:lined from all open IRCnet servers that way! ---------------------------------------------------------------------- [deleted] · 2012-11-25 11:09 [deleted] ---------------------------------------------------------------------- darknyan · 2012-11-25 13:10 Nope, dozens of networks. ---------------------------------------------------------------------- darknyan · 2012-11-25 13:11 Appreciate it mate. :) ---------------------------------------------------------------------- cheez0r · 2012-11-25 14:41 Might make more sense to base it around an IRCD if you can get the agreement of the network to be logged. ---------------------------------------------------------------------- jonconley · 2012-11-26 15:51 You could ask various networks operators if they have archived and maybe get access. I'm sure they wouldn't give up private channels but the larger more public channels, might not be an issue. ---------------------------------------------------------------------- PriceChild · 2012-11-28 14:40 Lots of projects already make their logs available, such as https://irclogs.ubuntu.com If you're after "something", just to get stats, why not use the logs of projects such as Ubuntu that apparently don't mind you doing so? Joining lots of channels, deeming them 'public' and using the logs for your own means might get quite a lot of people upset. ---------------------------------------------------------------------- [deleted] · 2012-11-29 17:27 [deleted] ---------------------------------------------------------------------- WornOutMeme · 2013-01-05 06:36 If the channel allows logging then you consent to being logged by joining the channel. ---------------------------------------------------------------------- Jan- · 2013-02-04 08:14 IRSeeK is a project that will interest you I'm sure. they're not doing it anymore I think http://www.crunchbase.com/company/irseek you could contact their founders and maybe get a better understanding how it could be done.