Feb 162014
 

Edit: This post is pretty old and Elasticsearch/Logstash/Kibana have evolved a lot since it was written.

To make sure I understood how to find data using Kibana3, I started collecting input from IRC.

kibana-irc

I have a ZNC bouncer set up on my network. 192.168.1.10

http://wiki.znc.in/ZNC

I have it set to Keep Buffer, Prepend Timestamps.
Timestamp Format:

[%Y-%m-%d %H:%M:%S]

I used the IRC input for Logstash to have something to search with. Since I’m obsessing over this, might as well make a dashboard showing what I really want to see!

channel: "#logstash" OR channel: "#elasticsearch" OR message: "elasticsearch" OR message: "logstash" OR message: "kibana" OR message: "splunk" OR message: "syslog" OR message: "graylog*" OR message: "nxlog"

kibana-irc-search

Below is my logstash configuration.

Advertisement:

logstash-irc.conf

input {
  irc {
    channels => "#chat"
        host => "192.168.1.10"
        nick => "ragingcomputer"
        password => "username:password"
        port => 6667
        secure => true
        user => "ragingcomputer"
  }

  irc {
    channels => [ "##boxee-hacking", "#archlinux-arm", "#arduino", "#avr", "#boxeeplus", "#chumby", "#elasticsearch", "#ffmpeg", "#launchpad", "#linuxcnc", "#linuxmce", "#lockresearch", "#logstash", "#mythtv-users", "#raspbian", "#sickbeard", "#sparkfun", "#ubuntu-mythtv", "#videolan", "#archlinux", "#ubuntu", "#debian", "#perl", "#znc", "##windows", "#pfsense", "#owncloud", "#redis", "#zabbix", "#nagios", "#reddit-sysadmin", "#sensu", "#graylog2", "#plex", "#couchpotato", "#ossec", "#graphite", "#hadoop", "#icinga", "#pauldotcom" ]
        host => "192.168.1.10"
        nick => "ragingcomputer"
        password => "username:password"
        port => 6667
        secure => true
        user => "ragingcomputer"
  }

  irc {
    channels => "#twitlive"
        host => "192.168.1.10"
        nick => "ragingcomputer"
        password => "username:password"
        port => 6667
        secure => true
        user => "ragingcomputer"
  }

}

filter {
  if [message] =~ /ACTION\s\[201[0-9]-[0-9][0-9]-[0-9][0-9]\s[0-9][0-9]:[0-9][0-9]:[0-9][0-9]\]/ {
    drop { }
  }
  if [message] =~ /^\[201[0-9]-[0-9][0-9]-[0-9][0-9]\s[0-9][0-9]:[0-9][0-9]:[0-9][0-9]\]/ {
    drop { }
  }
  if [nick] =~ /^\*/ {
    drop { }
  }
}

output {

  elasticsearch {
    host => "127.0.0.1"
    cluster => "logcatcher"
  }
}

  2 Responses to “Searching IRC Activity with Logstash / Elasticsearch / Kibana”

  1. Hey,

    I have 2 questions:

    1) Are you connecting to ZNC instead of parsing logs ?
    2) Why do you have multiple irc setups using same ZNC server and same user just with a different channel list

    Thanks

  2. I am connecting to ZNC. I have multiple users connecting to multiple server networks.
    The Username for ZNC doesn’t seem to matter. The password is how you identify and authenticate to ZNC.
    The password is more than just the password. It’s username:password

    I was connecting directly to the IRC bouncer instead of watching logs because my IRC bouncer is on a different machine. I’m not sure real-time log searching is as important as I first thought so I’ll probably start rsync the log files to my logstash machine and watching log files so I can spin down that VM without losing anything.

 Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

(required)

(required)