Mar 082014
 

Edit: This post is pretty old and Elasticsearch/Logstash/Kibana have evolved a lot since it was written.

Since I mostly deal with Windows computers, and partially because I couldn’t figure out why phantomjs on my debian netinstall rendered fonts to be UGLY, I figured I’d use one of the many windows computers to render and email the reports. I’m using my gmail account for testing this.

gmail-message

Samples:
overview.pdf
overview.png

That looks pretty awesome, right? The secret sauce here is PhantomJS It pretty much a headless WebKit. A browser without a display. And it’s cross-platform!

I use PhantomJS to screenshot for PNG output and print to PDF. I then use ImageMagick to crop the PNG to a reasonable size. I use sendEmail to, well, send an email with the files attached.

Enough jibber-jabber.

Make a folder at the root of your C drive “captureweb”. Make a folder “phantomjs-script” within that.

C:\captureweb
C:\captureweb\phantomjs-script

You’ll need to download a few programs, but they don’t need to be installed. This can be moved from one computer to another just by copying the entire folder and making a new scheduled task.

http://phantomjs.org/download
The file I used is phantomjs-1.9.7-windows.zip. Extract all. Copy the phantomjs folder to “C:\captureweb” and rename to phantomjs. Verify the phantomjs.exe path matches the one listed.

C:\captureweb\phantomjs
C:\captureweb\phantomjs\phantomjs.exe

http://www.imagemagick.org/script/binary-releases.php
The file I used is ImageMagick-6.8.8-7-Q16-x86-windows.zip. Extract all. Copy the ImageMagick folder to “C:\captureweb” and rename to ImageMagick. Verify the convert.exe path matches the one listed.

C:\captureweb\ImageMagick
C:\captureweb\ImageMagick\convert.exe

http://caspian.dotconf.net/menu/Software/SendEmail/
The file I used is sendEmail-v156.zip. Extract all. Copy the sendEmail folder to “C:\captureweb” and rename to sendEmail. Verify the sendEmail.exe path matches the one listed.

C:\captureweb\sendEmail
C:\captureweb\sendEmail\sendEmail.exe

PhantomJS Configuration
This is my implementation of the solution suggested in Kibana Issue 509

The phantomjs is the rasterize.js example with a few minor modifications. Mainly, it adds a variable waitTime to easily change the time PhantomJS waits before finishing the page render and dumping the output. I also modified the page.viewportSize so the output from kibana looked reasonable.
I also added this line to log in since I have my kibana/elasticsearch secured with nginx.

page.customHeaders={'Authorization': 'Basic '+btoa('user:pass')};

The only difference between kibanapdf.js and kibanapng.js is the viewport size. I found that 900px is a great viewport width for printing to PDF and 720p is the resolution I want for the PNG.

C:\captureweb\phantomjs-script\kibanapdf.js

var page = require('webpage').create(),
    system = require('system'),
    address, output, size;

//waitTime is how long to to wait for kibana and the data to load in millis.
//Increase this if your queries appear incomplete in your PDFs. 10s seems to work for me.
var waitTime = 10 * 1000;

if (system.args.length < 3 || system.args.length > 5) {
    console.log('Usage: rasterize.js URL filename [paperwidth*paperheight|paperformat] [zoom]');
    console.log('  paper (pdf output) examples: "5in*7.5in", "10cm*20cm", "A4", "Letter"');
    console.log('  image (png/jpg output) examples: "1920px" entire page, window width 1920px');
    console.log('                                   "800px*600px" window, clipped to 800x600');
    phantom.exit(1);
} else {
    address = system.args[1];
    output = system.args[2];
    // Uncomment the following line to login. Replace user:pass with your username and password.
    //page.customHeaders={'Authorization': 'Basic '+btoa('user:pass')};
    page.viewportSize = { width: 900, height: 500 };
    if (system.args.length > 3 && system.args[2].substr(-4) === ".pdf") {
        size = system.args[3].split('*');
        page.paperSize = size.length === 2 ? { width: size[0], height: size[1], margin: '0px' }
                                           : { format: system.args[3], orientation: 'portrait', margin: '1cm' };
    } else if (system.args.length > 3 && system.args[3].substr(-2) === "px") {
        size = system.args[3].split('*');
        if (size.length === 2) {
            pageWidth = parseInt(size[0], 10);
            pageHeight = parseInt(size[1], 10);
            page.viewportSize = { width: pageWidth, height: pageHeight };
            page.clipRect = { top: 0, left: 0, width: pageWidth, height: pageHeight };
        } else {
            console.log("size:", system.args[3]);
            pageWidth = parseInt(system.args[3], 10);
            pageHeight = parseInt(pageWidth * 3/4, 10); // it's as good an assumption as any
            console.log ("pageHeight:",pageHeight);
            page.viewportSize = { width: pageWidth, height: pageHeight };
        }
    }
    if (system.args.length > 4) {
        page.zoomFactor = system.args[4];
    }
    page.open(address, function (status) {
        if (status !== 'success') {
            console.log('Unable to load the address!');
            phantom.exit();
        } else {
            window.setTimeout(function () {
                page.render(output);
                phantom.exit();
            }, waitTime);
        }
    });
}

C:\captureweb\phantomjs-script\kibanapng.js

Advertisement:
var page = require('webpage').create(),
    system = require('system'),
    address, output, size;

//waitTime is how long to to wait for kibana and the data to load in millis.
//Increase this if your queries appear incomplete in your PDFs. 10s seems to work for me.
var waitTime = 10 * 1000;

if (system.args.length < 3 || system.args.length > 5) {
    console.log('Usage: rasterize.js URL filename [paperwidth*paperheight|paperformat] [zoom]');
    console.log('  paper (pdf output) examples: "5in*7.5in", "10cm*20cm", "A4", "Letter"');
    console.log('  image (png/jpg output) examples: "1920px" entire page, window width 1920px');
    console.log('                                   "800px*600px" window, clipped to 800x600');
    phantom.exit(1);
} else {
    address = system.args[1];
    output = system.args[2];
    // Uncomment the following line to login. Replace user:pass with your username and password.
    //page.customHeaders={'Authorization': 'Basic '+btoa('user:pass')};
    page.viewportSize = { width: 1280, height: 720 };
    if (system.args.length > 3 && system.args[2].substr(-4) === ".pdf") {
        size = system.args[3].split('*');
        page.paperSize = size.length === 2 ? { width: size[0], height: size[1], margin: '0px' }
                                           : { format: system.args[3], orientation: 'portrait', margin: '1cm' };
    } else if (system.args.length > 3 && system.args[3].substr(-2) === "px") {
        size = system.args[3].split('*');
        if (size.length === 2) {
            pageWidth = parseInt(size[0], 10);
            pageHeight = parseInt(size[1], 10);
            page.viewportSize = { width: pageWidth, height: pageHeight };
            page.clipRect = { top: 0, left: 0, width: pageWidth, height: pageHeight };
        } else {
            console.log("size:", system.args[3]);
            pageWidth = parseInt(system.args[3], 10);
            pageHeight = parseInt(pageWidth * 3/4, 10); // it's as good an assumption as any
            console.log ("pageHeight:",pageHeight);
            page.viewportSize = { width: pageWidth, height: pageHeight };
        }
    }
    if (system.args.length > 4) {
        page.zoomFactor = system.args[4];
    }
    page.open(address, function (status) {
        if (status !== 'success') {
            console.log('Unable to load the address!');
            phantom.exit();
        } else {
            window.setTimeout(function () {
                page.render(output);
                phantom.exit();
            }, waitTime);
        }
    });
}

And now to tie it all together
C:\captureweb\captureweb.bat makes this all work.

Assuming my elasticsearch/kibana is on 192.168.1.100 and the URL for the dashboard I want is https://192.168.1.100/#/dashboard/elasticsearch/IRC%20Overview

You’ll need to pay special attention to the %20 for the space in that URL. Windows interprets %2 as the second argument from running the batch file. This requires escaping the % by doubling it up as %%
More info on escaping characters http://www.robvanderwoude.com/escapechars.php

Since I’m using a self-signed certificate for nginx, I have to use –ignore-ssl-errors=yes with PhantomJS to connect.

I’m setting the output of date /t to datevar and time /t to timevar. I then use these to put the date and time into the subject and message.

You’ll need to replace this with your source address.

-f USERNAME@gmail.com

You’ll need to put in you username and password. With gmail, your username is your email address.

-xu USERNAME@gmail.com
-xp PASSWORD

You’ll need to replace this with who you’re sending the email to.

-t USERNAME@gmail.com

C:\captureweb\captureweb.bat

@echo off
echo Making PDF of IRC Overview
"C:\captureweb\phantomjs\phantomjs.exe" --ignore-ssl-errors=yes "C:\captureweb\phantomjs-script\kibanapdf.js" https://192.168.1.100/#/dashboard/elasticsearch/IRC%%20Overview "C:\captureweb\overview.pdf" letter

echo Making PNG of IRC Overview
"C:\captureweb\phantomjs\phantomjs.exe" --ignore-ssl-errors=yes "C:\captureweb\phantomjs-script\kibanapng.js" https://192.168.1.100/#/dashboard/elasticsearch/IRC%%20Overview "C:\captureweb\tmp.png" 1280x720
echo Cropping PNG of IRC Overview to 720p
"C:\captureweb\ImageMagick\convert.exe" "C:\captureweb\tmp.png" -crop 1280x720+0+0 "C:\captureweb\overview.png"
echo Removing temp PNG
del "C:\captureweb\tmp.png"

for /f "delims=" %%A in ('date /t') do set "datevar=%%A"
for /f "delims=" %%A in ('time /t') do set "timevar=%%A"
set "subjectvar=Kibana Overivew %datevar%"
set "messagevar=Report Prepared: %datevar% - %timevar%"

echo Sending Email: %subjectvar%
"C:\captureweb\sendEmail\sendEmail.exe" -f USERNAME@gmail.com -xu USERNAME@gmail.com -xp PASSWORD -o tls=yes -t USERNAME@gmail.com -u "%subjectvar%" -s smtp.gmail.com:587 -m "%messagevar%" -a "C:\captureweb\overview.png" "C:\captureweb\overview.pdf"

echo Completed.

Use Task Scheduler in the Control Panel under Administrative Tools to set up a scheduled task to run as often as you need.

PRO TIP
Use the Window Resizer plugin for Chrome to set your browser size when designing your dashboards. If it looks good in chrome with the viewport you want, odds are good it’ll look good in the report you receive.

  4 Responses to “Kibana3 Automated Email Reports Using Windows”

  1. Hi..Thanks for this helpful post. I would like to know if there is a way to call the setrelativeFilter() of timpicker panel or a way to set the time range we need in the Kibana Dashboard before we render it to pdf. Thanks for the help. 🙂

  2. I noticed in the resulting PDF that the colors in the Legend turn black and you can’t tell (aside from ordering) which column belongs to which label. Have you found any way of fixing this?

    For example, in your PDF there is no way to know in the CHANNEL column chart that 33windows is the red column without counting

  3. How can I send an email with the screen shot in the body of the email and not just as an attachment?

  4. Hi .. Even I am looking for the ways of sending in the body of the email and not just as an attachment.
    Let me know if you have figured it out.

 Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

(required)

(required)