Fluentd, InfluxDB and Grafana

Jan 13, 2019 10:44 · 628 words · 3 minute read tutorial devops

What are we going to talk about?

This article will be a brief presentation of the three tools and how to use them to create awesome dashboards and monitor any metrics you like.

Use case

Fluentd, InfluxDB, and Grafana can be used for a lot of purposes, but it's mainly for storing and displaying data points. We will focus this tutorial on the following use-case: you have an application that produces lots of logs with numerical values and you want to monitor them.

For example, if we want to stay in the artificial intelligence area, it could be a reinforcement learning algorithm where you want to monitor the reward, entropy, and other useful metrics.

What are they for?

Fluentd

Fluentd is a tool that aggregates all your logs and feeds them into any other things, like databases or just for archiving. It's in fact like logstash, but with more plugins, so more data sources and outputs possible.

InfluxDB

InfluxDB is a database designed to store real-time metrics and time series. There are lots of similar product for this purpose but seems an easy to use one. I have also looked for prometheus but was little more difficult to set up with Fluentd, needed to specify each key store as a metric in the Fluentd config.

However, if InfluxDB is not the right choice, after all, it will be easy to change for another one as Fluentd supports nearly all databases, at least all compatible directly with Grafana.

Grafana

Grafana is a front-end to visualize your metrics. It is very easy to use, and you can any type of visualization you want with an only little amount of clicks. It is also compatible directly with InfluxDB.

Quickstart using docker-compose

Deploying

We will deploy the three tools via docker compose, so make sure you have it installed (here a link for installation).

I wrote a simple compose file and basic configs for the stack, you can clone it and use it

git clone https://github.com/Coac/fluentd-influxdb-grafana
cd fluentd-influxdb-grafana
docker-compose up

It will pull and build the required images, and run the three tools as containers, you will get the logs outputted in your stdout.

Feeding logs

Per default, I configured Fluentd to look for JSON data in logs/logs.json and feed them into InfluxDB every 5 seconds. If you want another log file, you will need to modify fluentd/fluent.conf.

We will use bash to creates some dummy logs, but in practice, you will use your real application, and your favourite language. For instance, for Golang, you will uses logrus and set the correct log file and json formatter. The following bash command will create 100 json logs with the current date as time and random data.

for i in {1..100}; do echo \{\"time\":$(($(date +"%s")+$i)),\"foo\":\"foo\",\"bar\":$RANDOM\} >> logs/logs.json; done

Visualization

You can browse grafana using http://localhost:3000/

  • Username: admin
  • Password: admin

Select the InfluxDB data source and use the following configuration:

  • URL: http://influxdb:8086
  • Database: test
  • User: admin
  • Password: admin

We use http://influxdb:8086 because per default, docker-compose created a network for the three services and for each of them used an IP alias corresponding to their service name defined in the docker-compose file.

Then click on New dashboard > Graph to create an empty graph. Hover your mouse on top of the figure, on Panel title and click on the down arrow, then Edit.

Use the following configuration to select the data stored in foo key. graph_metrics

You can then click on the top right corner the arrow circle to refresh the graph and see something similar as this graph_screen

Voila, now you can start to build fancy dashboards to monitor fancy stuff.


Hope you enjoyed this article, if you find any bugs or has suggestions, feel free to create a pull request or an issue in the repository.

tweet Share