HTTPS and Wireshark

Using the two browsers (Firefox and Chrome), I’ll show you how to analyze the TLS traffic with Wireshark. If you only want to use one of the browsers, you can, of course.

What you need?

Wireshark (latest version)
Google Chrome (latest version)
Firefox (latest version)

Let’s start

After export do not change or restart you terminal. Or set an environmental variable (global/user specific) for example in .bashrc/.bash_profile/etc. file.

# create empty file
$ touch ~/Desktop/keys.log

# create environment variable
$ export SSLKEYLOGFILE=$HOME/Desktop/keys.log

# start Firefox
$ /Applications/Firefox.app/Contents/MacOS/firefox-bin --ssl-key-log-file=$HOME/Desktop/keys.log

# start Chrome
$ /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --ssl-key-log-file=$HOME/Desktop/keys.log

In another terminal, you can watch the file.

# tail file (optional)
$ tail -f ~/Desktop/keys.log
...
CLIENT_RANDOM
33da89e4b6d87d25956fd8e8c1e6965575e379ca263b145c8c1240c7f76b0d2a
348d23440ef23807a88c9bda8c8e5826316b15bba33bbfe776120fb9d711c1b04dcf8
1e99e4a58e9d0c57ac955f12a7
...

Wireshark and open Preferences -> Protocols -> SSL. Browse here for file “$HOME/Desktop/keys.log” and confirm your settings.

Start your record (may with filters) and open URL in browser. For example, you can now view the data in Wireshark via the “Decrypted SSL data” tab.

Create QA dashboards with Grafana (Part 5)

In the last tutorial of this series, we create the dashboards for testing and support. Part 1, Part 2, Part 3 and Part 4 should have been successfully completed.

Dashboard Testing

First, I show the result. With a little Drag & Drop and resizing, your dashboard might look like this. But after all, it’s up to your creativity how the results look.

I will just show now the most important configurations, from the 4th part you should already know the others.

Add following panels. 1x Graph panel, 2x Singlestat panel and 1x Pie Chart panel. Now edit the Graph panel.

Under tab General enter for Title: Results: $Testers. On tab Metrics select Data Source InfluxDB_test_db and enter From: default suite WHERE: qa =~ /^$Testers$/, SELECT: field(passed) alias(Test Passed) field(skipped) alias(Tests Skipped) field(failed) alias(Tests Failed), FORMAT AS: Time series and ALIAS BY: $col.

For tab Axes just enable Show checkbox on Left Y and X-Axis, Unit: short, Scale: linear and Mode: Time. On tab Legend enable checkboxes Show, As Table, Min, Max and Current. On tab Display enable only checkbox Lines.

Now you can edit the Singlestat panels (one after the other). Here the important screens for Passed Total: $Testers:

Tab Options (Singlestat)

For Singlestat Failed Total: $Testers, just change field(passed) to field(failed).

The Pie Chart Average: $Testers is simple, too. Here are the most important settings.

Tab Options (Pie Chart)

You’re done with 2nd dashboard. Don’t forget to save (incl. variables and so on)!

Dashboard Support

Let’s get to the last dashboard (Support). Again, the result.

Now add the following panels. 1x Graph panel and 1x Pie Chart panel (we dynamically generate the others).

Here now the important settings. On tab General have attentions to Repeat!

Tab Metrics (Graph)

Tab Axes (Graph)

Tab Display (Graph)

The tab Legend for Graph panel have only Show checkbox enabled. Let’s go to the Pie Chart settings. On tab General have attentions to Repeat!

Tab Metrics (Pie Chart)

Tab Options (Pie Chart)

That’s it with this series. I hope you have all the knowledge to create with Grafana awesome QA dashboards.

Create QA dashboards with Grafana (Part 4)

Finally, we can create the first dashboard. The prerequisite for this is that you have successfully completed the previous tutorials (Part1, Part2, Part3).

Pipeline graph panel

Now go to the still empty dashboard Pipeline. Add the Graph Panel here and select Edit in the title of panel.

In the Metrics tab you now enter the following values. Data Source: InfluxDB_pipeline_db, From: default, pipeline, WHERE: stage =~ /^$Stage$/, SELECT: field(duration), FORMAT AS: Time series and ALIAS BY: Duration.

Note: In case your graph is not displayed correctly, select on the Time picker dropdown: Last 30 days.

Change to General tab and enter for Title: Duration: $Stage.

Change to Axes tab and enable checkboxes Show for Left Y and X-Axis. The Unit value for Left Y should be: minutes(m).

Under tab Legend choose enable following checkboxes: Show, As Table, Min and Max. For Decimals enter value “2”.

Our last action for the Graph panel will be done in tab Display. Here we add the Threshold. T1: gt, 15, Color critical, enable checkbox Fill and Y-Axis: left.

We are done with Graph panel … don’t forget to save!

Pipeline singlestat panel (Last Status)

Now we add 2 more singlestat panels. One should show the Last Status and the other Last Duration. Press Edit for Last Status panel.

In the Metrics tab you now enter the following values. Data Source: InfluxDB_pipeline_db, From: default, pipeline, WHERE: stage =~ /^$Stage$/, SELECT: field(status)last() and FORMAT AS: Time series.

Change to General tab and enter for Title: Last status: $Stage.

Close Panel edit mode and save.

Pipeline singlestat panel (Last Duration)

Last Singlestat will have following Metrics. Data Source: InfluxDB_pipeline_db, From: default, pipeline, WHERE: stage =~ /^$Stage$/, SELECT: field(duration) and FORMAT AS: Time series. Under tab General just add Title: Last Duration: $Stage.

For tab Options select Stat: Current, Font size: 50%, Unit: minutes(m), Thresholds: 10,15, enable checkbox Show for Gauge, Min: 0, Max: 30 and enable checkboxes Threshold labels plus Threshold markers. Close Edit mode and save.

Final Pipeline Dashboard

Now you can play with the size and placement of the panels. My Pipeline dashboard now looks like this:

If you change the variables (S1, S2, S3), the values of the panels should change.

This leaves only 2 dashboards left. See you in next tutorial.

Create QA dashboards with Grafana (Part 3)

In the third part of this tutorial series we will create Query Variables and Annotations for our three Dashboards (Testing, Support, Pipeline). Thus, we can later provide interactive dashboards (instead of static ones). Stakeholders should be able to independently filter or select information, which makes the Grafana dashboards more interesting.

Note: This part assumes that you have completed the first and second part successfully.

Variables for dashboard Testing

For Dashboard Testing we need a query variable: QA. Go to the Dashboard Settings on Testing. Now select the “Add variable” button in the “Variables” category.

Assign a unique name and label: Testers, select as Type: Query, Data source: InfluxDB_test_db and Refresh: On Dashboard Load. As Query insert: SHOW TAG VALUES WITH KEY = “qa”. In the Preview you should see already all values. Last enable checkboxes for Multi-value and Include all value. Press “Add” and “Save” buttons when you’r done.

On the empty dashboard should now be a dropdown, with the values visible.

Variables and annotations for dashboard Support

For Dashboard Support we need a query variable: Support. Assign a unique name and label: Support, select as Type: Query, Data source: InfluxDB_support_db, Refresh: On Dashboard Load and Sort: Alphabetical (asc). As Query insert: SHOW TAG VALUES WITH KEY = “support”. In the Preview you should see already all values. Last enable checkboxes for Multi-value and Include all value. Press “Add” and “Save” buttons when you’r done.

In addition, this dashboard should also have an annotation. Select the “Add Annotation Query” button in the “Annotations” category.

Give name: In vector, select Data source: InfluxDB_support_db, enable checkbox: Enabled and add Query: SELECT FROM “in” FROM “tickets” WHERE $timeFilter AND “in” >= 20. Press button’s “Add” and “Save”.

On the empty dashboard should now be a dropdown and a checkbox.

Variables and annotations for dashboard Pipelines

Our last dashboard Pipelines will have a variable and annotation, too. Create therefore a variable with following configuration. Name/Label: Stage, Type: Query, Data source: InfluxDB_pipeline_db, Refresh: On Dashboard Load, Query: SHOW TAG VALUES WITH KEY = “stage” and Sort: Alphabetical (asc). Press buttons “Add” and “Save”.

For the annotation give following values. Name: High Duration, Data source: InfluxDB_pipeline_db, Enabled and Query: SELECT “duration” FROM “pipeline” WHERE $timeFilter AND “duration” >= 15. Press “Add” and “Save”.

On the empty dashboard should now be a dropdown and a checkbox.

Okay … that’s it for the third part. In the 4th part we will finally add the pannels in the respective dashboards.

Create QA dashboards with Grafana (Part 2)

In the first part of this tutorial series, you created the environment and recorded some fictional data into InfluxDB. Now log in to Grafana and create data sources for your dashboard pannels.

The next steps will only be performed if you have stopped the environment via docker-compose stop.

# go to Project directory
$ cd ~/Projects/GrafanaDemo/

# start environment
$ docker-compose start

# open Grafana in browser (and login)
$ open http://localhost:3000

You should now find yourself on the Home Dashboard.

Data Sources

The first step is to create the InfluxDB data sources for the metrics to be available in Grafana. To do this, navigate to Data Sources now.

Now click on “Add data source”. We will do this step 2 more times. For each database (test_db, support_db, pipeline_db) in InfluxDB we will specify a separate connection.

Assign a unique name (like InfluxDB_test_db), select as Type: InfluxDB, enter the URL: http://10.1.0.10:8086, select Access: Server (Default) and Database: test_db. Finally, press the “Save & Test” button. Additional values are not necessary for this tutorial.

If everything went well, you should get the hint “Data source is working.”

Repeat the process for the other two InfluxDB databases (support_db, pipeline_db). Pay attention to the values of name and database. In the end, your configuration should look like this:

Dashboards

In this step, you will now create three empty dashboards (under General). Similar to the Data Sources – one for each database. Switch to Manage – Dashboards.

Select the “+ Dashboard” button. Before adding panels, give the dashboard a unique name. To do this, select the “Settings” icon (top right).

Give the name value, for example “Testing” and for Timezone choose: Local Browser Time. Finally, press the “Save” button.

Repeat the process 2 more times. In the end, you should have created 3 empty dashboards (eq. Testing, Support and Pipelines).

Perfect, you should now be able to see your 3 dashboards on the “Home Dashboard”.

In the third part of the series, you will learn to create annotation and variables for each dashboard. I hope you had fun and that you have been able to learn something new till here. See you soon in 3rd part.

Create QA dashboards with Grafana (Part 1)

Since I have my new role (Head of QA), many employees constantly want metrics from me. That means a lot of work for me. But since I do not always want to deal with such things, I have searched for a simpler way. So the question was – how can I deliver this data at any time and possibly from different sources (eq. JIRA, pipelines, test results, Salesforce, etc.)? Hmmm … Grafana is awesome – not only for DevOps! So in this tutorial series, I’d like to show you how to create nice and meaningful dashboards for your QA metrics in Grafana.

What you need?

Docker installed (latest version)
Bash (min. 3.2.x)

Prepare the project

In order to create dashboards in Grafana, you need a small environment (Grafana/InfluxDB) as well as some data. The next steps will help you to create them. The environment/services are simulated by docker containers. For the fictitious data, just use the bash script which I created for this tutorial.

# create project
$ mkdir -p ~/Projects/GrafanaDemo && cd ~/Projects/GrafanaDemo

# create file docker-compose.yml
$ touch docker-compose.yml

# create file CreateData.sh
$ touch CreateData.sh

# change file permissions
$ chmod u+x CreateData.sh

Now copy/paste the content of the two files with your favorite editor. The content of docker-compose.yml.

version: '3'

networks:
  grafana-demo:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 10.1.0.0/24

services:

  influxdb:
    container_name: influxdb
    networks:
      grafana-demo:
        ipv4_address: 10.1.0.10
    ports:
      - '8086:8086'
    image: influxdb

  grafana:
    container_name: grafana
    networks:
      grafana-demo:
        ipv4_address: 10.1.0.20
    ports:
      - '3000:3000'
    image: grafana/grafana
    environment:
      - 'GF_INSTALL_PLUGINS=grafana-piechart-panel'
    depends_on:
      - influxdb

And here the content of CreateData.sh.

#!/usr/bin/env  bash

# shell options
#set -x
#set -v
set -e
set -u
set -f

# magic variables
declare -r OPTS="htsp"
declare -a TIMESTAMPS
declare -a OPTIONS=(false false false)
declare -r -a DATABASES=(test_db support_db pipeline_db)
declare -r -a TESTER=(Tina Robert)
declare -r -a SUPPORTER=(Jennifer Mary Tom)
declare -r -a STAGES=(S1 S2 S3)
declare -r -a BUILD=(SUCCESS FAILURE ABORTED)
declare -r -i SUCCESS=0
declare -r -i BAD_ARGS=85
declare -r -i NO_ARGS=86

# functions
function usage() {
  local count
  local file_name=$(basename "$0")

  printf "Usage: %s [options...]\n" "$file_name"
  for (( count=1; count<${#OPTS}; count++ )); do
    printf "%s\tcreate %s and content\n" "-${OPTS:$count:1}" "${DATABASES[$count - 1]}"
  done
  exit "$SUCCESS"
}

function bad_args() {
  printf "Error: Wrong arguments supplied\n"
  usage
  exit "$BAD_ARGS"
}

function no_args() {
  printf "Error: No options were passed\n"
  usage
  exit "$NO_ARGS"
}

function create_timestamp_array() {
  local counter=1
  local timestamp

  while [ "$counter" -le 30 ]; do
    timestamp=$(date -v -"$counter"d +"%s")
    TIMESTAMPS+=("$timestamp")
    ((counter++))
  done
}

function curl_post() {
  local url=$(printf 'http://localhost:8086/write?db=%s&precision=s' "$2")

  curl -i -X POST "$url" --data-binary "$1"
}

function create_test_results() {
  local passed
  local failed
  local skipped
  local count
  local str

  for count in "${TESTER[@]}"; do
    passed=$((RANDOM % 30 + 20))
    failed=$((RANDOM % passed))
    skipped=$((passed - failed))
    str=$(printf 'suite,app=demo,qa=%s passed=%i,failed=%i,skipped=%i %i' "$count" "$passed" "$failed" "$skipped" "$1")
    echo "$2: $str"
    curl_post "$str" "$2"
  done
}

function create_support_results() {
  local items=(1 2 none)
  local in
  local out
  local str
  local count

  for count in "${SUPPORTER[@]}"; do
    in=$((RANDOM % 25))
    out=$((RANDOM % 25))
    str=$(printf 'tickets,support=%s in=%i,out=%i %i' "$count" "$in" "$out" "$1")
    echo "$2: $str"
    curl_post "$str" "$2"
  done
}

function create_pipeline_results() {
  local status
  local duration
  local str
  local count

  for count in "${STAGES[@]}"; do
    status=${BUILD[$RANDOM % ${#BUILD[@]} ]}
    duration=$(( 3+RANDOM%(3-17) )).$(( RANDOM%999 ))
    str=$(printf 'pipeline,stage=%s status="%s",duration=%s %i' "$count" "$status" "$duration" "$1")
    echo "$2: $str"
    curl_post "$str" "$2"
  done
}

function main() {
  local repeat=$(printf '=%.0s' {1..80})

  create_timestamp_array

  for ((i = 0; i < ${#OPTIONS[@]}; ++i)); do
    if [[ "${OPTIONS[$i]}" == "true" ]]; then
      printf "Create database: %s\n" "${DATABASES[$i]}"
      printf "%s\n" "$repeat"
      curl -i -X POST http://localhost:8086/query --data-urlencode "q=CREATE DATABASE ${DATABASES[$i]}"
      printf "Generate content of database: %s\n" "${DATABASES[$i]}"
      printf "%s\n" "$repeat"
      for item in "${TIMESTAMPS[@]}"; do
        if [[ "${DATABASES[$i]}" == "${DATABASES[0]}" ]]; then
          create_test_results "$item" "${DATABASES[$i]}"
        fi
        if [[ "${DATABASES[$i]}" == "${DATABASES[1]}" ]]; then
          create_support_results "$item" "${DATABASES[$i]}"
        fi
        if [[ "${DATABASES[$i]}" == "${DATABASES[2]}" ]]; then
          create_pipeline_results "$item" "${DATABASES[$i]}"
        fi
      done
    fi
  done
}

# script arguments
while getopts "$OPTS" OPTION; do
  case "$OPTION" in
    h)
        usage;;
    t)
        OPTIONS[0]="true";;
    s)
        OPTIONS[1]="true";;
    p)
        OPTIONS[2]="true";;
    *)
        bad_args;;
  esac
done

if [ $OPTIND -eq 1 ]; then
  no_args
fi

# main function
main

# exit
exit "$SUCCESS"

Start environment and create data

Once the project and the files have been created, you can build and start the environment. For this you use Docker Compose.

# validate docker-compose file (optional)
$ docker-compose config

# run environment
$ docker-compose up -d

# get IP from grafana container (optional)
$ docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' grafana
10.1.0.20

# get IP from influxdb container (optional)
$ docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' influxdb
10.1.0.10

# show created docker network (optional)
$ docker network ls | grep -i 'grafana*'

# show docker containers (optional)
$ docker ps -a | grep -i 'grafana\|influxdb'

In next step you create the InfluxDB databases (incl. fictitious measurements, series and data) via Bash script.

# ping influxdb (optional)
$ curl -I http://localhost:8086/ping

# show help (optional)
$ ./CreateData.sh -h

# create databases and contents (execute only 1x)
$ ./CreateData.sh -t -s -p

# show current databases (optional)
$ curl -G http://localhost:8086/query?pretty=true --data-urlencode "q=SHOW DATABASES"

# show test_db series (optional)
$ curl -G http://localhost:8086/query?pretty=true --data-urlencode "db=test_db" --data-urlencode "q=SHOW SERIES"

# show test_db measurements (optional)
$ curl -G http://localhost:8086/query?pretty=true --data-urlencode "db=test_db" --data-urlencode "q=SHOW MEASUREMENTS"

Note: You could use influx command to administrate InfluxDB directly.

# exec docker container
$ docker exec -ti influxdb influx

# list all databases
> show databases

# use specific database test_db
> use test_db

# show series of test_db
> show series

# show measurements of test_db
> show measurements

# drop measurements of test_db (in case something went wrong)
> drop measurement suite

Okay the environment preparation is done. Now start Grafana in browser.

# open Grafana in browser
$ open http://localhost:3000

The default username and password is “admin:admin“. Note, if you use docker-compose down you have to repeat most of steps like data creation. Better use docker-compose stop! … See you in 2nd part – where we add data sources and create dashboards.

Docker, Telegraf, InfluxDB and Grafana

I have already presented various tutorials on docker monitoring. This time we will use Telegraf.

Project preparation

# create new project
$ mkdir -p Projects/DTIG/influxdb && cd Projects/DTIG/

# create telegraf.conf
$ touch telegraf.conf

InfluxDB preparation

# start InfluxDB
$ docker run --name influxdb -p 8086:8086 -v $PWD/influxdb:/var/lib/influxdb influxdb

# create new user
$ curl -G http://localhost:8086/query --data-urlencode "q=CREATE USER telegraf WITH PASSWORD 'password123' WITH ALL PRIVILEGES"

# create database for telegraf
$ curl -G http://localhost:8086/query -u telegraf:password123 --data-urlencode "q=CREATE DATABASE telegraf_db"

# show ip of influxdb container
$ docker inspect --format '{{ .NetworkSettings.IPAddress }}' influxdb
...
172.17.0.2
...

Telegraf preparation

# Telegraf Configuration

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = ""
  debug = false
  quiet = false
  logfile = ""
  hostname = ""
  omit_hostname = false

[[outputs.influxdb]]
  urls = ["http://172.17.0.2:8086"] # required
  database = "telegraf_db" # required
  retention_policy = ""
  write_consistency = "any"
  timeout = "5s"

[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false

[[inputs.disk]]
  ignore_fs = ["tmpfs", "devtmpfs", "devfs"]

[[inputs.docker]]
  endpoint = "unix:///var/run/docker.sock"
  container_names = []
  timeout = "5s"
  perdevice = true
  total = false
  # docker_label_include = []
  # docker_label_exclude = []

… Read more about Telegraf on documentation page …

# edit telegraf.conf
$ vim telegraf.conf

# start Telegraf
$ docker run --name telegraf -v $PWD/telegraf.conf:/etc /telegraf/telegraf.conf -v /var/run/docker.sock:/var/run/docker.sock telegraf

Note: The space after /etc is just because of security settings of my provider.

… after short time …

# show measurements (be patient)
$ curl -G http://localhost:8086/query -u telegraf:password123 --data-urlencode "db=telegraf_db" --data-urlencode "q=SHOW MEASUREMENTS"

Grafana preparation

# run grafana container
$ docker run --name grafana -i -p 3000:3000 grafana/grafana

# open Grafana WebUI and login (admin:admin)
$ open -a Safari http://localhost:3000

After login (admin:admin) you can add new Data Source.

Okay … all done … you can start to create Dashboards or search for existing Dashboards for import.

Docker, cAdvisor, InfluxDB and Grafana

In previous tutorials I showed the basics for docker monitoring with Prometheus and Jenkins. Here are now the basics with cAdvisor. Many steps are similar and can be implemented just as quickly.

Preparation

# create project
$ mkdir -p Projects/DCIG/influxdb && cd Projects/DCIG/

InfluxDB preparation

This time we start with InfluxDB, because cAdvisor needs it for connection!

# start InfluxDB
$ docker run --name influxdb -p 8086:8086 -v $PWD/influxdb:/var/lib/influxdb influxdb

# create new user
$ curl -G http://localhost:8086/query --data-urlencode "q=CREATE USER cadvisor WITH PASSWORD 'password123' WITH ALL PRIVILEGES"

# create database for cadvisor
$ curl -G http://localhost:8086/query -u cadvisor:password123 --data-urlencode "q=CREATE DATABASE cadvisor_db"

# show ip of influxdb container
$ docker inspect --format '{{ .NetworkSettings.IPAddress }}' influxdb
...
172.17.0.2
...

cAdvisor preparation

Now we will use User, Password and Database name. You can find the documentation here.

# start cAdvisor
$ docker run -d --name=cadvisor -v /:/rootfs:ro -v /var/run:/var/run:rw -v /sys:/sys:ro -v /var/lib/docker/:/var/lib/docker:ro -p 8080:8080 google/cadvisor:latest -storage_driver=influxdb -storage_driver_db=cadvisor_db -storage_driver_user=cadvisor -storage_driver_password=password123 -storage_driver_host=172.17.0.2:8086

# check cAdvisor logs (optional)
$ docker logs -f cadvisor

# open cAdvisor WebUI
$ open -a Safari http://localhost:8080

After a while we can also see if cAdvisor sends metrics to InfluxDB.

# show measurements (be patient)
$ curl -G http://localhost:8086/query -u cadvisor:password123 --data-urlencode "db=cadvisor_db" --data-urlencode "q=SHOW MEASUREMENTS"

prepare and run Grafana

# run grafana container
$ docker run --name grafana -i -p 3000:3000 grafana/grafana

# open Grafana WebUI and login (admin:admin)
$ open -a Safari http://localhost:3000

After login (admin:admin) configure new DataSource for InfluxDB.

When DataSource is configured we import the Grafana Dashboard. (ID: 1367)

… be patient (don’t press any button) …

Select already created DataSource (InfluxDB) and press “Import” button. Now you should see the Grafana Dashboard.

– Softwaretester –

knowledge is power

Monitoring