Jenkins and Sitespeed.io

While surfing the internet I stumbled across Sitespeed.io. It’s a amazing collection of Open Source Tools, which make performance measuring for developers and testers super easy. I tried it out and was immediately impressed. Here’s a little tutorial on how to use Jenkins and Sitespeed.

Requirements

  • Docker (latest)

Environment setup

With minimal 2 commands the environment (via Docker) is already created. Most of the time will be needed for the plugins installation.

On setup wizard finish: unlock Jenkins, install the suggested plugins, create an account and finish the instance configuration.

Jenkins permissions to /var/run/docker.sock

Before you start with Jenkins job configuration, ensure that user jenkins has permissions to /var/run/docker.sock.

Jenkins job configuration

When Jenkins is ready (restarted), install the HTML Publisher PlugIn (no restart after installation of plugin required).

Jenkins HTML Publisher Plugin

Create a new free-style project named SiteSpeed.

Jenkins SiteSpeed Project

Attention: You need to specify later the absolute path to the local directory /target/workspace/SiteSpeed. If you do not know how, press save and start the build without any job information (empty job configuration) and follow the optional instructions.

In my case the path is: “/Users/steffen/Projects/Sitespeed/target/workspace/SiteSpeed”. Under job configuration section “Build” enable “Execute shell” and paste following command.

Via Post-Build-Action: Publish HTML reports you can enter the report very simple from the job project page.

Jenkins SiteSpeed Job Configuration

Save everything and run the job. After a short time you can look at the HTML report. See “Pages” > “https://www.sitespeed.io/” for screenshots, HAR and video files. On the website of sitespeed.io is a very detailed documentation and many more examples. Have fun!

Create QA dashboards with Grafana (Part 5)

In the last tutorial of this series, we create the dashboards for testing and support. Part 1, Part 2, Part 3 and Part 4 should have been successfully completed.

Dashboard Testing

First, I show the result. With a little Drag & Drop and resizing, your dashboard might look like this. But after all, it’s up to your creativity how the results look.

Final Dashboard Testing

I will just show now the most important configurations, from the 4th part you should already know the others.

Add following panels. 1x Graph panel, 2x Singlestat panel and 1x Pie Chart panel. Now edit the Graph panel.

Under tab General enter for Title: Results: $Testers. On tab Metrics select Data Source InfluxDB_test_db and enter From: default suite WHERE: qa =~ /^$Testers$/, SELECT: field(passed) alias(Test Passed) field(skipped) alias(Tests Skipped) field(failed) alias(Tests Failed), FORMAT AS: Time series and ALIAS BY: $col.

Testing Graph Metrics

For tab Axes just enable Show checkbox on Left Y and X-Axis, Unit: short, Scale: linear and Mode: Time. On tab Legend enable checkboxes Show, As Table, Min, Max and Current. On tab Display enable only checkbox Lines.

Now you can edit the Singlestat panels (one after the other). Here the important screens for Passed Total: $Testers:

Testing Singlestat Metrics

Options

Testing Singlestat Options

For Singlestat Failed Total: $Testers, just change field(passed) to field(failed).

The Pie Chart Average: $Testers is simple, too. Here are the most important settings.

Testing Pie Chart Metrics

Options

Testing Pie Chart Options

You’re done with 2nd dashboard. Don’t forget to save (incl. variables and so on)!

Dashboard Support

Let’s get to the last dashboard (Support). Again, the result.

Final Dashboard Support

Now add the following panels. 1x Graph panel and 1x Pie Chart panel (we dynamically generate the others).

Here now the important settings. On tab General have attentions to Repeat!

Support Graph General

Metrics

Support Graph Metrics

Axes

Support Graph Axes

Display

Support Graph Display

The tab Legend for Graph panel have only Show checkbox enabled. Let’s go to the Pie Chart settings. On tab General have attentions to Repeat!

Support Pie Chart General

Metrics

Support Pie Chart Metrics

Options

Support Pie Chart Options

That’s it with this series. I hope you have all the knowledge to create with Grafana awesome QA dashboards.

Create QA dashboards with Grafana (Part 4)

Finally, we can create the first dashboard. The prerequisite for this is that you have successfully completed the previous tutorials (Part1, Part2, Part3).

Pipeline graph panel

Now go to the still empty dashboard Pipeline. Add the Graph Panel here and select Edit in the title of panel.

Pipeline Graph Panel

In the Metrics tab you now enter the following values. Data Source: InfluxDB_pipeline_db, From: default, pipeline, WHERE: stage =~ /^$Stage$/, SELECT: field(duration), FORMAT AS: Time series and ALIAS BY: Duration.

Note: In case your graph is not displayed correctly, select on the Time picker dropdown: Last 30 days.

Pipeline Graph Metrics

Change to General tab and enter for Title: Duration: $Stage.

Pipeline Graph General

Change to Axes tab and enable checkboxes Show for Left Y and X-Axis. The Unit value for Left Y should be: minutes(m).

Pipeline Graph Axes

Under tab Legend choose enable following checkboxes: Show, As Table, Min and Max. For Decimals enter 2.

Pipeline Graph Legend

Our last action for the Graph panel will be done in tab Display. Here we add the Threshold. T1: gt, 15, Color critical, enable checkbox Fill and Y-Axis: left.

Pipeline Graph Display

We are done with Graph panel … don’t forget to save!

Pipeline singlestat panel (Last Status)

Now we add 2 more singlestat panels. One should show the Last Status and the other Last Duration. Press Edit for Last Status panel.

Pipeline Singlestat Panel

In the Metrics tab you now enter the following values. Data Source: InfluxDB_pipeline_db, From: default, pipeline, WHERE: stage =~ /^$Stage$/, SELECT: field(status)last() and FORMAT AS: Time series.

Pipeline Singlestat Metrics

Change to General tab and enter for Title: Last status: $Stage.

Pipeline Singlestat General

Close Panel edit mode and save.

Pipeline singlestat panel (Last Duration)

Last Singlestat will have following Metrics. Data Source: InfluxDB_pipeline_db, From: default, pipeline, WHERE: stage =~ /^$Stage$/, SELECT: field(duration) and FORMAT AS: Time series. Under tab General just add Title: Last Duration: $Stage.

Pipeline Singlestat Options

For tab Options select Stat: Current, Font size: 50%, Unit: minutes(m), Thresholds: 10,15, enable checkbox Show for Gauge, Min: 0, Max: 30 and enable checkboxes Threshold labels plus Threshold markers. Close Edit mode and save.

Final Pipeline Dashboard

Now you can play with the size and placement of the panels. My Pipeline dashboard now looks like this:

Pipeline Final Dashboard

If you change the variables (S1, S2, S3), the values ​​of the panels should change.

This leaves only 2 dashboards left. See you in next tutorial.

Create QA dashboards with Grafana (Part 3)

In the third part of this tutorial series we will create Query Variables and Annotations for our three Dashboards (Testing, Support, Pipeline). Thus, we can later provide interactive dashboards (instead of static ones). Stakeholders should be able to independently filter or select information, which makes the Grafana dashboards more interesting.

Note: This part assumes that you have completed the first and second part successfully.

Variables for dashboard Testing

For Dashboard Testing we need a query variable: QA. Go to the Dashboard Settings on Testing. Now select the “Add variable” button in the “Variables” category.

Grafana Dashboard Variables

Assign a unique name and label: Testers, select as Type: Query, Data source: InfluxDB_test_db and Refresh: On Dashboard Load. As Query insert: SHOW TAG VALUES WITH KEY = “qa”. In the Preview you should see already all values. Last enable checkboxes for Multi-value and Include all value. Press “Add” and “Save” buttons when you’r done.

Grafana Variables Testing

On the empty dashboard should now be a dropdown, with the values ​​visible.

Variables and annotations for dashboard Support

For Dashboard Support we need a query variable: Support. Assign a unique name and label: Support, select as Type: Query, Data source: InfluxDB_support_db, Refresh: On Dashboard Load and Sort: Alphabetical (asc). As Query insert: SHOW TAG VALUES WITH KEY = “support”. In the Preview you should see already all values. Last enable checkboxes for Multi-value and Include all value. Press “Add” and “Save” buttons when you’r done.

Grafana Variables Support

In addition, this dashboard should also have an annotation. Select the “Add Annotation Query” button in the “Annotations” category.

Grafana Dashboard Annotations

Give name: In vector, select Data source: InfluxDB_support_db, enable checkbox: Enabled and add Query: SELECT FROM “in” FROM “tickets” WHERE $timeFilter AND “in” >= 20. Press button’s “Add” and “Save”.

Grafana Annotations Support

On the empty dashboard should now be a dropdown and a checkbox.

Variables and annotations for dashboard Pipelines

Our last dashboard Pipelines will have a variable and annotation, too. Create therefore a variable with following configuration. Name/Label: Stage, Type: Query, Data source: InfluxDB_pipeline_db, Refresh: On Dashboard Load, Query: SHOW TAG VALUES WITH KEY = “stage” and Sort: Alphabetical (asc). Press buttons “Add” and “Save”.

Grafana Variables Pipelines

For the annotation give following values. Name: High Duration, Data source: InfluxDB_pipeline_db, Enabled and Query: SELECT “duration” FROM “pipeline” WHERE $timeFilter AND “duration” >= 15. Press “Add” and “Save”.

Grafana Annotations Pipelines

On the empty dashboard should now be a dropdown and a checkbox.

Okay … that’s it for the third part. In the 4th part we will finally add the pannels in the respective dashboards.

Create QA dashboards with Grafana (Part 2)

In the first part of this tutorial series, you created the environment and recorded some fictional data into InfluxDB. Now log in to Grafana and create data sources for your dashboard pannels.

The next steps will only be performed if you have stopped the environment via docker-compose stop.

You should now find yourself on the Home Dashboard.

Data Sources

The first step is to create the InfluxDB data sources for the metrics to be available in Grafana. To do this, navigate to Data Sources now.

Grafana Data Sources

Now click on “Add data source”. We will do this step 2 more times. For each database (test_db, support_db, pipeline_db) in InfluxDB we will specify a separate connection.

Assign a unique name (like InfluxDB_test_db), select as Type: InfluxDB, enter the URL: http://10.1.0.10:8086, select Access: Server (Default) and Database: test_db. Finally, press the “Save & Test” button. Additional values ​​are not necessary for this tutorial.

Grafana Data Source Configuration

If everything went well, you should get the hint “Data source is working.”

Repeat the process for the other two InfluxDB databases (support_db, pipeline_db). Pay attention to the values ​​of name and database. In the end, your configuration should look like this:

Grafana InfluxDB Data Sources

Dashboards

In this step, you will now create three empty dashboards (under General). Similar to the Data Sources – one for each database. Switch to Manage – Dashboards.

Grafana Dashboards

Select the “+ Dashboard” button. Before adding panels, give the dashboard a unique name. To do this, select the “Settings” icon (top right).

Give the name value, for example “Testing” and for Timezone choose: Local Browser Time. Finally, press the “Save” button.

Grafana Dashboard Settings

Repeat the process 2 more times. In the end, you should have created 3 empty dashboards (eq. Testing, Support and Pipelines).

Grafana Dashboards Overview

Perfect, you should now be able to see your 3 dashboards on the “Home Dashboard”.

Grafana Home Dashboard

In the third part of the series, you will learn to create annotation and variables for each dashboard. I hope you had fun and that you have been able to learn something new till here. See you soon in 3rd part.

Create QA dashboards with Grafana (Part 1)

Since I have my new role (Head of QA), many employees constantly want metrics from me. That means a lot of work for me. But since I do not always want to deal with such things, I have searched for a simpler way. So the question was – how can I deliver this data at any time and possibly from different sources (eq. JIRA, pipelines, test results, Salesforce, etc.)? Hmmm … Grafana is awesome – not only for DevOps! So in this tutorial series, I’d like to show you how to create nice and meaningful dashboards for your QA metrics in Grafana.

What you need?

  • Docker installed (latest version)
  • Bash (min. 3.2.x)

Prepare the project

In order to create dashboards in Grafana, you need a small environment (Grafana/InfluxDB) as well as some data. The next steps will help you to create them. The environment/services are simulated by docker containers. For the fictitious data, just use the bash script which I created for this tutorial.

Now copy/paste the content of the two files with your favorite editor. The content of docker-compose.yml.

And here the content of CreateData.sh.

Start environment and create data

Once the project and the files have been created, you can build and start the environment. For this you use Docker Compose.

In next step you create the InfluxDB databases (incl. fictitious measurements, series and data) via Bash script.

Note: You could use influx command to administrate InfluxDB directly.

Okay the environment preparation is done. Now start Grafana in browser.

New Grafana Login

The default username and password is “admin:admin“. Note, if you use docker-compose down you have to repeat most of steps like data creation. Better use docker-compose stop! … See you in 2nd part – where we add data sources and create dashboards.

File encryption/decryption using GPG

There are just too many people and organizations who are interested in our data. Thus, the secure transmission of data is important. Through encryption/decryption, data can be protected from access by third parties. There are already very long easy ways for the encryption/decryption but I have to find again and again that these are quite unknown. Herewith a little tutorial where I want to show possibilities by means of GPG.

Requirements

  • Docker (latest)

Environment preparation

By means of two Docker containers, we now want to simulate 2 persons who exchange the encrypted data.

Container (user_a)

Container (user_b)

No passphrase prompt

If you want to use the encryption/decryption without prompt, for example in a bash script, you can use the following options. Depending on the version, it can come to a distinction. Option 1 is by default not available in the Docker containers.

Multiple files

You can also use a simple loop to encrypt/decrypt multiple files. Please note the available GPG version/options. Here now a simple example without prompt.

Encryption and Decryption via keys

Container (user_a)

Container (user_b)

Both public keys are available.

Both clients need to import the public key from other.

Our user_a now encrypt data.

User_b now decrypt data.

I hope that you have found an entry point into the topic and I have woken up your interest.

Man in the Middle Attack (MITM)

In this tutorial you will learn how to work a man in the middle attack. For this you will create and configure a simple test environment. The test environment simulates a small home network with a NAT router, a client (victim) and another client (evil) that has already penetrated the network. For the attack itself, you will get in touch with popular mitmf framework.

Attention: The tutorial is presented just for educational purposes. If you do what you have learned outside the test environment, you may be liable to prosecution.

Requirements

  • VirtualBox (5.2.18)
  • Vagrant (2.1.5)

Prepare environment

In the first step, you need to configure, setup and provision the environment. Vagrant will help you here. Via Vagrant you will create all needed virtual machines (incl. SSH keys) and install the needed packages on the evil engine. Via file machines.yml you could add Vagrant boxes for Windows, macOS as well.

Note: Please remove the spaces behind etc (in the Vagrantfile)! These are only because of the security settings of my provider.

Small network changes

You must now switch from typical NAT to NAT network. For that you stop (halt) all VM’s. In the next steps you will create a new NAT network and configure the VM network adapters for this network. In the end, you simulated a simple home network.

Start all VM’s again

In this step we start all VM’s but without Vagrant.

Now check the network interfaces for both VM’s. Please note down the IP’s, you will need them in next steps. You can login in both with credentials vagrant:vagrant.

Note: In the example the evil VM has the IP: 192.168.15.5 and the victim the IP: 192.168.15.6 – this could be different for you.

In order not to use the VirtualBox Terminal, create a port forward from the localhost to the evil VM.

Man-in-the-middle attack

You made it, the test environment is finally ready. If you have been able to learn something new up to this point, I am glad. Now imagine the following situation. You are the victim and you surf the Internet, logging in on your popular websites. Can you imagine what can happen? In a few minutes you will see it.

Once the Ubuntu has booted, run the following command (as evil) and surf the web using the Firefox browser (as victim). If the mitmf returns an error message, repeat the command in the terminal. Be a bit patient on successful call.

Mitmf still offers a lot of plug-ins, just give it a try.