GitLab and Sitespeed.io

Within this tutorial I will explain shortly the combination of GitLab and Sitespeed.io. Normally GitLab provides this feature (Browser Performance Testing) but only with Premium or Silver editions. I imagine for this example that you know already the basics about GitLab pipelines, Docker in Docker and Sitespeed.io.

Preparation

As a first step you create a simple project directory with some files and one folder.

# create project directory
$ mkdir -p ~/Projects/SitespeedGitLab/ci

# change directory
cd ~/Projects/SitespeedGitLab

# create login file
$ vim ci/login.js

# create urls file
$ vim ci/urls.txt

# create pipeline file
$ vim .gitlab-ci.yml

# show structure (optional)
$ tree .
.
|__.gitlab-ci.yml
|__ci
  |__login.js
  |__urls.txt

Content of files

For the login we use the simplest version of a script which serves as pre script. You can expand it later as needed. If you do not need a login, you do not have to create this file (leave this out later in the command --preScript login.js).

module.exports = async function(context, commands) {

  // navigate to login page
  await commands.navigate('your domain');

  // add values into input fields
  await commands.addText.byId('username', 'input by id');
  await commands.addText.byId('password', 'input by id');

  // find the submit button and click it
  await commands.click.byClassName('Button');

  // wait to verify that you are logged in (incl. max time)
  return commands.wait.byId('specific tag with id',8000);
};

Let’s get to the next file. Here you simply enter all URLs to be checked. Each line represents a URL to be checked which (in our case) always precedes the login. There is also the possibility to login once for all sub-pages as well as various actions on the pages. In this case you would have to script everything (similar to pre/post scripts).

your url one
your url two
your url three

The files urls.txt and login.js have to be mounted inside the container. Therefore we choose the folder “ci”. After the successfully execution this folder will also provide the sitespeed reports.

Pipeline

The last step is the definition of GitLab pipeline. Here now a very simple example for .gitlab-ci.yml, with only one pipeline stage (test) and job (perf_tests). You can also expand this file later as you like (It’s already possible to build Docker images inside).

stages:
  - test

variables:
  DOCKER_HOST: tcp://localhost:2375/
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ""

default:
  image: docker:stable
  services:
    - name: docker:dind

perf_tests:
  stage: test
  variables:
    MOUNT_PATH: "$CI_BUILDS_DIR/ci/"
    DOCKER_IMAGE: 'sitespeedio/sitespeed.io'
    BROWSER: 'firefox'
    ITERATION: '1'
  artifacts:
    name: "performance-reports"
    paths:
      - "$CI_BUILDS_DIR/ci/sitespeed-result/your domain"
  script:
    - docker run --shm-size=1g --rm -v $MOUNT_PATH:/sitespeed.io $DOCKER_IMAGE -b $BROWSER -n $ITERATION --preScript login.js urls.txt

Okay … done. You can commit/push everything into GitLab. After successfully pipeline run you can access the reports as GitLab pipeline artifacts (but not via repository folder -> because if the job is done the container and reports are gone).

ZAP API Basics

In this tutorial, I’d like to share a few ZAP API basics. This should make it possible for anyone to integrate ZAP into various pipelines.

Requirements

  • ZAP installed
  • jq installed

Minimum configuration of ZAP

Start ZAP now, if you get asked for select the persistent session – just select option “No, I don’t want…” and press button “Start”.

Select persist ZAP Session

Now open “Preferences” and ensure that ZAP API is enabled.

Enable ZAP API

Our last action for configuration is to enable ZAP Proxy.

ZAP Proxy

Start ZAP via command line

# show help (macOS)
$ /Applications/OWASP\ ZAP.app/Contents/MacOS/OWASP\ ZAP.sh -h

# show default directory (macOS)
$ ls -la ~/Library/Application\ Support/ZAP/

# start ZAP in daemon mode with specific port and apikey (macOS)
$ /Applications/OWASP\ ZAP.app/Contents/MacOS/OWASP\ ZAP.sh -daemon -port 8090 -config api.key=12345

# open ZAP API in browser
$ open http://localhost:8090/UI

Add URL (Site)

# add URL
$ curl -s "http://localhost:8090/JSON/core/action/accessUrl/?apikey=12345&url=https://www.webscantest.com&followRedirects=false" | jq .

Show ZAP Sites and Hosts

# list all sites
$ curl -s "http://localhost:8090/JSON/core/view/sites/?apikey=12345" | jq .

# list all hosts
$ curl -s "http://localhost:8090/JSON/core/view/hosts/?apikey=12345" | jq .

ZAP HTTP Sessions

# list all httpSession sites
$ curl -s "http://localhost:8090/JSON/httpSessions/view/sites/?apikey=12345" | jq .

# create new httpSession
$ curl -s "http://localhost:8090/JSON/httpSessions/action/createEmptySession/?apikey=12345&site=www.webscantest.com:443&session=session1" | jq .

# show active httpSession
$ curl -s "http://localhost:8090/JSON/httpSessions/view/activeSession/?apikey=12345&site=www.webscantest.com:443" | jq .

ZAP Spider scan

# start spider scan
$ curl -s "http://localhost:8090/JSON/spider/action/scan/?apikey=12345&zapapiformat=JSON&formMethod=GET&url=https://www.webscantest.com"

# show spider scan status
$ curl -s "http://localhost:8090/JSON/spider/view/status/?apikey=12345" | jq .

ZAP Context

# list all context
$ curl -s "http://localhost:8090/JSON/context/view/contextList/?apikey=12345" | jq .

# create context
$ curl -s "http://localhost:8090/JSON/context/action/newContext/?apikey=12345&contextName=Default+Context" | jq .

# show specific context
$ curl -s "http://localhost:8090/JSON/context/view/context/?apikey=12345&contextName=Default+Context" | jq .

# add regex into includeInContext
$ curl -s "http://localhost:8090/JSON/context/action/includeInContext/?apikey=12345&contextName=Default+Context&ex=https://www.webscantest.com.*" | jq .

# list all includeRegexs
$ curl -s "http://localhost:8090/JSON/context/view/includeRegexs/?apikey=12345&contextName=Default+Context" | jq .

ZAP Active scan

# start active scan
$ curl -s "http://localhost:8090/JSON/ascan/action/scan/?apikey=12345&zapapiformat=JSON&formMethod=GET&url=https://www.webscantest.com&recurse=&inScopeOnly=false&scanPolicyName=&method=&postData=&contextId="

# show active scan status
$ curl -s "http://localhost:8090/JSON/ascan/view/status/?apikey=12345" | jq .

ZAP alerts and reports

# list alert counts by url
$ curl -s "http://localhost:8090/JSON/alert/view/alertCountsByRisk/?apikey=12345&url=https://www.webscantest.com&recurse=True" | jq .

# list alerts by risk
curl -s "http://localhost:8090/JSON/alert/view/alertsByRisk/?apikey=12345&url=https://www.webscantest.com&recurse=True" | jq .

# show json report
$ curl -s "http://localhost:8090/OTHER/core/other/jsonreport/?apikey=12345" | jq .

# list all alerts
$ curl -s "http://localhost:8090/JSON/core/view/alerts/?apikey=12345" | jq .

ZAP shutdown

# shutdown
$ curl -s "http://localhost:8090/JSON/core/action/shutdown/?apikey=12345"

Running ZAP Attack Proxy on Jenkins

This tutorial will explain how easy you implement ZAP Attack Proxy into Jenkins. Therefor we create a Freestyle job and will use the “Official OWASP ZAP Jenkins Plugin“. That you can follow and reproduce the tutorial, you need a running Jenkins instance with SSH access to it and proper system rights (OS, Jenkins).

Install ZAP Attack Proxy

Following steps needs to be done when SSH connection, to Jenkins, is established.

# download installer script
$ wget https://github.com/zaproxy/zaproxy/releases/download/v2.8.0/ZAP_2_8_0_unix.sh

# set chmod of script
$ chmod +x ZAP_2_8_0_unix.sh

# execute installer script
$ ./ZAP_2_8_0_unix.sh

# add environment variable (ZAPROXY_HOME)
$ echo "ZAPROXY_HOME=/usr/local/bin/" >> /etc/environment

# restart Jenkins
$ systemctl restart jenkins

Note: If you don’t restart Jenkins after creating “ZAPROXY_HOME”, you will run into trouble like “java.lang.IllegalArgumentException: ZAP INSTALLATION DIRECTORY IS MISSING, PROVIDED [ null ]”

Install needed Jenkins PlugIn’s

Search for “OWAS ZAP” and for “HTML Publisher” plugins.

Jenkins Plugin OWASP ZAP
Official OWASP ZAP
Jenkins Plugin HTML Publisher
HTML Publisher

Configure Jenkins Freestyle job

All what we need is there, we can start to setup a Jenkins “Freestyle project” with the name “ZAPAttackProxy”.

Create new Jenkins Freestyle Project
Jenkins Freestyle Project

The next setting is optional… I recommend to find your own value (I go with 5 for that example).

Discard old builds
Max # of builds to keep

On every build (Jenkins job run) the workspace should be clean. Please enable the checkbox.

Delete workspace before build starts
Delete workspace before build starts

We add now the build step. This build step is available because of the PlugIn “Official OWASP ZAP“.

Add build step Execute ZAP
Build step: Execute ZAP

Now we have many fields to configure. We start to set the values for section “Admin Configurations”.

ZAP Admin Configuration
Admin Configuration

As we already installed ZAP and created the entry into /etc/environment, we can now use that variable.

ZAP Installation Method
Installation Method

For ZAP Home Directory we add the path to the workspace and let the build create the directory “.ZAP”. For Session Management we choose “Persist Session” and give filename “my_session”.

ZAP Home Directory and Session Management
Home Directory & Session Management

Under section “Session Properties” you add the Context Name “default” and for “Include in Context” you can add IP’s and/or Domains. For that example I choose “http://scanme.nmap.org/*”.

ZAP Session Properties
Session Properties

In section “Attack Method” you can choose different attack methods like Spider Scan and so on. Please set always a “Starting Point”. The settings here are self explainable.

ZAP Attack Method
Attack Method

Enable checkbox “Generate Reports” in section “Finalize Run”. Now enter a filename and select “XML” and “HTML” format.

ZAP Finalize Run
Finalize Run

Note: You can ignore the HTTP 404 error.

We are done! To provide on our job dashboard a link for HTML report, you can use now the HTML Publisher.

ZAP Publish HTML reports
Publish HTML reports

Execute the job and play with “Attack Methods”…

Nessus on AWS

Nessus is a vulnerability scanner from Tenable. In this tutorial I will show how you can install Nessus on AWS (Debian), how you connect your local browser and perform a simple network scan. You need only a AWS account (eq Free Tier), SSH and a web browser.

Note: Please have a look on that page about pentesting on AWS first.

Create new EC2 instance

Login into your AWS console (or use AWSCLI), create a new SecurityGroup with SSH port 22 only (inbound) and launch a new instance. Search for “Debian”…

AWS EC2 Debian 9
Debian 9 on AWS Maretplace

Press button “Select” and finish all needed following steps (save your keys). After your EC2 instance is ready check for IP or DNS and connect.

# connect via SSH to EC2 instance
$ ssh -i ~/.ssh/ admin@<instance>

# compile a list of locale definition files (optional)
$ sudo locale-gen UTF-8

Install Nessus

Open download page and select latest version for Debian (as I wrote this tutorial it was Nessus-8.5.1-debian6_amd64.deb). Confirm and download. Via SCP, in new terminal, you can upload the file to your EC2 instance.

# copy file from local to remote
$ scp -i ~/.ssh/ ~/Downloads/Nessus-8.5.1-debian6_amd64.deb  admin@<instance>:/tmp

Back to instance terminal … Now install and start Nessus.

# install package
$ sudo dpkg -i /tmp/Nessus-8.5.1-debian6_amd64.deb

# start Nessus
$ sudo /etc/init.d/nessusd start

Use Nessus

To make our life easier, we will create a simple SSH port-forward.

# create port-forwarding
$ ssh -i ~/.ssh/ -L 8834:127.0.0.1:8834 admin@<instance>

# open browser
$ open https://localhost:8834

Now you can open your favourite browser with URL: https://localhost:8834.

Nessus Initialization
Initialisation of Nessus

When the initialization has been completed successfully, login and create a new scan. Select “Basic Network Scan” and add URL: http://scanme.nmap.org. Select “Basic Network Scan” and “Port scan (common ports)” for scan settings. Save and start your created scan. Please be patient, the scan will take a while.

Nessus scan
Running Nessus scan

Create a scan report

After a while, the scan is complete. Now you can create a “Custom” report. BTW … feature is only available for completed scans. So select “Export” – “Custom” and generate the report.

Nessus Report
Create custom HTML report

Apache Guacamole

Apache Guacamole … What is it about? It’s a client-less remote gateway for Telnet, SSH, RDP and VNC. Client-less, because there is no need to install any plugin or additional software for users (clients). The client will use just the browser (also without any plugin). In this tutorial we will create a very simple environment via Vagrant and use Guacamole. Why the tutorial? Because I know a lot of testers for example – who work with Windows, who are not allowed to install any software (eq Putty) but still need access to environments. … Next point are for example public security groups on cloud providers. Here only one port would be needed to support different protocols on different hosts (incl. file transfer).

What we need?

Project preparation

# create project
$ mkdir -p ~/Projects/Guacamole/src

# change directory
$ cd ~/Projects/Guacamole/

# create needed files in root folder
$ touch {Vagrantfile,ShellProvisioner.sh}

# create needed files in root folder
$ touch ./src/{guacamole.properties,server.xml,user-mapping.xml,Xwrapper.config}

# show project (optional)
$ tree ~/Projects/Guacamole/
|____src
| |____guacamole.properties
| |____server.xml
| |____user-mapping.xml
| |____Xwrapper.config
|____ShellProvisioner.sh
|____Vagrantfile

Okay, via your favorite editor you now add the content of all files. All files inside directory “src” are configuration files (installed on Guacamole host).

# Hostname and port of guacamole proxy
guacd-hostname:      localhost
guacd-port:          4822
available-languages: en, de

auth-provider: net.sourceforge.guacamole.net.basic.BasicFileAuthenticationProvider
basic-user-mapping: /etc/guacamole/user-mapping.xml
<?xml version="1.0" encoding="UTF-8"?>
<Server port="-1" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
  <GlobalNamingResources>
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>
  <Service name="Catalina">
    <Connector port="55555" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />
    <Engine name="Catalina" defaultHost="localhost">
      <Realm className="org.apache.catalina.realm.LockOutRealm">
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
               resourceName="UserDatabase"/>
      </Realm>
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log" suffix=".txt"
               pattern="%h %l %u %t "%r" %s %b" />
      </Host>
    </Engine>
  </Service>
</Server>

This file (user-mapping.xml) is the configuration for all your connections.

<user-mapping>

  <authorize username="USERNAME" password="PASSWORD">
    <!--
    <connection name="Debian2: RDP Connection">
      <protocol>rdp</protocol>
      <param name="hostname">localhost</param>
      <param name="port">3389</param>
    </connection>
    -->

    <connection name="Debian2: VNC Connection">
      <protocol>vnc</protocol>
      <param name="hostname">localhost</param>
      <param name="port">5901</param>
      <param name="password">vagrant</param>
    </connection>

    <connection name="Debian2: SSH Connection">
      <protocol>ssh</protocol>
      <param name="hostname">localhost</param>
      <param name="port">22</param>
      <param name="username">vagrant</param>
    </connection>

    <connection name="Debian1: SSH Connection">
      <protocol>ssh</protocol>
      <param name="hostname">192.168.10.5</param>
      <param name="port">22</param>
      <param name="username">vagrant</param>
    </connection>

    <connection name="Debian2: Telnet Connection">
      <protocol>telnet</protocol>
      <param name="hostname">localhost</param>
      <param name="port">23</param>
      <param name="username">vagrant</param>
    </connection>
  </authorize>
</user-mapping>
allowed_users=anybody

The ShellProvisioner.sh includes all installation and configuration for Guacamole All examples are provided but for Debian RDP is currently not working and I commented out.

echo '>>>>Install some default packages<<<<<'
sudo apt update -y -q
sudo apt install -y -q build-essential htop libcairo2-dev libjpeg62-turbo-dev libjpeg-dev libpng-dev libossp-uuid-dev
# install optional guacamole packages eq FFmpeg, SSH
sudo apt install -y -q libavcodec-dev libavutil-dev libswscale-dev libpango1.0-dev libssh2-1-dev libssl-dev libvorbis-dev libwebp-dev

echo '>>>>Install and configure tomcat packages<<<<<'
sudo apt install -y -q tomcat9 tomcat9-admin tomcat9-common tomcat9-user
sudo mkdir -p /usr/share/tomcat9/.guacamole
sudo cp /tmp/guacamole/server.xml /etc/tomcat9/server.xml
sudo chown root:tomcat /etc/tomcat9/server.xml
sudo chmod 0640 /etc/tomcat9/server.xml

echo '>>>>Configure default guacamole directory structure<<<<<'
sudo cp -r /tmp/guacamole /etc/guacamole
sudo mkdir -p /etc/guacamole/{extensions,lib}
sudo chown -R root:root /etc/guacamole
sudo chmod 0640 /etc/guacamole/user-mapping.xml
sudo chown root:tomcat /etc/guacamole/user-mapping.xml
sudo ln -s /etc/guacamole/guacamole.properties /var/lib/tomcat9/.guacamole

echo '>>>>Install and configure telnet packages<<<<<'
sudo apt install -y -q telnetd libtelnet-dev

echo '>>>>Install and configure xrdp packages<<<<<'
# actualy broken becauce of freerdp2-dev on debian
# sudo apt install -y -q xrdp freerdp2-dev
# sudo cp /etc/guacamole/Xwrapper.config /etc/X11/Xwrapper.config
# sudo chown root:root /etc/X11/Xwrapper.config
# sudo chmod 0644 /etc/X11/Xwrapper.config
# sudo systemctl enable xrdp.service
# sudo systemctl enable xrdp-sesman.service
# sudo systemctl start xrdp
# sudo systemctl start xrdp-sesman

echo '>>>>Install and configure vnc packages'
sudo apt install -y -q xfce4 xfce4-goodies gnome-icon-theme tightvncserver libvncserver-dev libpulse-dev

echo '>>>>Install guacamole client and restart tomcat<<<<<'
curl -s -O -J -L "http://apache.org/dyn/closer.cgi?action=download&filename=guacamole/1.0.0/binary/guacamole-1.0.0.war"
sudo cp guacamole-1.0.0.war /var/lib/tomcat9/webapps/guacamole.war
sudo chown tomcat:tomcat /var/lib/tomcat9/webapps/guacamole.war
sudo systemctl restart tomcat9

echo '>>>>Install guacamole server<<<<<'
curl -s -O -J -L "http://apache.org/dyn/closer.cgi?action=download&filename=guacamole/1.0.0/source/guacamole-server-1.0.0.tar.gz"
tar xzf guacamole-server-1.0.0.tar.gz
cd guacamole-server-1.0.0/
# space after etc is wrong
sudo ./configure --with-init-dir=/etc /init.d
sudo make
sudo make install
sudo ldconfig
sudo update-rc.d guacd defaults

echo '>>>>Start guacamole server/daemon<<<<<'
sudo systemctl start guacd

echo '>>>>Show open ports<<<<<'
sudo lsof -i -P -n | grep LISTEN

echo '>>>>Start clean-up<<<<<'
sudo rm /etc/guacamole/Xwrapper.config
sudo rm /etc/guacamole/server.xml
sudo rm -fr /tmp/guacamole
sudo rm -fr /home/vagrant/guacamole-server-1.0.0s
sudo rm /home/vagrant/guacamole-server-1.0.0.tar.gz
sudo rm /home/vagrant/guacamole-1.0.0.war
# -*- mode: ruby -*-
# vi: set ft=ruby :

BOX_1_NAME = "debian-1-guacamole"
BOX_2_NAME = "debian-2-guacamole"
BOX_BASE = "generic/debian10"
BOX_RAM_MB = 1024
BOX_CPU_COUNT = 1
BOX_GUI = false
BOX_SYNC_DIR = true

Vagrant.configure("2") do |config|

  config.vm.define BOX_1_NAME do |deb1|
    deb1.vm.box = BOX_BASE
    deb1.vm.synced_folder ".", "/vagrant", disabled: BOX_SYNC_DIR
    deb1.vm.hostname = BOX_1_NAME
    deb1.vm.network "private_network", ip: "192.168.10.5"
    deb1.vm.provider "virtualbox" do |vb1|
      vb1.name = BOX_1_NAME
      vb1.cpus = BOX_CPU_COUNT
      vb1.memory = BOX_RAM_MB
      vb1.gui = BOX_GUI
    end
  end

  config.vm.define BOX_2_NAME do |deb2|
    deb2.vm.box = BOX_BASE
    deb2.vm.synced_folder ".", "/vagrant", disabled: BOX_SYNC_DIR
    deb2.vm.hostname = BOX_2_NAME
    deb2.vm.network "forwarded_port", guest: 55555, host: 55555
    # deb2.vm.network "forwarded_port", guest: 5901, host: 5901
    # deb2.vm.network "forwarded_port", guest: 3389, host: 3389
    # deb2.vm.network "forwarded_port", guest: 23, host: 2323
    deb2.vm.network "private_network", ip: "192.168.10.10"
    deb2.vm.provider "virtualbox" do |vb2|
      vb2.name = BOX_2_NAME
      vb2.cpus = BOX_CPU_COUNT
      vb2.memory = BOX_RAM_MB
      vb2.gui = BOX_GUI
    end
    deb2.vm.provision "file", source: "./src", destination: "/tmp/guacamole"
    deb2.vm.provision "shell", name: "install", path: "./ShellProvisioner.sh"
  end

end

Usage

First start-up the environment (via simple Vagrant command) and next start the VNC inside the box. You can do via vagrant ssh or you start the VNC via Browser (SSH).

# start environment (be patient)
$ vagrant up

# show status (optional)
$ vagrant status

# ssh into 2nd box
$ vagrant ssh debian-2-guacamole

# start VNC server on user vagrant
$ vncserver

# Password: vagrant
# Verify: vagrant
# Would you like to enter a view-only password (y/n)? n

# exit ssh into box
$ exit

# open browser with URL
$ open http://localhost:55555/guacamole

Now login with “USERNAME/PASSWORD” (see src/user-mapping.xml) on http://localhost:55555/guacamole. If everything works it should look like this:

Guacamole on browser

Please have a look here https://guacamole.apache.org/doc/gug/index.html to learn more about configuration and authentication. All files which we used in this tutorial are available via https://github.com/Lupin3000/GuacamoleExample.

Little SonarQube tutorial

In this tiny tutorial, I would like to introduce SonarQube usage. I will show the usage of SonarQube Server via Docker and will give some hints about the SonarQube Scanner. Therefore we create three very simple example files (html, css and javascript).

Requirements

Prepare your project

Okay,… first we start the Docker container for SoanrQube and create all necessary folders and files for our project.

# pull Docker image (optional)
$ docker pull sonarqube

# run Docker container
$ docker run -d --name sonarqube -p 9000:9000 sonarqube

# list all plugins (optional)
$ docker exec -it sonarqube ls extensions/plugins
...
sonar-css-plugin-1.0.3.724.jar
sonar-html-plugin-3.1.0.1615.jar
sonar-javascript-plugin-5.1.1.7506.jar
...

# show SonarQube configuration (optional)
$ docker exec -it sonarqube cat conf/sonar.properties

# open url in browser (admin/admin)
$ open http://localhost:9000

# create project directory
$ mkdir -p ~/Projects/SonarQubeExample/src && cd ~/Projects/SonarQubeExample

# create files for project
$ touch {src/index.html,src/styles.css,src/scripts.js,sonar-project.properties}

Content of created files:

# Default SonarQube server
sonar.host.url=http://localhost:9000

# Default SonarQube key
sonar.projectKey=SonarQube:Example

# Encoding of the source code (optional)
sonar.sourceEncoding=UTF-8

# Relative path to source code (optional)
sonar.sources=src

# Disable collection of SCM information (optional)
sonar.scm.disabled=True
<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>SonarQube Example</title>
  <link rel="stylesheet" href="styles.css">
</head>
<body>
  <h1>Hello world...</h1>
  <p id="demo">lorem <b>ipsum</b> dolor <i>sit</i> amet...</p>
  <button onclick="displayDate()">The time is?</button><br>
  <script src="scripts.js" type="text/javascript"></script>
</body>
</html>
function displayDate() {
  alert(Date());
}
@charset "UTF-8";

h1 {
  color: #c74;
}
p {
  margin-top: 5px;
}
demo {
  font-weight: bold;
}

Download sonar-scanner

You need to download the SonarQube Scanner by your self. You will find it here incl. all important informations.

# download macos version
$ curl -LOJ https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-3.3.0.1492-macosx.zip

# unzip and delete
$ unzip sonar-scanner-cli-3.3.0.1492-macosx.zip && rm sonar-scanner-cli-3.3.0.1492-macosx.zip

Note: At this point you could also move the scanner files to the right place and create a symbolic link. I will skip that step and use the path to binary.

Execute sonar-scanner

If not done till now, open SonarQube in your browser (http://localhost:9000) and login with admin/admin.

# create variable with timestamp
$ SCAN_VERSION="$(date +'%s')"

# execute sonar-scanner run
$ sonar-scanner-3.3.0.1492-macosx/bin/sonar-scanner -D sonar.version="$SCAN_VERSION" > sonar_log.txt

You should now be able to see the result of the scan in SonarQube.

SonarQube Scan Results

Analyzing the scan from command line

To see the results in SonarQube is perfect but now we will try to get them in our command-line.

# show content of sonar_log.txt (optional)
$ cat sonar_log.txt

# create variable with taskid
$ TASK_ID="$(cat < sonar_log.txt | grep "task?id=" | awk -F "id=" '/id=/{print $2}')"

# show detailed task status (optional)
$ curl -s "http://localhost:9000/api/ce/task?id=$TASK_ID" | jq -r .

# show task status
$ curl -s "http://localhost:9000/api/ce/task?id=$TASK_ID" | jq -r .task.status

# create variable with analysisid
$ ANALYSIS_ID="$(curl -s "http://localhost:9000/api/ce/task?id=$TASK_ID" | jq -r .task.analysisId)"

# show detailed quality gate status (optional)
$ curl -s "http://localhost:9000/api/qualitygates/project_status?analysisId=$ANALYSIS_ID" | jq -r .

# show quality gate status
$ curl -s "http://localhost:9000/api/qualitygates/project_status?analysisId=$ANALYSIS_ID" | jq -r .projectStatus.status

Simple Jenkins pipeline on AWS (Part 3)

Okay,… The pipeline has already two steps “Build” and “Deploy” running, but the last step “Test” is missing. In this part I will show a simple example with Python, Selenium and Docker (standalone-chrome) for test step.

Preconditions

Install additional packages on AWS EC2

There is a need to install additional packages on AWS EC2 Linux instance (Jenkins).

# start ssh connection
$ ssh -i ~/.ssh/ExampleKeyPair.pem ec2-user@<EC2 IP|DNS>

# change to root user
$ sudo su -

# install python pip
$ easy_install pip

# install virtualenv
$ pip install virtualenv

# exit root and go back to ec2-user
$ exit

# exit ec2-user (ssh connection)
$ exit

Create new files and folder (Project/Repository)

You need to create a new directory called “test”. Inside that directory you will create a file “example.py” with following content.

#!/usr/bin/env python
import unittest
from selenium import webdriver


class ExampleTest(unittest.TestCase):

    def setUp(self):
        """Start web driver"""
        options = webdriver.ChromeOptions()
        options.add_argument('--no-sandbox')
        options.add_argument('--headless')
        options.add_argument('--disable-gpu')

        self.driver = webdriver.Remote('http://0.0.0.0:4444/wd/hub', options.to_capabilities())
        self.driver.get("APPLICATION_URL")

    def test_search_headline(self):
        """TestCase 1"""
        title = 'DemoPipeline'
        assert title in self.driver.title

    def test_search_text(self):
        """TestCase 2"""
        element = self.driver.find_element_by_tag_name('body')
        assert element.text == 'Hello world...'

    def tearDown(self):
        """Stop web driver"""
        self.driver.quit()

if __name__ == "__main__":
    unittest.main(verbosity=2)

When you are done you have to modify the “Jenkinsfile” and the bash script “test.sh”.

pipeline {
  agent any
  parameters {
    string(name: 'REPONAME', defaultValue: 'example/nginx', description: 'AWS ECR Repository Name')
    string(name: 'ECR', defaultValue: '237724776192.dkr.ecr.eu-central-1.amazonaws.com/example/nginx', description: 'AWS ECR Registry URI')
    string(name: 'REGION', defaultValue: 'eu-central-1', description: 'AWS Region code')
    string(name: 'CLUSTER', defaultValue: 'ExampleCluster', description: 'AWS ECS Cluster name')
    string(name: 'TASK', defaultValue: 'ExampleTask', description: 'AWS ECS Task name')
  }
  stages {
    stage('BuildStage') {
      steps {
        sh "./cicd/build.sh -b ${env.BUILD_ID} -n ${params.REPONAME} -e ${params.ECR} -r ${params.REGION}"
      }
    }
    stage('DeployStage') {
      steps {
        sh "./cicd/deploy.sh -b ${env.BUILD_ID} -e ${params.ECR} -c ${params.CLUSTER} -t ${params.TASK}"
      }
    }
    stage('TestStage') {
      steps {
        sh "./cicd/test.sh -c ${params.CLUSTER} -t ${params.TASK}"
      }
    }
  }
}
#!/usr/bin/env bash

## shell options
set -e
set -u
set -f

## magic variables
declare CLUSTER
declare TASK
declare TEST_URL
declare -r -i SUCCESS=0
declare -r -i NO_ARGS=85
declare -r -i BAD_ARGS=86
declare -r -i MISSING_ARGS=87

## script functions
function usage() {
  local FILE_NAME

  FILE_NAME=$(basename "$0")

  printf "Usage: %s [options...]\n" "$FILE_NAME"
  printf " -h\tprint help\n"
  printf " -c\tset esc cluster name uri\n"
  printf " -t\tset esc task name\n"
}

function no_args() {
  printf "Error: No arguments were passed\n"
  usage
  exit "$NO_ARGS"
}

function bad_args() {
  printf "Error: Wrong arguments supplied\n"
  usage
  exit "$BAD_ARGS"
}

function missing_args() {
  printf "Error: Missing argument for: %s\n" "$1"
  usage
  exit "$MISSING_ARGS"
}

function get_test_url() {
  local TASK_ARN
  local TASK_ID
  local STATUS
  local HOST_PORT
  local CONTAINER_ARN
  local CONTAINER_ID
  local INSTANCE_ID
  local PUBLIC_IP

  # list running task
  TASK_ARN="$(aws ecs list-tasks --cluster "$CLUSTER" --desired-status RUNNING --family "$TASK" | jq -r .taskArns[0])"
  TASK_ID="${TASK_ARN#*:task/}"

  # wait for specific container status
  STATUS="PENDING"
  while [ "$STATUS" != "RUNNING" ]; do
    STATUS="$(aws ecs describe-tasks --cluster "$CLUSTER" --task "$TASK_ID" | jq -r .tasks[0].containers[0].lastStatus)"
  done

  # get container id
  CONTAINER_ARN="$(aws ecs describe-tasks --cluster "$CLUSTER" --tasks "$TASK_ID" | jq -r .tasks[0].containerInstanceArn)"
  CONTAINER_ID="${CONTAINER_ARN#*:container-instance/}"

  # get host port
  HOST_PORT="$(aws ecs describe-tasks --cluster "$CLUSTER" --tasks "$TASK_ID" | jq -r .tasks[0].containers[0].networkBindings[0].hostPort)"

  # get instance id
  INSTANCE_ID="$(aws ecs describe-container-instances --cluster "$CLUSTER" --container-instances "$CONTAINER_ID" | jq -r .containerInstances[0].ec2InstanceId)"

  # get public IP
  PUBLIC_IP="$(aws ec2 describe-instances --instance-ids "$INSTANCE_ID" | jq -r .Reservations[0].Instances[0].PublicIpAddress)"

  TEST_URL="$(printf "http://%s:%d" "$PUBLIC_IP" "$HOST_PORT")"
}

function clean_up() {
  # stop container
  if [ "$(docker inspect -f {{.State.Running}} ChromeBrowser)" == "true" ]; then
    docker rm -f ChromeBrowser
  fi

  # delete virtualenv
  if [ -d .env ]; then
    rm -fr .env
  fi
}

function run_selenium_test() {
  local TEST_TEMPLATE
  local TEST_FILE

  # clean up
  clean_up

  # pull image (standalone-chrome)
  docker pull selenium/standalone-chrome

  # run docker container (standalone-chrome)
  docker run -d -p 4444:4444 --name ChromeBrowser selenium/standalone-chrome

  # create and activate virtualenv
  virtualenv .env && source .env/bin/activate

  # install Selenium
  pip install -U selenium

  # read test template into variable
  TEST_TEMPLATE=$(cat ./test/example.py)

  # replace string with URL
  TEST_FILE="${TEST_TEMPLATE/APPLICATION_URL/$TEST_URL}"

  # save into final test file
  echo "$TEST_FILE" > ./test/suite.py

  # execute test
  python -B ./test/suite.py

  # deactivate virtualenv
  deactivate
}

## check script arguments
while getopts "hc:t:" OPTION; do
  case "$OPTION" in
    h) usage
       exit "$SUCCESS";;
    c) CLUSTER="$OPTARG";;
    t) TASK="$OPTARG";;
    *) bad_args;;
  esac
done

if [ "$OPTIND" -eq 1 ]; then
  no_args
fi

if [ -z "$CLUSTER" ]; then
  missing_args '-c'
fi

if [ -z "$TASK" ]; then
  missing_args '-t'
fi

## run main function
function main() {
  get_test_url
  printf "Test Application URL: %s\n" "$TEST_URL"

  run_selenium_test
}

main

# exit
exit "$SUCCESS"

Ensure that “example.py” has all needed permission rights. $ chmod +x example.py Commit all changes now and wait that the Jenkins job gets triggered (or trigger manually).

jenkins trigger with parameters

That’s already all… your job should execute all steps. This part is done super fast. 😉

Some last words

There is a lot of space for improvements here, but I think you learned already much and had some fun. Some hints now:

  • you can add any other test methods by your self on this step (eq. Performance- and Security tests)
  • Unit tests and Static Code Analysis could executed on build step (before create image)
  • check out AWS ECS Services
  • use a proxy for Jenkins and enable SSL
  • create other pipelines and ECS clusters to enable staging
  • create “Lifecycle policy rules” on ECR
  • use Git Webhook’s to trigger the Jenkins jobs
  • add a post step in your Jenkins pipeline to store metrics and/or inform about build status

Simple Jenkins pipeline on AWS (Part 2)

In previous tutorial I showed you how to create the environment and how to implement the build steps for Jenkins pipeline. Now I will show you to setup the deploy step.

Preconditions

AWS ECS Cluster

Create a very small AWS ECS cluster in region “Frankfurt” (eu-central-1). Therefore enter Amazon ECS Clusters and press button “Create Cluster”.

AWS ECS create cluster

Select template “EC2 Linux + Networking” and continue to next step.

AWS ECS cluster template

On section “Configure cluster” you give a name like “ExampleCluster”.

AWS ECS configure cluster

On section “Instance configuration” select “On-Demand Instance”, “t2.micro”, “1”, “22” and “None – unable to SSH”.

AWS ECS instance configuration

In the section “Networking” you have to be careful now. Your values ​​will be different from mine! Under VPC, select the same value as for the EC2 Jenkins instance (I selected default VPC). Now you can choose one of the subnets. We created the security group together with the EC2 Jenkins instance, so select “ExampleSecurityGroup” here.

AWS ECS networking

Okay, press button “Create” and wait till the cluster is created. The cluster creation can take a while, so please be patient.

AWS ECS Task Definition

The cluster is running and the “Task Definition” can be created. So press button “Create new Task Definition”.

AWS ECS task definition

Select “EC2” on page launch type compatibility and press button “Next step”.

AWS ECS task launch type

On section “Configure task and container definitions” set value “ExampleTask” for input field “Task Definition Name” and for “Network Mode” select “<default>”.

AWS ECS task definition name

On section “Container Definition” press button “Add Container”. A new window will slide in. Here give the “Container name” value “ExampleContainer”, add under image your latest version from ECR (my latest is 24). Set values “128” for “Memory Limits (MiB)”, “80:80” for “Port mappings” and press button “Add”.

AWS ECS task add container

You are done with your task definition configuration, scroll down and press button “Create”.

AWS IAM

Before we can go through the next steps, we need to adjust the group policy for “PipelineExampleGroup”. You must add the “AmazonECS_FullAccess” policy. _For our example this is okay, but never use this policy in production!_

AWS ECS IAM

Run task on ECS cluster (via Jenkins)

Now you only need to modify two files in your repository. Replace the content of “deploy.sh” and “Jenkinsfile” with following contents.

pipeline {
  agent any
  parameters {
    string(name: 'REPONAME', defaultValue: 'example/nginx', description: 'AWS ECR Repository Name')
    string(name: 'ECR', defaultValue: '237724776192.dkr.ecr.eu-central-1.amazonaws.com/example/nginx', description: 'AWS ECR Registry URI')
    string(name: 'REGION', defaultValue: 'eu-central-1', description: 'AWS Region code')
    string(name: 'CLUSTER', defaultValue: 'ExampleCluster', description: 'AWS ECS Cluster name')
    string(name: 'TASK', defaultValue: 'ExampleTask', description: 'AWS ECS Task name')
  }
  stages {
    stage('BuildStage') {
      steps {
        sh "./cicd/build.sh -b ${env.BUILD_ID} -n ${params.REPONAME} -e ${params.ECR} -r ${params.REGION}"
      }
    }
    stage('DeployStage') {
      steps {
        sh "./cicd/deploy.sh -b ${env.BUILD_ID} -e ${params.ECR} -c ${params.CLUSTER} -t ${params.TASK}"
      }
    }
    stage('TestStage') {
      steps {
        sh "./cicd/test.sh"
      }
    }
  }
}
#!/usr/bin/env bash

## shell options
set -e
set -u
set -f

## magic variables
declare ECR
declare CLUSTER
declare TASK
declare BUILD_NUMBER
declare -r -i SUCCESS=0
declare -r -i NO_ARGS=85
declare -r -i BAD_ARGS=86
declare -r -i MISSING_ARGS=87

## script functions
function usage() {
  local FILE_NAME

  FILE_NAME=$(basename "$0")

  printf "Usage: %s [options...]\n" "$FILE_NAME"
  printf " -h\tprint help\n"
  printf " -e\tset ecr repository uri\n"
  printf " -c\tset esc cluster name uri\n"
  printf " -t\tset esc task name\n"
  printf " -b\tset build number\n "
}

function no_args() {
  printf "Error: No arguments were passed\n"
  usage
  exit "$NO_ARGS"
}

function bad_args() {
  printf "Error: Wrong arguments supplied\n"
  usage
  exit "$BAD_ARGS"
}

function missing_args() {
  printf "Error: Missing argument for: %s\n" "$1"
  usage
  exit "$MISSING_ARGS"
}

## check script arguments
while getopts "he:c:t:b:" OPTION; do
  case "$OPTION" in
    h) usage
       exit "$SUCCESS";;
    e) ECR="$OPTARG";;
    c) CLUSTER="$OPTARG";;
    t) TASK="$OPTARG";;
    b) BUILD_NUMBER="$OPTARG";;
    *) bad_args;;
  esac
done

if [ "$OPTIND" -eq 1 ]; then
  no_args
fi

if [ -z "$ECR" ]; then
  missing_args '-e'
fi

if [ -z "$CLUSTER" ]; then
  missing_args '-c'
fi

if [ -z "$TASK" ]; then
  missing_args '-t'
fi

if [ -z "$BUILD_NUMBER" ]; then
  missing_args '-b'
fi

## run main function
function main() {
  local TASK_ARN
  local TASK_ID
  local ACTIVE_TASK_DEF
  local TASK_DEFINITION
  local TASK_DEF_ARN

  # list running task
  TASK_ARN="$(aws ecs list-tasks --cluster "$CLUSTER" --desired-status RUNNING --family "$TASK" | jq -r .taskArns[0])"
  TASK_ID="${TASK_ARN#*:task/}"

  # stop running task
  if [ -n "$TASK_ID" ] && [ "$TASK_ID" != "null" ]; then
    printf "INFO: Stop Task %s\n" "$TASK_ID"
    aws ecs stop-task --cluster "$CLUSTER" --task "$TASK_ID"
  fi

  # list active task definition
  ACTIVE_TASK_DEF="$(aws ecs list-task-definitions --family-prefix "$TASK" --status ACTIVE | jq -r .taskDefinitionArns[0])"

  # derigister task definition
  if [ -n "$ACTIVE_TASK_DEF" ]; then
    printf "INFO: Deregister Task Definition %s\n" "$ACTIVE_TASK_DEF"
    aws ecs deregister-task-definition --task-definition "$ACTIVE_TASK_DEF"
  fi

  # read task definition template
  TASK_DEFINITION=$(cat ./cicd/task_definition.json)

  # create new task definition file
  TASK_DEFINITION="${TASK_DEFINITION/URI/$ECR}"
  echo "${TASK_DEFINITION/NUMBER/$BUILD_NUMBER}" > ecs_task_definition.json

  # register new task definition
  TASK_DEF_ARN="$(aws ecs register-task-definition --cli-input-json file://ecs_task_definition.json | jq -r .taskDefinition.taskDefinitionArn)"

  # run task by task definition
  aws ecs run-task --task-definition "$TASK_DEF_ARN" --cluster "$CLUSTER"
}

main

# exit
exit "$SUCCESS"

Commit your changes and wait for build trigger (or trigger manually). After successful deployment, your ECS cluster will have a running task now. On section “Container” you can see the link.

AWS ECS cluster task container

Every time when you modify files and commit them into your Git repository, the pipeline will be triggered and latest version will be visible in browser.

That’s it with this part of the series. Cu soon in next part.