GitLab and Sitespeed.io

Within this tutorial I will explain shortly the combination of GitLab and Sitespeed.io. Normally GitLab provides this feature (Browser Performance Testing) but only with Premium or Silver editions. I imagine for this example that you know already the basics about GitLab pipelines, Docker in Docker and Sitespeed.io.

Preparation

As a first step you create a simple project directory with some files and one folder.

# create project directory
$ mkdir -p ~/Projects/SitespeedGitLab/ci

# change directory
cd ~/Projects/SitespeedGitLab

# create login file
$ vim ci/login.js

# create urls file
$ vim ci/urls.txt

# create pipeline file
$ vim .gitlab-ci.yml

# show structure (optional)
$ tree .
.
|__.gitlab-ci.yml
|__ci
  |__login.js
  |__urls.txt

Content of files

For the login we use the simplest version of a script which serves as pre script. You can expand it later as needed. If you do not need a login, you do not have to create this file (leave this out later in the command --preScript login.js).

module.exports = async function(context, commands) {

  // navigate to login page
  await commands.navigate('your domain');

  // add values into input fields
  await commands.addText.byId('username', 'input by id');
  await commands.addText.byId('password', 'input by id');

  // find the submit button and click it
  await commands.click.byClassName('Button');

  // wait to verify that you are logged in (incl. max time)
  return commands.wait.byId('specific tag with id',8000);
};

Let’s get to the next file. Here you simply enter all URLs to be checked. Each line represents a URL to be checked which (in our case) always precedes the login. There is also the possibility to login once for all sub-pages as well as various actions on the pages. In this case you would have to script everything (similar to pre/post scripts).

your url one
your url two
your url three

The files urls.txt and login.js have to be mounted inside the container. Therefore we choose the folder “ci”. After the successfully execution this folder will also provide the sitespeed reports.

Pipeline

The last step is the definition of GitLab pipeline. Here now a very simple example for .gitlab-ci.yml, with only one pipeline stage (test) and job (perf_tests). You can also expand this file later as you like (It’s already possible to build Docker images inside).

stages:
  - test

variables:
  DOCKER_HOST: tcp://localhost:2375/
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ""

default:
  image: docker:stable
  services:
    - name: docker:dind

perf_tests:
  stage: test
  variables:
    MOUNT_PATH: "$CI_BUILDS_DIR/ci/"
    DOCKER_IMAGE: 'sitespeedio/sitespeed.io'
    BROWSER: 'firefox'
    ITERATION: '1'
  artifacts:
    name: "performance-reports"
    paths:
      - "$CI_BUILDS_DIR/ci/sitespeed-result/your domain"
  script:
    - docker run --shm-size=1g --rm -v $MOUNT_PATH:/sitespeed.io $DOCKER_IMAGE -b $BROWSER -n $ITERATION --preScript login.js urls.txt

Okay … done. You can commit/push everything into GitLab. After successfully pipeline run you can access the reports as GitLab pipeline artifacts (but not via repository folder -> because if the job is done the container and reports are gone).

M3U8 download app for MacOS

Usually I work via Terminal but sometimes I don’t remember all parameters of a binary and search for such takes time. Same issue I had for ffmpeg downloads of M3U8 files. So I created a small apple script (for some dialogs) and saved this as very simple application. I use it regulary now and after all I thought to share here.

Preparation

A little preparation is needed, if you have ffmpeg binary allready installed you can skip to next section. So download the ffmpeg binary as an archive from https://www.ffmpeg.org/, unzip and follow next commands. In my example the binary was unzipped into folder “Downloads”.

# move ffmpeg binary
$ mv ~/Downloads/ffmpeg /usr/local/bin/ffmpeg

# set permissions
$ chmod +x /usr/local/bin/ffmpeg

# check version (optional)
$ ffmpeg -version

Apple Script

Open the Scripteditor and copy/paste the following script there.

#!/usr/bin/osascript

global theURL
global theOutputFolder
global theOutputFileName

on SetURL()
  set theTitle to "Video URL"

  try
    set theURLDialog to display dialog "What's the file URL?" default answer "" with title theTitle buttons {"Continue"}
    set theURL to text returned of theURLDialog
  on error
    quit
  end try

  if theURL as string is equal to "" then
    quit
  end if
end SetURL

on SetOutputFolder()
  try
    set theOutputFolder to choose folder with prompt "In what folder you will save the file?"
  on error
    quit
  end try
end SetOutputFolder

on SetOutputFileName()
  set theTitle to "File Name"

  try
    set theOutputFileNameDialog to display dialog "What's your target file name?" default answer "" with title theTitle buttons {"Continue"}
    set theOutputFileName to text returned of theOutputFileNameDialog
  on error
    quit
  end try

  if theOutputFileName as string is equal to "" then
    quit
  end if
end SetOutputFileName

on RunTerminal()
  set theTargetPath to POSIX path of theOutputFolder & theOutputFileName
  set theCommand to "ffmpeg -i " & theURL & " -c copy -bsf:a aac_adtstoasc " & theTargetPath

  tell application "Terminal"
    activate
    do script with command theCommand in window 1
  end tell
end RunTerminal

on quit
  display dialog "Thanks for trying this!" buttons {"Continue"}
  continue quit
end quit

on run
  SetURL()
  SetOutputFolder()
  SetOutputFileName()

  RunTerminal()
end run

Export

Now you can “save” or “export” the script as “app”.

Save/Export applescript as application
Save/Export AppleScript as app

If you don’t like the icon, you can change it. Download from the source of you choose an “.icns” file. Select the app and hit “Command + i” keys. Now drag the icon over the original icon and close info window. Ready … move it into Applications folder and use it.

Install tmux on macOS without Homebrew

Many people (including me) don’t want to use Homebrew. But if you search online for “How to install tmux on macOS” you will found mostly Homebrew descriptions. So here now the very easy solution without.

Requirements

In case that Command Line Tools are not installed, execute following command and follow the installation steps in new popup window. This process can take some time, so be patient.

# install Command Line Tools
$ xcode-select --install

# verify installation (optional)
$ ls /Library/Developer/CommandLineTools/usr/bin/

Installation

To execute next steps you need to download and unzip two release tarballs. Here you will found latest version of libevent and tmux (ncurses is already installed on macOS).

# change into unzipped libevent directory
$ cd Downloads/libevent-2.1.11-stable/

# check dependencies
$ ./configure

# build package
$ make

# install package
$ sudo make install

# change into unzipped tmux directory
$ cd Downloads/tmux-3.0a/

# check dependencies
$ ./configure

# build package
$ make

# install package
$ sudo make install

# show tmux version (optional)
$ tmux -V

That’s it already. If you’re new to tmux you should have a look on this Cheat Sheet.

Get IP of headless Virtualbox VM

This short article will describe how you will get quickly the IP for an headless running Virtualbox VM. For demonstration purpose, I have assigned an “Bridged Interface” on NIC 1. So the IP is dynamically allocated (IP address is being assigned by DHCP).

# start VM headless (if not running)
$ VBoxManage startvm --type headless "vm name"

# check VM state (optional)
$ VBoxManage showvminfo "vm name" | grep "State"

# show IP
$ VBoxManage guestproperty get "vm name" "/VirtualBox/GuestInfo/Net/0/V4/IP"
...
Value: 172.20.10.6
...

That was super easy … now you can connect via SSH (if SSH service is running).

# start SSH connection
$ ssh -C4 remote_user@172.20.10.6

Fix macOS wireless issues (hard way)

Sometimes I run into issues with my macOS wireless. Yes I know that most problems come by myself by wrong configurations. But here now a very handy short way which helped me already many times. Attention! You will need it to rejoin the network.

# change directory
$ cd /Library/Preferences/SystemConfiguration/

# list files (optional)
$ ls -la

# delete specific files
$ rm com.apple.airport.preferences.plist com.apple.network.identification.plist NetworkInterfaces.plist preferences.plist

Finally reboot the system and join your network. This forces the macOS to create all preference files again.

ZAP API Basics

In this tutorial, I’d like to share a few ZAP API basics. This should make it possible for anyone to integrate ZAP into various pipelines.

Requirements

  • ZAP installed
  • jq installed

Minimum configuration of ZAP

Start ZAP now, if you get asked for select the persistent session – just select option “No, I don’t want…” and press button “Start”.

Select persist ZAP Session

Now open “Preferences” and ensure that ZAP API is enabled.

Enable ZAP API

Our last action for configuration is to enable ZAP Proxy.

ZAP Proxy

Start ZAP via command line

# show help (macOS)
$ /Applications/OWASP\ ZAP.app/Contents/MacOS/OWASP\ ZAP.sh -h

# show default directory (macOS)
$ ls -la ~/Library/Application\ Support/ZAP/

# start ZAP in daemon mode with specific port and apikey (macOS)
$ /Applications/OWASP\ ZAP.app/Contents/MacOS/OWASP\ ZAP.sh -daemon -port 8090 -config api.key=12345

# open ZAP API in browser
$ open http://localhost:8090/UI

Add URL (Site)

# add URL
$ curl -s "http://localhost:8090/JSON/core/action/accessUrl/?apikey=12345&url=https://www.webscantest.com&followRedirects=false" | jq .

Show ZAP Sites and Hosts

# list all sites
$ curl -s "http://localhost:8090/JSON/core/view/sites/?apikey=12345" | jq .

# list all hosts
$ curl -s "http://localhost:8090/JSON/core/view/hosts/?apikey=12345" | jq .

ZAP HTTP Sessions

# list all httpSession sites
$ curl -s "http://localhost:8090/JSON/httpSessions/view/sites/?apikey=12345" | jq .

# create new httpSession
$ curl -s "http://localhost:8090/JSON/httpSessions/action/createEmptySession/?apikey=12345&site=www.webscantest.com:443&session=session1" | jq .

# show active httpSession
$ curl -s "http://localhost:8090/JSON/httpSessions/view/activeSession/?apikey=12345&site=www.webscantest.com:443" | jq .

ZAP Spider scan

# start spider scan
$ curl -s "http://localhost:8090/JSON/spider/action/scan/?apikey=12345&zapapiformat=JSON&formMethod=GET&url=https://www.webscantest.com"

# show spider scan status
$ curl -s "http://localhost:8090/JSON/spider/view/status/?apikey=12345" | jq .

ZAP Context

# list all context
$ curl -s "http://localhost:8090/JSON/context/view/contextList/?apikey=12345" | jq .

# create context
$ curl -s "http://localhost:8090/JSON/context/action/newContext/?apikey=12345&contextName=Default+Context" | jq .

# show specific context
$ curl -s "http://localhost:8090/JSON/context/view/context/?apikey=12345&contextName=Default+Context" | jq .

# add regex into includeInContext
$ curl -s "http://localhost:8090/JSON/context/action/includeInContext/?apikey=12345&contextName=Default+Context&ex=https://www.webscantest.com.*" | jq .

# list all includeRegexs
$ curl -s "http://localhost:8090/JSON/context/view/includeRegexs/?apikey=12345&contextName=Default+Context" | jq .

ZAP Active scan

# start active scan
$ curl -s "http://localhost:8090/JSON/ascan/action/scan/?apikey=12345&zapapiformat=JSON&formMethod=GET&url=https://www.webscantest.com&recurse=&inScopeOnly=false&scanPolicyName=&method=&postData=&contextId="

# show active scan status
$ curl -s "http://localhost:8090/JSON/ascan/view/status/?apikey=12345" | jq .

ZAP alerts and reports

# list alert counts by url
$ curl -s "http://localhost:8090/JSON/alert/view/alertCountsByRisk/?apikey=12345&url=https://www.webscantest.com&recurse=True" | jq .

# list alerts by risk
curl -s "http://localhost:8090/JSON/alert/view/alertsByRisk/?apikey=12345&url=https://www.webscantest.com&recurse=True" | jq .

# show json report
$ curl -s "http://localhost:8090/OTHER/core/other/jsonreport/?apikey=12345" | jq .

# list all alerts
$ curl -s "http://localhost:8090/JSON/core/view/alerts/?apikey=12345" | jq .

ZAP shutdown

# shutdown
$ curl -s "http://localhost:8090/JSON/core/action/shutdown/?apikey=12345"

Running ZAP Attack Proxy on Jenkins

This tutorial will explain how easy you implement ZAP Attack Proxy into Jenkins. Therefor we create a Freestyle job and will use the “Official OWASP ZAP Jenkins Plugin“. That you can follow and reproduce the tutorial, you need a running Jenkins instance with SSH access to it and proper system rights (OS, Jenkins).

Install ZAP Attack Proxy

Following steps needs to be done when SSH connection, to Jenkins, is established.

# download installer script
$ wget https://github.com/zaproxy/zaproxy/releases/download/v2.8.0/ZAP_2_8_0_unix.sh

# set chmod of script
$ chmod +x ZAP_2_8_0_unix.sh

# execute installer script
$ ./ZAP_2_8_0_unix.sh

# add environment variable (ZAPROXY_HOME)
$ echo "ZAPROXY_HOME=/usr/local/bin/" >> /etc/environment

# restart Jenkins
$ systemctl restart jenkins

Note: If you don’t restart Jenkins after creating “ZAPROXY_HOME”, you will run into trouble like “java.lang.IllegalArgumentException: ZAP INSTALLATION DIRECTORY IS MISSING, PROVIDED [ null ]”

Install needed Jenkins PlugIn’s

Search for “OWAS ZAP” and for “HTML Publisher” plugins.

Jenkins Plugin OWASP ZAP
Official OWASP ZAP
Jenkins Plugin HTML Publisher
HTML Publisher

Configure Jenkins Freestyle job

All what we need is there, we can start to setup a Jenkins “Freestyle project” with the name “ZAPAttackProxy”.

Create new Jenkins Freestyle Project
Jenkins Freestyle Project

The next setting is optional… I recommend to find your own value (I go with 5 for that example).

Discard old builds
Max # of builds to keep

On every build (Jenkins job run) the workspace should be clean. Please enable the checkbox.

Delete workspace before build starts
Delete workspace before build starts

We add now the build step. This build step is available because of the PlugIn “Official OWASP ZAP“.

Add build step Execute ZAP
Build step: Execute ZAP

Now we have many fields to configure. We start to set the values for section “Admin Configurations”.

ZAP Admin Configuration
Admin Configuration

As we already installed ZAP and created the entry into /etc/environment, we can now use that variable.

ZAP Installation Method
Installation Method

For ZAP Home Directory we add the path to the workspace and let the build create the directory “.ZAP”. For Session Management we choose “Persist Session” and give filename “my_session”.

ZAP Home Directory and Session Management
Home Directory & Session Management

Under section “Session Properties” you add the Context Name “default” and for “Include in Context” you can add IP’s and/or Domains. For that example I choose “http://scanme.nmap.org/*”.

ZAP Session Properties
Session Properties

In section “Attack Method” you can choose different attack methods like Spider Scan and so on. Please set always a “Starting Point”. The settings here are self explainable.

ZAP Attack Method
Attack Method

Enable checkbox “Generate Reports” in section “Finalize Run”. Now enter a filename and select “XML” and “HTML” format.

ZAP Finalize Run
Finalize Run

Note: You can ignore the HTTP 404 error.

We are done! To provide on our job dashboard a link for HTML report, you can use now the HTML Publisher.

ZAP Publish HTML reports
Publish HTML reports

Execute the job and play with “Attack Methods”…

Nessus on AWS

Nessus is a vulnerability scanner from Tenable. In this tutorial I will show how you can install Nessus on AWS (Debian), how you connect your local browser and perform a simple network scan. You need only a AWS account (eq Free Tier), SSH and a web browser.

Note: Please have a look on that page about pentesting on AWS first.

Create new EC2 instance

Login into your AWS console (or use AWSCLI), create a new SecurityGroup with SSH port 22 only (inbound) and launch a new instance. Search for “Debian”…

AWS EC2 Debian 9
Debian 9 on AWS Maretplace

Press button “Select” and finish all needed following steps (save your keys). After your EC2 instance is ready check for IP or DNS and connect.

# connect via SSH to EC2 instance
$ ssh -i ~/.ssh/ admin@<instance>

# compile a list of locale definition files (optional)
$ sudo locale-gen UTF-8

Install Nessus

Open download page and select latest version for Debian (as I wrote this tutorial it was Nessus-8.5.1-debian6_amd64.deb). Confirm and download. Via SCP, in new terminal, you can upload the file to your EC2 instance.

# copy file from local to remote
$ scp -i ~/.ssh/ ~/Downloads/Nessus-8.5.1-debian6_amd64.deb  admin@<instance>:/tmp

Back to instance terminal … Now install and start Nessus.

# install package
$ sudo dpkg -i /tmp/Nessus-8.5.1-debian6_amd64.deb

# start Nessus
$ sudo /etc/init.d/nessusd start

Use Nessus

To make our life easier, we will create a simple SSH port-forward.

# create port-forwarding
$ ssh -i ~/.ssh/ -L 8834:127.0.0.1:8834 admin@<instance>

# open browser
$ open https://localhost:8834

Now you can open your favourite browser with URL: https://localhost:8834.

Nessus Initialization
Initialisation of Nessus

When the initialization has been completed successfully, login and create a new scan. Select “Basic Network Scan” and add URL: http://scanme.nmap.org. Select “Basic Network Scan” and “Port scan (common ports)” for scan settings. Save and start your created scan. Please be patient, the scan will take a while.

Nessus scan
Running Nessus scan

Create a scan report

After a while, the scan is complete. Now you can create a “Custom” report. BTW … feature is only available for completed scans. So select “Export” – “Custom” and generate the report.

Nessus Report
Create custom HTML report