Apache Guacamole

Apache Guacamole … What is it about? It’s a client-less remote gateway for Telnet, SSH, RDP and VNC. Client-less, because there is no need to install any plugin or additional software for users (clients). The client will use just the browser (also without any plugin). In this tutorial we will create a very simple environment via Vagrant and use Guacamole. Why the tutorial? Because I know a lot of testers for example – who work with Windows, who are not allowed to install any software (eq Putty) but still need access to environments. … Next point are for example public security groups on cloud providers. Here only one port would be needed to support different protocols on different hosts (incl. file transfer).

What we need?

Project preparation

# create project
$ mkdir -p ~/Projects/Guacamole/src

# change directory
$ cd ~/Projects/Guacamole/

# create needed files in root folder
$ touch {Vagrantfile,ShellProvisioner.sh}

# create needed files in root folder
$ touch ./src/{guacamole.properties,server.xml,user-mapping.xml,Xwrapper.config}

# show project (optional)
$ tree ~/Projects/Guacamole/
|____src
| |____guacamole.properties
| |____server.xml
| |____user-mapping.xml
| |____Xwrapper.config
|____ShellProvisioner.sh
|____Vagrantfile

Okay, via your favorite editor you now add the content of all files. All files inside directory “src” are configuration files (installed on Guacamole host).

# Hostname and port of guacamole proxy
guacd-hostname:      localhost
guacd-port:          4822
available-languages: en, de

auth-provider: net.sourceforge.guacamole.net.basic.BasicFileAuthenticationProvider
basic-user-mapping: /etc/guacamole/user-mapping.xml
<?xml version="1.0" encoding="UTF-8"?>
<Server port="-1" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
  <GlobalNamingResources>
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>
  <Service name="Catalina">
    <Connector port="55555" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />
    <Engine name="Catalina" defaultHost="localhost">
      <Realm className="org.apache.catalina.realm.LockOutRealm">
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
               resourceName="UserDatabase"/>
      </Realm>
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log" suffix=".txt"
               pattern="%h %l %u %t "%r" %s %b" />
      </Host>
    </Engine>
  </Service>
</Server>

This file (user-mapping.xml) is the configuration for all your connections.

<user-mapping>

  <authorize username="USERNAME" password="PASSWORD">
    <!--
    <connection name="Debian2: RDP Connection">
      <protocol>rdp</protocol>
      <param name="hostname">localhost</param>
      <param name="port">3389</param>
    </connection>
    -->

    <connection name="Debian2: VNC Connection">
      <protocol>vnc</protocol>
      <param name="hostname">localhost</param>
      <param name="port">5901</param>
      <param name="password">vagrant</param>
    </connection>

    <connection name="Debian2: SSH Connection">
      <protocol>ssh</protocol>
      <param name="hostname">localhost</param>
      <param name="port">22</param>
      <param name="username">vagrant</param>
    </connection>

    <connection name="Debian1: SSH Connection">
      <protocol>ssh</protocol>
      <param name="hostname">192.168.10.5</param>
      <param name="port">22</param>
      <param name="username">vagrant</param>
    </connection>

    <connection name="Debian2: Telnet Connection">
      <protocol>telnet</protocol>
      <param name="hostname">localhost</param>
      <param name="port">23</param>
      <param name="username">vagrant</param>
    </connection>
  </authorize>
</user-mapping>
allowed_users=anybody

The ShellProvisioner.sh includes all installation and configuration for Guacamole All examples are provided but for Debian RDP is currently not working and I commented out.

echo '>>>>Install some default packages<<<<<'
sudo apt update -y -q
sudo apt install -y -q build-essential htop libcairo2-dev libjpeg62-turbo-dev libjpeg-dev libpng-dev libossp-uuid-dev
# install optional guacamole packages eq FFmpeg, SSH
sudo apt install -y -q libavcodec-dev libavutil-dev libswscale-dev libpango1.0-dev libssh2-1-dev libssl-dev libvorbis-dev libwebp-dev

echo '>>>>Install and configure tomcat packages<<<<<'
sudo apt install -y -q tomcat9 tomcat9-admin tomcat9-common tomcat9-user
sudo mkdir -p /usr/share/tomcat9/.guacamole
sudo cp /tmp/guacamole/server.xml /etc/tomcat9/server.xml
sudo chown root:tomcat /etc/tomcat9/server.xml
sudo chmod 0640 /etc/tomcat9/server.xml

echo '>>>>Configure default guacamole directory structure<<<<<'
sudo cp -r /tmp/guacamole /etc/guacamole
sudo mkdir -p /etc/guacamole/{extensions,lib}
sudo chown -R root:root /etc/guacamole
sudo chmod 0640 /etc/guacamole/user-mapping.xml
sudo chown root:tomcat /etc/guacamole/user-mapping.xml
sudo ln -s /etc/guacamole/guacamole.properties /var/lib/tomcat9/.guacamole

echo '>>>>Install and configure telnet packages<<<<<'
sudo apt install -y -q telnetd libtelnet-dev

echo '>>>>Install and configure xrdp packages<<<<<'
# actualy broken becauce of freerdp2-dev on debian
# sudo apt install -y -q xrdp freerdp2-dev
# sudo cp /etc/guacamole/Xwrapper.config /etc/X11/Xwrapper.config
# sudo chown root:root /etc/X11/Xwrapper.config
# sudo chmod 0644 /etc/X11/Xwrapper.config
# sudo systemctl enable xrdp.service
# sudo systemctl enable xrdp-sesman.service
# sudo systemctl start xrdp
# sudo systemctl start xrdp-sesman

echo '>>>>Install and configure vnc packages'
sudo apt install -y -q xfce4 xfce4-goodies gnome-icon-theme tightvncserver libvncserver-dev libpulse-dev

echo '>>>>Install guacamole client and restart tomcat<<<<<'
curl -s -O -J -L "http://apache.org/dyn/closer.cgi?action=download&filename=guacamole/1.0.0/binary/guacamole-1.0.0.war"
sudo cp guacamole-1.0.0.war /var/lib/tomcat9/webapps/guacamole.war
sudo chown tomcat:tomcat /var/lib/tomcat9/webapps/guacamole.war
sudo systemctl restart tomcat9

echo '>>>>Install guacamole server<<<<<'
curl -s -O -J -L "http://apache.org/dyn/closer.cgi?action=download&filename=guacamole/1.0.0/source/guacamole-server-1.0.0.tar.gz"
tar xzf guacamole-server-1.0.0.tar.gz
cd guacamole-server-1.0.0/
# space after etc is wrong
sudo ./configure --with-init-dir=/etc /init.d
sudo make
sudo make install
sudo ldconfig
sudo update-rc.d guacd defaults

echo '>>>>Start guacamole server/daemon<<<<<'
sudo systemctl start guacd

echo '>>>>Show open ports<<<<<'
sudo lsof -i -P -n | grep LISTEN

echo '>>>>Start clean-up<<<<<'
sudo rm /etc/guacamole/Xwrapper.config
sudo rm /etc/guacamole/server.xml
sudo rm -fr /tmp/guacamole
sudo rm -fr /home/vagrant/guacamole-server-1.0.0s
sudo rm /home/vagrant/guacamole-server-1.0.0.tar.gz
sudo rm /home/vagrant/guacamole-1.0.0.war
# -*- mode: ruby -*-
# vi: set ft=ruby :

BOX_1_NAME = "debian-1-guacamole"
BOX_2_NAME = "debian-2-guacamole"
BOX_BASE = "generic/debian10"
BOX_RAM_MB = 1024
BOX_CPU_COUNT = 1
BOX_GUI = false
BOX_SYNC_DIR = true

Vagrant.configure("2") do |config|

  config.vm.define BOX_1_NAME do |deb1|
    deb1.vm.box = BOX_BASE
    deb1.vm.synced_folder ".", "/vagrant", disabled: BOX_SYNC_DIR
    deb1.vm.hostname = BOX_1_NAME
    deb1.vm.network "private_network", ip: "192.168.10.5"
    deb1.vm.provider "virtualbox" do |vb1|
      vb1.name = BOX_1_NAME
      vb1.cpus = BOX_CPU_COUNT
      vb1.memory = BOX_RAM_MB
      vb1.gui = BOX_GUI
    end
  end

  config.vm.define BOX_2_NAME do |deb2|
    deb2.vm.box = BOX_BASE
    deb2.vm.synced_folder ".", "/vagrant", disabled: BOX_SYNC_DIR
    deb2.vm.hostname = BOX_2_NAME
    deb2.vm.network "forwarded_port", guest: 55555, host: 55555
    # deb2.vm.network "forwarded_port", guest: 5901, host: 5901
    # deb2.vm.network "forwarded_port", guest: 3389, host: 3389
    # deb2.vm.network "forwarded_port", guest: 23, host: 2323
    deb2.vm.network "private_network", ip: "192.168.10.10"
    deb2.vm.provider "virtualbox" do |vb2|
      vb2.name = BOX_2_NAME
      vb2.cpus = BOX_CPU_COUNT
      vb2.memory = BOX_RAM_MB
      vb2.gui = BOX_GUI
    end
    deb2.vm.provision "file", source: "./src", destination: "/tmp/guacamole"
    deb2.vm.provision "shell", name: "install", path: "./ShellProvisioner.sh"
  end

end

Usage

First start-up the environment (via simple Vagrant command) and next start the VNC inside the box. You can do via vagrant ssh or you start the VNC via Browser (SSH).

# start environment (be patient)
$ vagrant up

# show status (optional)
$ vagrant status

# ssh into 2nd box
$ vagrant ssh debian-2-guacamole

# start VNC server on user vagrant
$ vncserver

# Password: vagrant
# Verify: vagrant
# Would you like to enter a view-only password (y/n)? n

# exit ssh into box
$ exit

# open browser with URL
$ open http://localhost:55555/guacamole

Now login with “USERNAME/PASSWORD” (see src/user-mapping.xml) on http://localhost:55555/guacamole. If everything works it should look like this:

Guacamole on browser

Please have a look here https://guacamole.apache.org/doc/gug/index.html to learn more about configuration and authentication. All files which we used in this tutorial are available via https://github.com/Lupin3000/GuacamoleExample.

Jenkins and Sitespeed.io

While surfing the internet I stumbled across Sitespeed.io. It’s a amazing collection of Open Source Tools, which make performance measuring for developers and testers super easy. I tried it out and was immediately impressed. Here’s a little tutorial on how to use Jenkins and Sitespeed.

Requirements

Docker (latest)

Environment setup

With minimal 2 commands the environment (via Docker) is already created. Most of the time will be needed for the plugins installation.

# create Project
$ mkdir -p ~/Projects/Sitespeed/target && cd ~/Projects/Sitespeed

# pull latest sitespeed image (optional)
$ docker pull sitespeedio/sitespeed.io:latest

# start Jenkins container
$ docker run -e JAVA_OPTS="-Dhudson.model.DirectoryBrowserSupport.CSP=\"sandbox allow-scripts; style-src 'unsafe-inline' *;script-src 'unsafe-inline' *;\"" --name jenkins -v $(pwd)/target:/var/jenkins_home -v /var/run/docker.sock:/var/run/docker.sock -v $(which docker):$(which docker) -p 8080:8080 -p 9000:9000 jenkins/jenkins:lts

# open Jenkins in browser (be patient)
$ open http://localhost:8080

On setup wizard finish: unlock Jenkins, install the suggested plugins, create an account and finish the instance configuration.

Jenkins permissions to /var/run/docker.sock

Before you start with Jenkins job configuration, ensure that user jenkins has permissions to /var/run/docker.sock.

# test permissions
$ docker exec -ti jenkins docker info Got permission denied...

# create group docker
$ docker exec -ti -u 0 jenkins groupadd -for -g 0 docker

# add jenkins to group
$ docker exec -ti -u 0 jenkins usermod -aG docker jenkins

# restart jenkins container
$ docker restart jenkins

Jenkins job configuration

When Jenkins is ready (restarted), install the HTML Publisher PlugIn (no restart after installation of plugin required).

Jenkins HTML Publisher Plugin

Create a new free-style project named SiteSpeed.

Jenkins SiteSpeed Project

Attention: You need to specify later the absolute path to the local directory /target/workspace/SiteSpeed. If you do not know how, press save and start the build without any job information (empty job configuration) and follow the optional instructions.

# change directory (optional)
$ cd ~/Projects/Sitespeed/target/workspace/SiteSpeed

# get absolute path (optional)
$ pwd

In my case the path is: “/Users/steffen/Projects/Sitespeed/target/workspace/SiteSpeed”. Under job configuration section “Build” enable “Execute shell” and paste following command.

docker run --rm --shm-size=1g -v /Users/steffen/Projects/Sitespeed/target/workspace/SiteSpeed:/sitespeed.io sitespeedio/sitespeed.io --visualMetrics --video --outputFolder output https://www.sitespeed.io/ -n 1

Via Post-Build-Action: Publish HTML reports you can enter the report very simple from the job project page.

Jenkins SiteSpeed Job Configuration

Save everything and run the job. After a short time you can look at the HTML report. See “Pages” > “https://www.sitespeed.io/” for screenshots, HAR and video files. On the website of sitespeed.io is a very detailed documentation and many more examples. Have fun!

Setup learning environment for security testing

In a previous tutorial, I showed you how to set up a security learning environment quickly. Since there are some changes now (ex: Webswing), I’ll do the tutorial again. By the way i use macOS, some commands could be different but similar for Linux or Windows.

Requirements

  • Docker (17.12.0-ce)
  • Safari (11.02)
  • Firefox (58.0)

Search and download needed docker images

# search DVWA image (optional)
$ docker search dvwa

# pull DVWA image
$ docker pull citizenstig/dvwa

# search ZAP image (optional)
$ docker search zap

# pull ZAP image
$ docker pull owasp/zap2docker-stable

# search ThreadFix image (optional)
$ docker search threadfix

# pull ThreadFix image
$ docker pull jmbmxer/threadfix

# list images (optional)
$ docker images
...
REPOSITORY                 TAG       IMAGE ID        CREATED         SIZE
owasp/zap2docker-stable    latest    40848e80b7fb    2 months ago    1.33GB
jmbmxer/threadfix          latest    b6f1907a61cd    22 months ago   941MB
citizenstig/dvwa           latest    c8312743bc09    3 years ago     478MB
...

Run DVWA container

# run DVWA container
$ docker run -d -p 8081:80 --name dvwa citizenstig/dvwa

# check DVWA logs for startup (optional)
$ docker logs -f dvwa

# get local ip
$ ipconfig getifaddr en0
...
192.168.192.39
...

# start DVWA in browser
$ open -a Safari http://192.168.192.39:8081/

DVWA inside Browser

…Setup/create new database…

setup dvwa

Run ZAP container

# create folder
$ mkdir -p /tmp/reports

# run ZAP container
$ docker run -u zap -i -p 8080:8080 -p 8090:8090 -v /tmp/reports:/home/zap/reports --name zap owasp/zap2docker-stable zap-webswing.sh

# start ZAP in Browser
$ open -a Safari http://localhost:8080/?anonym=true&app=ZAP

ZAP inside browser

Please check via “Tools” -> “Options” -> “Local Proxies” the right configuration!

ZAP Proxy configuration

You need do use the non-routable meta address (0.0.0.0)!

Run ThreadFix container

# run ThreadFix container
$ docker run -d -p 8443:8443 --name threadfix jmbmxer/threadfix start

# check ThreadFix logs for startup (optional)
$ docker logs -f threadfix
...
Jan 30, 2018 8:56:40 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 49986 ms
...

# start ThreadFix in Browser
$ open -a Safari https://localhost:8443/threadfix/

Login into ThreadFix with “user/password

ThreadFix login

Create new team with application.

Configure Firefox

# get local ip
$ ipconfig getifaddr en0
...
192.168.192.39
...

# open Firefox preferences
$ /Applications/Firefox.app/Contents/MacOS/firefox --preferences

Configure manual proxy with your local IP and 2nd ZAP port (8090). Afterwards you enable checkbox “Use this proxy server for all protocols” and press OK button.

Firefox proxy configuration

When all the configurations have been made, you can temporarily quit Firefox.

Use environment

# list all containers (optional)
$ docker ps -a

# open DVWA via Proxy
$ open -a firefox http://192.168.192.39:8081

Start recording and save your XML report “Report -> Generate XML Report”.

Save ZAP XML report

# open reports directory in finder
$ open /tmp/reports

Import XML report

Next steps

Now it’s time to study! If you need help, you can found a lot of training materials on Youtube.

Create a simple video test environment (Part 2)

In the first part we created the video test environment and you learned how to extend it. At the end of this tutorial you will know how to embed video content in the video test environment. Therefore, a few basics are shown around ffmpeg (how to create, edit and use videos).

Record and prepare some videos

The recording should contain video and sound and should be 5 minutes long. The content of the video does not matter!

# open Quicktime Player
$ open -a "QuickTime Player"

# press Control-Command-N, start record (approximately 5 min)
# save record into project folder as movie.mov (~/Projects/VideoTest/movie.mov)

As soon as a video is ready we have to create more.

# copy binary (optional)
$ sudo cp ~/Projects/VideoTest/ffmpeg /usr/local/bin/ffmpeg && sudo chmod a+rx /usr/local/bin/ffmpeg

# convert mov into mp4 (copy)
$ ffmpeg -i movie.mov -vcodec copy -acodec copy demo.mp4

# resize mp4 to 320x240 (filter_graph)
$ ffmpeg -i demo.mp4 -vf scale=320:240 ./src/demo_scaled.mp4

# create poster from mp4 (position and frame)
$ ffmpeg -i ./src/demo_scaled.mp4 -ss 00:00:30 -vframes 1 ./src/demo_poster.png

# create m3u8/ts files from mp4 (HLS - Apple HTTP Live Stream)
$ ffmpeg -i demo.mp4 -b:v 1M -g 60 -hls_time 2 -hls_list_size 0 -hls_segment_size 500000 ./src/output.m3u8

# run specific SHELL provisioner
$ vagrant provision --provision-with video

Note: After this step you will have many video files which you will use

  • ./movie.mov (original)
  • ./demo.mp4 (converted)
  • ./src/demo_scaled.mp4 (converted and resized)
  • ./src/output.m3u8
  • ./src/\*.ts

Get in contact with ffmpeg

I assume that ffmpeg is properly installed and the test environment is running.

# create target folder
$ mkdir ~/Projects/VideoTest/test

# extract some images from video
$ ffmpeg -i movie.mov -ss 00:00:30 -t 0.1 -f image2 -qscale 2 -vcodec mjpeg ./test/img-%03d.jpg

# create local m3u8/ts files from mp4
$ ffmpeg -i demo.mp4 -b:v 1M -g 60 -hls_time 2 -hls_list_size 0 -hls_segment_size 500000 ./test/output.m3u8

# extract mp4 from local m3u8/ts files
$ ffmpeg -i test/output.m3u8 -bsf:a aac_adtstoasc -vcodec copy -c copy -crf 50 ./test/output_local.mp4

# extract mp4 from url to m3u8 file (will not work with LiveStream)
$ ffmpeg -i http://localhost:8080/output.m3u8 -c copy -bsf:a aac_adtstoasc stream.mp4

Stream videos

# open browser
$ open -a Safari http://localhost:8080/livestream.html

# stream video (Real-Time Messaging Protocol)
$ ffmpeg -re -i demo.mp4 -vcodec libx264 -vprofile baseline -g 30 -acodec aac -strict -2 -f flv rtmp://localhost/show/stream

Stream from FaceTime HD Camera (macOS)

# open browser
$ open -a Safari http://localhost:8080/livestream.html

# list devices
$ ffmpeg -f avfoundation -list_devices true -i ""

# stream sound and video (Real-Time Messaging Protocol)
$ ffmpeg -f avfoundation -framerate 30 -i "0:0" -pix_fmt yuv420p -vcodec libx264 -vprofile baseline -g 30 -acodec libmp3lame -f flv rtmp://localhost/show/stream

Create a simple video test environment

In this series I would like to clarify the following questions. How can you test local videos and videos provided by a server? What tools are there? How could the test environment look like? … So these tutorials should provide an entry into the subject of video testing. In this part, we will provide the test environment.

Specification

additional Software

Files

# create project
$ mkdir -p ~/Projects/ && cd ~/Projects/

# clone all files from repository
$ git clone https://github.com/Lupin3000/VideoTest.git

# change directory
$ cd ~/Projects/VideoTest

Project Structure

$ tree .
├── src
│   ├── directstream.html
│   ├── index.html
│   ├── livestream.html
│   ├── nginx.conf
│   └── simplevideo.html
└── Vagrantfile

Prepare and run your test environment

The test environment will created by Vagrant. The Vagrant Base box needs to be provided with Debian 9. Therefore you have now 2 options. You can use a Debian 9 Base box from Vagrant cloud or you create your own. To create your own Debian 9 Base box you can use my Packer Git repository.

Please ensure, that the  Vagrantfile is properly configured (config.vm.box_url, config.vm.box) before start-up environment.

# modify Vagrantfile (box name)
$ vim Vagrantfile

# start new environment
$ vagrant up --provision-with install,prepare,start

# open in browser
$ open -a Safari http://localhost:8080/

After successful start you will find some informations on start page about How-to create, modify, upload and stream videos. In second tutorial we will have a detailed look on it.

StartPage Video Test Environment

To understand the background somewhat better, take a look!

# tail nginx log files (optional)
$ vagrant ssh -c 'sudo tail -f /usr/local/nginx/logs/*.log'

# show content of directory (optional)
$ vagrant ssh -c 'sudo ls -la /tmp/hls/'

Develop some more

There are even more ways to publish video files (Media Streaming Server). You can easily expand the environment. Just customize/create the configurations and files in the “src” folder as well as the Vagrantfile. For Dynamic Adaptive Streaming over HTTP (DASH) support take a look here. This link opens the commercial software solution.

Note: You simply link the IDE with the “src” folder. To change the images and videos, just run vagant provisioner.

# run specific SHELL provisioner (video)
$ vagrant provision --provision-with video

# restart after configuration
$ vagrant up --provision-with stop,prepare,start

Download and install ffmpeg binaries on macOS

Currently official ffmpeg packages are available for Windows, Linux (Debian, Ubuntu, Fedora, RedHat) and macOS. You can download latest versions here.

Here now a solution for macOS users, if you don’t like to install many additional software on your system (static FFmpeg binaries).

# download ffmpeg
$ curl -C - -k https://evermeet.cx/ffmpeg/ffmpeg-3.3.3.7z -o ~/Downloads/ffmpeg-3.3.3.7z

# install debian package
$ vagrant ssh -c 'sudo apt install -y p7zip-full'

# copy archive (into shared folder)
$ cp ~/Downloads/ffmpeg-3.3.3.7z ~/Projects/VideoTest/src/

# unzip archive
$ vagrant ssh -c '7z x /home/vagrant/src/ffmpeg-3.3.3.7z'

# mv binary into src
$ vagrant ssh -c 'mv /home/vagrant/ffmpeg /home/vagrant/src/ffmpeg'

# delete archive (on shared folder)
$ rm ~/Projects/VideoTest/src/ffmpeg-3.3.3.7z

# mv binary into project folder (from shared folder)
$ mv ~/Projects/VideoTest/src/ffmpeg ~/Projects/VideoTest/

# ensure binary is executable
$ chmod +x ffmpeg

Now some basic ffmpeg commands, which should work now. If you like,you can move the binary into directory “/usr/local/”.

# show version
$ ~/Projects/VideoTest/ffmpeg -version

# show help
$ ~/Projects/VideoTest/ffmpeg -help

# list codecs
$ ~/Projects/VideoTest/ffmpeg -codecs

# list formats
$ ~/Projects/VideoTest/ffmpeg -formats

Okay, that’s it for first tutorial.

Docker for Mac with insecure private registry

Sometimes you need an own Docker registry for testing purpose. Here a simple way to setup and use a private insecure registry. For production – don’t do that!

Requirements

Create insecure repository

SSH into your dedicated server…

# create directory
$ mkdir /opt/registry-data

# run docker registry
$ docker run -d --name registry --restart=always -p 5000:5000 -v /opt/registry-data:/var/lib/registry registry:2

# verify container run (optional)
$ docker ps -a

# logout
$ exit

Configure Docker for Mac

Start Docker for Mac and open “Preferences” – “Daemon”. Here just insert the IP plus specific port. When you are done press “Apply and Restart” button.

Docker for Mac Insecure registry

Build new image, tag and push

Create a new Dockerfile.

FROM alpine:latest

# install needed packages
RUN apk --update add wget

# download archive
RUN wget -q --no-check-certificate https://storage.googleapis.com/shellcheck/shellcheck-latest.linux.x86_64.tar.xz

# unzip archive
RUN tar xvfJ shellcheck-latest.linux.x86_64.tar.xz

# move binary
RUN mv /shellcheck-latest/shellcheck /usr/local/bin/shellcheck

# cleanup
RUN apk del wget
RUN rm -f shellcheck-latest.linux.x86_64.tar.xz
RUN rm -fr shellcheck-latest/

# change to mount directory
WORKDIR /mnt

# set entrypoint
ENTRYPOINT ["/usr/local/bin/shellcheck"]

Now build the image, tag the image and push to your private registry.

# build image
$ docker build -t alpine/shellcheck .

# tag image
$ docker tag alpine/shellcheck <IP>:5000/shellcheck

# push to private registry
$ docker push <IP>:5000/shellcheck

Use insecure registry

# remove all local images
$ docker rmi alpine/shellcheck
$ docker rmi <IP>:5000/shellcheck

# pull image from private registry
$ docker pull <IP>:5000/shellcheck

# list images (optional)
$ docker images

Additional

You can see the images stored on your registry. Therefor SSH into your dedicated server again.

# list content (optional)
$ ls -la /opt/registry-data/docker/registry/v2/repositories/

Simple Vault introduction

Today a tiny introduction to Vault from HashiCorp. I will just show the simplest usage. But this will help to get a first idea of Vault and the features.

Requirements

Preparation

# download vault (0.8.0)
$ curl -C - -k https://releases.hashicorp.com/vault/0.8.0/vault_0.8.0_darwin_amd64.zip -o ~/Downloads/vault.zip

# unzip and delete archive
$ unzip ~/Downloads/vault.zip && rm ~/Downloads/vault.zip

# move binary to target
$ sudo mv ~/Downloads/vault /usr/local/

Start Vault Server

# start in DEV mode
$ vault server -dev
...
Root Token: 6fdbf7b1-56a2-e665-aa31-0e3b5add5b77
...

Copy Root Token value to clipboard!!!

Insomnia

Create new environment “vault” under “Manage Environments” and store here your URL as “base_url” and Root Token as “api_key”.

insomnia vault environment

Now we create 4 simple requests

insomnia requests

for all requests we add Header

insomnia header

For first URL (POST: Add new secret) we use “{{ base_url }}/secret/MyFirstSecret” and we add following body as JSON.

{
  "value":"myNewSecret"
}

After send the key:value is stored inside Vault. You can modify the request (e.q. “{{ base_url }}/secret/MySecondSecret”) and send some more.

Our next request is to show all keys (GET: Get list of secret keys) “{{ base_url }}/secret?list=true”. The Preview will show similar output.

insomnia get vault keys

3rd request is to get the value from a specific key (GET: Get value of specific secret) “{{ base_url }}/secret/MySecret”.

insomnia get vault value

Last request is for delete (DEL: Delete specific secret) “{{ base_url }}/secret/MySecret”.

Tipp: if you lost the root token (Vault server is running) you can find the value!

# show file content
$ cat ~/.vault-token

Shell linter evaluation and usage

Tomorrow, the 1st of August is a national holiday in Switzerland … So I do one day off and have some time. For a long time I wanted to deal with Shell lint. After some research, i found a few open-source tools. By the way … linters are being written for many programming languages and document formats.

Preparation

For evaluation i will not install the tools on my local system,… so Vagrant (with CentOS 7) is my choice.

# create project
$ mkdir -p ~/Projects/ShellLint && cd ~/Projects/ShellLint

# create example.sh
$ vim example.sh

# create Vagrantfile
$ vim Vagrantfile
#!/bin/bash

declare -r VERSION="1.0.0"
declare -r FILE_NAME=$(basename "$0")

function fc_usage()
{
 printf "Usage: %s" "$FILE_NAME"
 printf " [-h] [-V]\n"
}

function failure()
{
 print "here is a error"

syntax() {
 print "this line has simply to many chars ... with a simple shell lint you should see"
}

function fc_bashism()
{
 echo -e "hello world"
}

function main()
{
 fc_usage
 fc_bashism
}

exit 0
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
  config.vm.box = "lupin/centos7"
  config.vm.box_check_update = false
  config.vm.synced_folder '.', '/vagrant', disabled: true

  config.vm.provision "file", source: "example.sh", destination: "example.sh"

  config.vm.provider "virtualbox" do |vb|
    vb.gui = false
    vb.name = "ShellLint"
    vb.memory = "1024"
    vb.cpus = 1
  end

end

Note: I created the Vagrant box “lupin/centos” via Packer … here my GitHub repository.

# create environment
$ vagrant up

# SSH into VM
$ vagrant ssh

Shell option -n

Many shell’s already offer a very simple script analysis. The option -n read commands in script, but do not execute them (syntax check).

# example bash
$ bash -n example.sh

# example shell
$ sh -n example.sh

Okay … but not really what I want… (more details are welcome)

shlint and checkbashisms

I found the repository here.

# install ruby (optional)
$ yum install -y ruby

# install json_pure (optional)
$ gem install json_pure

# install shlint
$ gem install shlint

# create config
$ echo -e 'shlint_shells="bash sh"\nshlint_debug=1' > .shlintrc

# run shlint
$ shlint example.sh

# run checkbashisms
$ checkbashisms -f example.sh

Note: for both tools you should change the shebang to “#!/bin/sh”

For shlint… I don’t get it. For checkbashisms … good if will write portable Shell scripts.

bashate

I found it here on Pypi.

# install epel repository
$ yum install -y epel-release

# install pip (python 2.x)
$ yum install -y python2-pip

# install bashate
$ pip install bashate

# run bashate
$ bashate example.sh

Nice … but not really all Standards.

Shellsheck

Shellcheck is known! Here the online service and here the repository.

# install epel repository
$ yum install -y epel-release

# update (optional)
$ yum update -y

# install ShellCheck
$ yum install -y ShellCheck

# run ShellCheck
$ shellcheck example.sh

I stay with that tool. Currently there are packages for almost every known OS.

Additional

Who knows me … knows that I do not like Installer and prefer Docker use. Here’s some fun.

# exit Vagrant and destroy
$ vagrant halt && vagrant destroy -f

# create Dockerfile
$ vim Dockerfile

# create Applescript
$ vim linter.scpt

# build Docker image
$ docker build -t alpine/shellcheck .

# change permission
$ chmod +x linter.scpt

# run Applescript
$ osascript linter.scpt
FROM alpine:latest

# install needed packages
RUN apk --update add wget

# download archive
RUN wget -q --no-check-certificate https://storage.googleapis.com/shellcheck/shellcheck-latest.linux.x86_64.tar.xz

# unzip archive
RUN tar xvfJ shellcheck-latest.linux.x86_64.tar.xz

# move binary
RUN mv /shellcheck-latest/shellcheck /usr/local/bin/shellcheck

# cleanup
RUN apk del wget
RUN rm -f shellcheck-latest.linux.x86_64.tar.xz
RUN rm -fr shellcheck-latest/

# change to mount directory
WORKDIR /mnt

# set entrypoint
ENTRYPOINT ["/usr/local/bin/shellcheck"]
#!/usr/bin/osascript

-- define global variables
global appName
global imageName

-- set magic values
set appName to "Docker"
set imageName to "alpine/shellcheck "

-- run docker linters
on LintShell(macPath)
	set posixPath to quoted form of POSIX path of macPath
	set fileName to do shell script "basename " & posixPath
	set dirName to do shell script "dirname " & posixPath
	set shellCmd to "docker run --rm -i -v " & dirName & ":/mnt " & imageName & fileName

	tell application "Terminal"
		set shell to do script shellCmd in front window
	end tell
end LintShell

-- display select box
on SelectFile()
	set dlTitle to "Nothing selected..."
	set dlMsg to "Process is terminated."

	try
		set macPath to choose file
	on error
		display dialog dlMsg buttons ["OK"] with title dlTitle
		return dlMsg
	end try

	LintShell(macPath)
end SelectFile

-- start Docker
on StartDocker()
	set dlTitle to "Docker cannot started"
	set dlMsg to "Something went wrong, could not start Docker!"

	try
		tell application appName
			activate
		end tell
	on error
		display dialog dlMsg buttons ["OK"] with title dlTitle
	end try
end StartDocker

-- check if Docker is running
on RunScript()
	set dlTitle to "Docker not running"
	set dlMsg to "Should Docker started?"

	if application appName is running then
		SelectFile()
	else
		display dialog dlMsg buttons ["OK", "No"] with title dlTitle
		if button returned of result is "OK" then
			StartDocker()
		end if
	end if
end RunScript

RunScript()

😉 just for fun…