Hashicorp Vault SSH OTP

With Vault’s SSH secret engine you can provide an secure authentication and authorization for SSH. With the One-Time SSH Password (OTP) you don’t need to manage keys anymore. The client requests the credentials from the Vault service and (if authorized) can connect to target service(s). Vault will take care that the OTP can be used only once and the access is logged. This tutorial will provide needed steps on a simple Docker infrastructure. Attention, in that tutorial Vault and Vault-SSH-Helper are running in Development Mode – don’t do that in production!

Conditions

Vault server

Let’s start and prepare the vault service.

# run vault-service container (local)
$ docker run -ti --name vault-service bitnami/minideb /bin/bash

# install packages
$ apt-get update && apt-get install -y ntp curl unzip ssh sshpass

# download vault
$ curl -C - -k https://releases.hashicorp.com/vault/0.10.4/vault_0.10.4_linux_amd64.zip -o /tmp/vault.zip

# unzip and delete archive
$ unzip -d /tmp/ /tmp/vault.zip && rm /tmp/vault.zip

# move binary
$ mv /tmp/vault /usr/local/bin/

# start vault (development mode)
$ vault server -dev -dev-listen-address='0.0.0.0:8200'

Don’t stop or close terminal session! Open new terminal. Note: The IP’s I use in this tutorial may be different to yours.

# get IP of container (local)
$ docker inspect -f '{{ .NetworkSettings.IPAddress }}' vault-service
...
172.17.0.2
...

# run commands on container (local)
$ docker exec -ti vault-service /bin/bash

# set environment variable
$ export VAULT_ADDR='http://0.0.0.0:8200'

# enable ssh secret engine
$ vault secrets enable ssh

# create new vault role
$ vault write ssh/roles/otp_key_role key_type=otp default_user=root cidr_list=0.0.0.0/0

Target server

Now we create and configure the target service.

Note: Because of the security settings of my provider, spaces are after “etc”. Please delete it after copy/paste.

# run target-service container (local)
$ docker run -ti --name target-service bitnami/minideb /bin/bash

# install packages
$ apt-get update && apt-get install -y ntp curl unzip ssh vim

# download vault-ssh-helper
$ curl -C - -k https://releases.hashicorp.com/vault-ssh-helper/0.1.4/vault-ssh-helper_0.1.4_linux_amd64.zip -o /tmp/vault-ssh-helper.zip

# unzip and delete archive
$ unzip -d /tmp/ /tmp/vault-ssh-helper.zip && rm /tmp/vault-ssh-helper.zip

# move binary
$ mv /tmp/vault-ssh-helper /usr/local/bin/

# create directory
$ mkdir /etc /vault-ssh-helper.d

# add content to file
$ cat > /etc /vault-ssh-helper.d/config.hcl << EOL
vault_addr = "http://172.17.0.2:8200"
ssh_mount_point = "ssh"
ca_cert = "/etc /vault-ssh-helper.d/vault.crt"
tls_skip_verify = false
allowed_roles = "*"
EOL

# verify config (optional)
$ vault-ssh-helper -dev -verify-only -config=/etc /vault-ssh-helper.d/config.hcl

Pam SSHD configuration (on target server)

# modify pam sshd configuration
$ vim /etc /pam.d/sshd
...
#@include common-auth
auth requisite pam_exec.so quiet expose_authtok log=/tmp/vaultssh.log /usr/local/bin/vault-ssh-helper -dev -config=/etc /vault-ssh-helper.d/config.hcl
auth optional pam_unix.so not_set_pass use_first_pass nodelay
...

SSHD configuration (on target server)

# modify sshd_config
$ vim /etc /ssh/sshd_config
...
ChallengeResponseAuthentication yes
UsePAM yes
PasswordAuthentication no
PermitRootLogin yes
...
# start SSHD
$ /etc /init.d/ssh start

# echo some content into file (optional)
$ echo 'Hello from target-service' > /tmp/target-service

Client server

Last container is for simulating a client.

# run some-client container (local)
$ docker run -ti --name some-client bitnami/minideb /bin/bash

# install packages
$ apt-get update && apt-get install -y ntp curl unzip ssh sshpass

# download vault
$ curl -C - -k https://releases.hashicorp.com/vault/0.10.4/vault_0.10.4_linux_amd64.zip -o /tmp/vault.zip

# unzip and delete archive
$ unzip -d /tmp/ /tmp/vault.zip && rm /tmp/vault.zip

# move binary
$ mv /tmp/vault /usr/local/bin/

# set environment variable
$ export VAULT_ADDR='http://172.17.0.2:8200'

# authenticate as root (root token)
$ vault auth <root token>

Usage

Most work is already done. Now we use the demo environment.

# get container IP of target-service (local)
$ docker inspect -f '{{ .NetworkSettings.IPAddress }}' target-service
...
172.17.0.3
...

# get container IP of some-client (local)
$ docker inspect -f '{{ .NetworkSettings.IPAddress }}' some-client
...
172.17.0.4
...
# create an OTP credential (vault-service)
$ vault write ssh/creds/otp_key_role ip=172.17.0.3
$ vault write ssh/creds/otp_key_role ip=172.17.0.4

Note: Because of the security settings of my provider, spaces are after “root”. Please delete it after copy/paste.

# connect via vault SSH (vault-service)
$ vault ssh -role otp_key_role -mode otp -strict-host-key-checking=no root @172.17.0.3

# connect via vault SSH (some-client)
$ vault ssh -role otp_key_role -mode otp -strict-host-key-checking=no root @172.17.0.3

# read content of file (via SSH connections)
$ cat /tmp/target-service

# tail logfile (target-service)
$ tail -f /tmp/vaultssh.log

Start with Vault 0.10.x

HashiCorp released Vault version 0.10.x on April 2018. The 0.10.x release delivers many new features and changes (eq. K/V Secrets Engine v2, Vault Web UI, etc.). Please have a look on vault/CHANGELOG for more informations. This tiny tutorial will concentrate now on usage of Vault’s Key-Value Secrets Engine via CLI.

Preparation

# download version 0.10.3
$ curl -C - -k https://releases.hashicorp.com/vault/0.10.3/vault_0.10.3_darwin_amd64.zip -o ~/Downloads/vault.zip

# unzip and delete archive
$ unzip ~/Downloads/vault.zip -d ~/Downloads/ && rm ~/Downloads/vault.zip

# change access permissions and move binary to target
$ chmod u+x ~/Downloads/vault && sudo mv ~/Downloads/vault /usr/local/

Start Vault server in development mode

# start in simple development mode
$ vault server -dev

Do not stop the process and open new tab on terminal [COMMAND] + [t].

# set environment variable
$ export VAULT_ADDR='http://127.0.0.1:8200'

# check vault status
$ vault status

Create, Read, Update and Delete secrets

# create secret (version: 1)
$ vault kv put secret/demosecret name=demo value=secret

# list secrets (optional)
$ vault kv list secret

# read secret
$ vault kv get secret/demosecret

# read secret (JSON)
$ vault kv get --format json secret/demosecret

# update secret (version: 2)
$ vault kv put secret/demosecret name=Demo value=secret foo=bar

# read secret (latest version)
$ vault kv get secret/demosecret

# read secret (specific version)
$ vault kv get --version 1 secret/demosecret

# read secret (specific field)
$ vault kv get --field=name secret/demosecret

# delete secret (latest version)
$ vault kv delete secret/demosecret

# show metadata
$ vault kv metadata get secret/demosecret

As you can see, there are minor changes to previous versions of Vault.

Note: The API for the Vault KV secrets engine even changed.

# read (version 1)
$ curl -H "X-Vault-Token: ..." https://127.0.0.1:8200/v1/secret/demosecret

# read (version 2)
$ curl -H "X-Vault-Token: ..." https://127.0.0.1:8200/v1/secret/data/demosecret

Okay, back to CLI and some examples which are better for automation. We will use the STDIN and a simple JSON file.

# create secret (version: 1)
$ echo -n "my secret" | vault kv put secret/demosecret2 name=-

# list secrets (optional)
$ vault kv list secret

# update secret (version: 2)
$ echo -n '{"name": "other secret"}' | vault kv put secret/demosecret2 -

# create JSON file
$ echo -n '{"name": "last secret"}' > ~/Desktop/demo.json

# update secret (version: 3)
$ vault kv put secret/demosecret2 @$HOME/Desktop/demo.json

# read secrets (different versions)
$ vault kv get --version 1 secret/demosecret2
$ vault kv get --version 2 secret/demosecret2
$ vault kv get --version 3 secret/demosecret2

# delete version permanent
$ vault kv destroy --versions 3 secret/demosecret2

# show metadata
$ vault kv metadata get secret/demosecret2

Web UI

Previously the Web UI was for Enterprise only, now it has been made open source.

# open URL in browser
$ open http://localhost:8200/

Now you can use the root token to sign in.