Snippets

Making vim.tiny an alternative for vim
Artifactory NuGet feed for PowerShell modules
Breaking out of containers
Custom format for docker ps
Populate the Docker build cache
Docker Compose network context
Selecting an init process for containers
Docker lessons learned
Docker locale
Docker and proxies
Docker lessons learned
Docker, timezone and time sync
Docker user namespace mapping
AutoSize for all tables in PowerShell
Example Docker storage plugin
Extend LVM volume
git: Show changes in a code block
Find terms using git grep
git: Little known facts and commands
git through a proxy
Clone a specific git commit
git credentials from environment variables
Format for writing to InfluxDB
Move git repo
Windows netsh portproxy
Packet sniffing
Pin packages on Ubuntu
Encode string for URL using PowerShell
Import remote PowerShell module
Import some members of a PowerShell module
Show confirmation by default in PowerShell
Setting a proxy in PowerShell
PowerShell Gallery on Powershell Core
Incremental deployments using rancher-compose
Pull images on all Rancher hosts
Container registries
Testing a reverse proxy
Redirecting stdout and stderr separately
Sudoers
Add Linux swap space in a file
Testing container images
Ubuntu packages for troubleshooting

Making vim.tiny an alternative for vim

#Linux
update-alternatives --install "/usr/bin/vim" "vim" "/usr/bin/vim.tiny" 1
update-alternatives --set "vim" "/usr/bin/vim.tiny"

Artifactory NuGet feed for PowerShell modules

#Docker
  1. Create NuGet repo in Artifactory
  2. Install NuGet on build agent
  3. Register new repository:

     Register-PSRepository -Name Artifactory -SourceLocation http://10.12.12.157:8081/artifactory/api/nuget/dillen-nuget -PublishLocation http://10.12.12.157:8081/artifactory/api/nuget/dillen-nuget -InstallationPolicy Trusted
    
  4. Publish module:

     Publish-Module -Name Foobar -Repository Artifactory -NuGetApiKey 'dillen:XXX'
    

The NuGetApiKey is retrieved from Artifactory using the “Set Me Up” button of the repository.

Breaking out of containers

#Docker

Mounting /proc and /dev

nsenter

Custom format for docker ps

#Docker

docker ps --format "table {{.Names}}\\t{{.Image}}\\t{{.Status}}"
cat ~/.docker/config.json
#...
"psFormat":"table {{.ID}}\\t{{.Names}}\\t{{.Image}}\\t{{.Status}}"
#...

Populate the Docker build cache

#Docker

Build agents usually have not seen a previous of an image. Solve this by pulling the last version and referencing it:

docker build \
    --tag myimage:mytag \
    --cache-from myimage:myoldtag \
    .

When using micro-labeling, make sure to move them to the bottom to prevent cache misses.

Docker Compose network context

#Docker #Compose

Mounting the socket creates a new network context and creaks name resolution between containers.

Docker-in-Docker starts containers in the same network context.

Selecting an init process for containers

#Docker #init
  • systemd
  • supervisord
  • dumb-init
  • tini
  • s6

Docker lessons learned

#Docker

Binary images

  • Pattern: Only use library images from Docker Hub
  • Anti-pattern: Don’t use community images
  • Tags: image, run, library, community, docker, hub, pattern
  • Abstract: XXX
  • Example: XXX

Reuse

  • Pattern: Derive from code
  • Anti-pattern: Don’t build from community images
  • Tags: image, run, library, community, docker, hub, code, pattern
  • Abstract: XXX
  • Example: XXX

Binary images

  • Pattern: Build yourself
  • Anti-pattern: Don’t use community images
  • Tags: image, run, library, community, docker, hub, build, pattern
  • Abstract: XXX
  • Example: XXX

Using tags

  • Pattern: Use versioned tags
  • Anti-pattern: Don’t use latest
  • Tags: image, run, tag, docker, hub, pattern
  • Abstract: XXX
  • Example: XXX

Test images

  • Pattern: Test images before promoting them to production
  • Anti-pattern: Don’t use untested images
  • Tags: image, docker, test, pattern
  • Abstract: XXX
  • Example: XXX

Fail early

  • Pattern: XXX
  • Anti-pattern: XXX
  • Tags: image, test, docker, build, shell, pattern
  • Abstract: XXX
  • Example: SHELL ["bash", "-e", "-x", "-c"]

Readability beats size

  • Pattern: XXX
  • Anti-pattern: XXX
  • Tags: image, write, docker, layer, size, pattern
  • Abstract: XXX
  • Example: XXX

Copy directory tree into image

  • Pattern: Use a single ADD statement
  • Anti-pattern: Don’t ADD multiple times
  • Tags: image, write, docker, add, pattern
  • Abstract: XXX
  • Example: ADD files /

Order of statements

  • Pattern: Install dependencies first
  • Anti-pattern: XXX
  • Tags: image, write, dependency, docker, pattern
  • Abstract: XXX
  • Example: XXX

Validate downloads

  • Pattern: Validate downloads against checksums or signature
  • Anti-pattern: XXX
  • Tags: image, write, artifact, download, file, checksum, signature, docker, pattern
  • Abstract: XXX
  • Example: XXX

Run as user

  • Pattern: Run service as user
  • Anti-Pattern: Don’t run as root
  • Tags: image, write, root, user, docker, pattern
  • Abstract: XXX
  • Example: XXX

Use gosu

  • Pattern: Use gosu instead of sudo
  • Anti-pattern: Don’t use sudo
  • Tags: image, write, sudo, gosu, docker, pattern
  • Abstract: XXX
  • Example: XXX

Avoid privileges containers

  • Pattern: Use as few capabilities as possible
  • Anti-pattern: Avoid privileges containers
  • Tags: image, run, capability, privileged, docker, pattern
  • Abstract: Only add requires capabilities, isolate into privileged sidekick, drop capabilities as early as possible
  • Example: XXX

Manage your secrets

  • Pattern: Use secrets management
  • Anti-Pattern: Don’t store secrets in environment variables
  • Tags: image, write, run, secrets, credentials, password, token, environment, variable, docker, pattern
  • Abstract: XXX
  • Example: XXX

Use label to specify maintainer

  • Pattern: Use LABEL over MAINTAINER
  • Anti-pattern: Avoid MAINTAINER
  • Tags: image, write, label, maintainer, docker, pattern
  • Abstract: XXX
  • Example: XXX

Plan for PID 1

  • Pattern: XXX
  • Anti-pattern: XXX
  • Tags: image, write, init, pid, docker, pattern
  • Abstract: Use exec, use init
  • Example: XXX

Avoid Docker inception

  • Pattern: XXX
  • Anti-pattern: Avoid Docker-in-Docker (dind)
  • Tags: image, run, dind, inception, docker-in-docker, docker, pattern
  • Abstract: Don`t mount the Docker socker into a container, don’t run dind in privileges container
  • Example: XXX

Pull during build

  • Pattern: Always use --pull on docker build
  • Anti-pattern: Don’t use outdated upstream images
  • Tags: image, build, pull, upstream, docker, pattern
  • Abstract: XXX
  • Example: docker build --pull

Embrace automation

  • Pattern: Use build pipeline over multi-stage builds
  • Anti-pattern: Avoid multi-stage builds
  • Tags: image, write, build, multi-stage, pipeline, docker, pattern
  • Abstract: XXX
  • Example: XXX

Use microlabeling

  • Pattern: Use microlabeling
  • Anti-pattern: Don’t publish untracable images
  • Tags: image, write, build, microlabeling, label, docker, pattern
  • Abstract: label-schema.org and competitor
  • Example: XXX

Separate build and runtime information

  • Pattern: Separate build and runtime information
  • Anti-pattern: Don’t mix build and runtime information
  • Tags: image, write, build, runtime, docker, pattern
  • Abstract: For example credentials or proxy configuration
  • Example: XXX

Parameterize docker-compose.yml

  • Pattern: Use environment variables in docker-compose.yml
  • Anti-pattern: Don’t duplicate docker-compose.yml
  • Tags: image, run, compose, docker-compose, variable, docker, pattern
  • Abstract: XXX
  • Example: XXX

Read-only containers

  • Pattern: Make containers immutable
  • Anti-pattern: Don’t run writable containers
  • Tags: image, run, read-only, ro, immutable, tmpfs, docker, pattern
  • Abstract: Use --read-only to make the root fs read-only and --tmpfs to mount writable in-memory fs
  • Example: XXX

Plan resources

  • Pattern: Plan resource reservations and limits
  • Anti-pattern: XXX
  • Tags: image, run, resource, reservation, limit, docker, pattern
  • Abstract: XXX
  • Example: XXX

Tame java

  • Pattern: Manage memory used by Java
  • Anti-pattern: Don’t let Java decide how much memory to use
  • Tags: image, run, java, memory, docker, pattern
  • Abstract: XXX
  • Example: XXX

Test deployments

  • Pattern: Always do test deployments
  • Anti-pattern: Don’t release untested images
  • Tags: image, test, run, deployment, docker, pattern
  • Abstract: XXX
  • Example: XXX

Monitor containers

  • Pattern: Monitor containers
  • Anti-pattern: XXX
  • Tags: image, run, monitoring, docker, pattern
  • Abstract: XXX
  • Example: XXX

Scan your artifacts

  • Pattern: Scan your artifacts
  • Anti-pattern: XXX
  • Tags: image, build, run, security, docker, pattern
  • Abstract: XXX
  • Example: XXX

Docker locale

#Docker
apt install language-pack-en
update-locale LANG=en_US.UTF-8 LC_MESSAGES=POSIX
locale-gen

Docker and proxies

#Docker

Environment variables http_proxy and https_proxy must be set.

How to build behind a proxy:

docker build --env http_proxy --env https_proxy --env no_proxy --tag myimage:mytag .

How to run behind a proxy:

docker run --env http_proxy --env https_proxy --env no_proxy myimage:mytag

How to configure the daemon behind a proxy (only situation to specify the proxy):

$ mkdir -p /etc/systemd/system/docker.service.d
$ cat >> /etc/systemd/system/docker.service.d/proxy.conf <<EOF
[Service]
Environment="http_proxy=http://1.2.3.4:3128" "https_proxy=http://1.2.3.4:3128" "no_proxy=localhost"
EOF
$ systemctl daemon-reload
$ service docker restart

How to use docker-compose behind a proxy:

$ cat docker-compose.yml
version: '2'
services:
  example1:
    build:
      context: .
      args:
        - http_proxy
        - https_proxy
        - no_proxy
  example2:
    image: myimage:mytag
    environment
      - http_proxy
      - https_proxy
      - no_proxy

How to use docker-machine behind the proxy:

docker-machine --engine-env http_proxy --engine-env https_proxy --engine-env no_proxy ...

Docker lessons learned

#Docker

Add ther following variables to the service definition:

  • USER
  • SHELL
  • LANG

Docker, timezone and time sync

#Docker

Use timezone of host:

docker run -v /etc/localtime:/etc/localtime myimage:mytag

Use specific timezone:

docker run -v /usr/share/zoneinfo/Europe/Berlin:/etc/localtime myimage:mytag

Containerized time sync:

cat ../docker-ntp/Dockerfile
FROM alpine:3.7

ENV TIME_SERVER=pool.ntp.org

RUN apk update \
 && apk add openntpd gettext

ADD files /
ENTRYPOINT /entrypoint.sh
cat ../docker-ntp/files/entrypoint.sh
#!/bin/sh

cat /etc/ntpd.conf.envsubst | envsubst > /etc/ntpd.conf
exec ntpd -d
cat ../docker-ntp/files/etc/ntpd.conf.envsubst
servers ${TIME_SERVER}
sensor *
constraints from "https://www.google.com"
$ docker build --tag myimage:mytag
$ docker run --cap-add SYS_TIME --cap-add SYS_NICE myimage:mytag

Docker user namespace mapping

#Docker

https://docs.docker.com/engine/security/userns-remap/#disable-namespace-remapping-for-a-container

AutoSize for all tables in PowerShell

#PowerShell
$PSDefaultParameterValues.Add('Format-Table:AutoSize', {if ($host.Name -eq "ConsoleHost"){$true}})

Example Docker storage plugin

#Docker #Storage #Plugin

XXX

Extend LVM volume

#Docker

How to extend a LVM volume:

  1. Rescan if disk not found:

     echo 1 >/sys/class/block/sda/device/rescan
    
  2. Create primary partition sda2 of type 8e (Linux LVM):

     fdisk /dev/sda
    
  3. Create physical volume:

     pvcreate /dev/sda2
    
  4. Note name of existing volume group:

     vgdisplay
    
  5. Extend volume group

     vgextend ubuntu-vg /dev/sda2
    
  6. Scan for physical volumes

     pvscan
    
  7. Note name of logical volume

     lvdisplay
    
  8. Extend logical volume

     lvextend /dev/ubuntu-vg/root /dev/sda2
    
  9. Resize filesystem

    a. extN:

     resize2fs /dev/ubuntu-vg/root
    

    b. xfs:

     xfs_growfs /dev/centos/root
    

git: Show changes in a code block

#git

XXX go

git log -L :initInstallCmd:cmd/docker-setup/install.go

XXX

git log -L '/assertMetadataIsLoaded/',+10:cmd/docker-setup/install.go

Find terms using git grep

#git

XXX

git grep extractVersion

git: Little known facts and commands

#git

Show history with diff:

git log -p

Set current branch to SHA:

git reset --hard SHA

Copy all files from tag TAG:

git checkout tags/TAG '*'

Interactive rebase from very first commit:

git rebase --interactive --root

Create orphan branch

git checkout --orphan ghpages

git through a proxy

#git #proxy

The proxy can be set on different levels (--local, --global or --system):

git config --global http[s].proxy 'http://proxy.mydomain.com:3128'

Clone a specific git commit

#git
git clone $URL
cd $PROJECT_NAME
git reset --hard $SHA1

git credentials from environment variables

#SSH #git

For HTTP(S) URLs use a custom credential helper (source):

git config --global credential.helper '!f() { sleep 1; echo "username=${GIT_USER}\npassword=${GIT_PASS}"; }; f'
GIT_USER=user GIT_PASS=pass git clone https://git-rd.haufe.io/...

For SSH based repos use a custom SSH command:

GIT_SSH_COMMAND='echo "${SSH_KEY}" | ssh-add -t 20 -; ssh' git clone git@github.com:nicholasdille/test-ssh.git

Format for writing to InfluxDB

#influxdb

<Measurement>[,<tag>=<value>] <field>=<value>[,<field>=<value>] [<timestamp>}

Move git repo

#git

Move a complete repo from oldserver to newserver:

git clone --mirror git@oldserver:oldproject.git
cd oldproject.git
git remote add new git@newserver:newproject.git
git push --mirror new

Windows netsh portproxy

#Windows #portproxy

netsh can be used to configure port forwarding:

netsh interface portproxy add v4tov4 listenport=443 connectaddress=10.0.0.21 connectport=443
netsh interface portproxy delete v4tov4 listenport=443

Packet sniffing

#Networking

Sniffing packets and displaying HTTP requests and responses:

tcpdump -A -s 0 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

Pin packages on Ubuntu

#Ubuntu
cat /etc/apt/preferences.d/docker-ce.pref
Package: docker-ce
Pin: version 17.09.*
Pin-Priority: 1000

Encode string for URL using PowerShell

#powershell

Some characters cannot be used in URLs without causing weird behaviour on servers and or clients:

[System.Web.HttpUtility]::UrlEncode('&')

Import remote PowerShell module

#PowerShell
  1. Create a session to the remote host:

     $Session = New-PSSession -Computer 'RemoteHost'
    
  2. Import the module in the remote session:

     Invoke-Command -Session $Session -ScriptBlock {
         Import-Module -Name 'MyModule'
     }
    
  3. Import remote session:

     $RemoteModule = Import-PSSession -Session $Session -Module 'MyModule'
    
  4. Load module from remote session:

     Import-Module -Name $RemoteModule -Global
    

Import some members of a PowerShell module

#PowerShell
Import-Module -Name Helpers -Function 'Get-Epoch'

Show confirmation by default in PowerShell

#PowerShell
$PSDefaultParameterValues.Add("*:Confirm",$True)

Setting a proxy in PowerShell

#PowerShell
$Proxy = 'http://proxy.mydomain.com:3128'
$PSDefaultParameterValues.Add('Register-PSRepository:Proxy') = $Proxy
$PSDefaultParameterValues.Add('Set-PSRepository:Proxy') = $Proxy
$PSDefaultParameterValues.Add('Install-Module:Proxy') = $Proxy
$PSDefaultParameterValues.Add('Invoke-WebRequest:Proxy') = $Proxy
$PSDefaultParameterValues.Add('Invoke-RestMethod:Proxy') = $Proxy

PowerShell Gallery on Powershell Core

#PowerShell

PowerShell Gallery is not available as a package repository on PowerShell Core. It is registered by running the following:

Register-PSRepository -Default
Set-PSRepository -Name PSGallery -InstallationPolicy Trusted

Incremental deployments using rancher-compose

#Rancher #Docker

Rancher deploys incrementally when docker-compose.yml is missing an existing service

Pull images on all Rancher hosts

#Rancher #Docker

Using rancher-compose pull on a docker-compose.yml forces all hosts to pull the specified images reducing deployment times

Container registries

#Container

Explore manifests and blob contents: https://explore.ggcr.dev/

Short-lived storage: https://ttl.sh

Testing a reverse proxy

#Docker

Reverse proxies usually service a long list of different services. Some of which are secured using HTTPS which others are still HTTP-only.

  1. Testing certificate:
  echo | openssl s_client -showcerts -servername name.to.test -connect reverse.proxy.name:443 | openssl x509 -inform PEM -noout -text
  1. Testing HTTP:
  curl -vH "Host: name.to.test" http://reverse.proxy.name
  1. Testing HTTPS:
  curl --resolve name.to.test:443:reverse.proxy.name https://name.to.test/v2/image/tags/list

Redirecting stdout and stderr separately

#bash

Process standard input and standard error separately:

(echo stdout && echo stderr >&2) > >(cat | while read -r LINE; do echo "stdout: ${LINE}"; done) 2> >(cat | while read -r LINE; do echo "stderr: ${LI
NE}" >&2; done)

Sudoers

#Linux
$ sudo cat /etc/sudoers.d/myuser
# allow myuser to execute all commands without a password
myuser  ALL=(ALL) NOPASSWD: ALL
# add the following line if executed command will not provide a tty
Defaults:myuser !requiretty
# add the following if using a proxy
Defaults:myuser env_keep+="http_proxy https_proxy no_proxy"

Add Linux swap space in a file

#Linux

If a separate swap partition was not configured:

dd if=/dev/zero of=/myswap bs=1M count=4096
chmod 0600 /myswap
mkswap /myswap
swapon /myswap

Insert the following line in /etc/fstab for swap from the next boot:

/myswap    none    swap    sw    0   0

Testing container images

#Docker

https://robots.thoughtbot.com/tdd-your-dockerfiles-with-rspec-and-serverspec https://blog.jevsejev.io/2016/06/09/docker-image-tests/ http://www.infrabricks.de/blog/2014/09/10/docker-container-mit-serverspec-testen/

Ubuntu packages for troubleshooting

#Docker #Ubuntu
  • ca-certificates
  • apt-transport-https
  • curl
  • wget
  • software-properties-common for add-apt-repository
  • net-tools for netstat
  • dnsutils for nslookup, dig
  • vim.tiny followed by

      update-alternatives --install "/usr/bin/vim" "vim" "/usr/bin/vim.tiny" 1
      update-alternatives --set "vim" "/usr/bin/vim.tiny"
    
  • iproute2