We had the great opportunity to listen for two days to some really great informative and inspirational talks. We learned a lot, met some really nice people and got some new insights into this whole cloud native thing ;-) The list of things to checkout and review after this conference alone is two pages long. In the next couple of posts we’ll present you some of our key takeaways of the conference talks we attended. All the presentations can be found on our internal Bitbucket repository.

One of my keytakeaways is that the industry is moving at an incredible pace. Even though I consider myself knowledgable about the current trends and tools, I’ve heard so much new and helpful stuff, that my impression is, that we have to spent a lot more time on conferences if we want to stay relevant in promoting state of the art software development and deployment to our customers. From my point of view customers demand excellence, state of the art technology and a partner who can provide some further insights which go beyond the normal stackoverflow answers. If we want to deliver on that expectation, we have to continuously learn and be passionate about the technologies we advocate. Share our experiences with others and in return learn from theirs, and where is a better place than on a conference with likeminded people.

#conferencesarecoming

Keynote - Holly Cummins “Having fun at work” #IBMGarage

Maslows Hirachy of needs

  1. Self Actualization
  2. . Esteem
  3. Love and belonging
  4. Safety needs
  5. Physiological Needs

Why is there no fun

Mangagement Model - Command and Control Puritan understanding of fun…

  • “Pleasure in the job puts perfection in the work” Aristotle
  • “You have to take time to play to be creative”.
  • “Your brain at positive is 31% greater productivity”.

Internet trolls just want to have fun.

Find the unfun things and get rid of them!

  1. Stop prohibiting fun!
  2. Let people make their own decisions
  3. Pair programming => prevents code reviews and is more collaberating
  4. Status self reporting
  5. #noestimates
  6. TTD is fun, gamified quality
  7. Automating stuff is fun => SRE is about removing toil
  8. devops helps making our live humane
  9. Releases should be boring

Gamification

31,1% dislike teambuilding activities

Add fun to work

  1. Get rid of unfun things
  2. Put fun things on top of it

Failing todo the second without the first.

“The Container Universe” - Nicolas Dille

Relevance of Docker

  • Docker Swarm lost to Kubernetes
  • containerd is growing quickly, docker is loosing relevance
  • Image build (e.g. buildah, img, kaniko)

Docker Buildkit

  • Docker build is the old buildengine
  • docker daemon delivers buildkit => enables multistage builds
  • parallel builds
  • Remote build cache!
  • Build mounts
  • Build secrets!!
  • SSH forwarding
  • Working rootless implementation

Build Cache

Image builds can take some time

How it works:

  • Images are made of layers defimed by…
    • Upstream layer
    • Comman executed
Buildkit Cache warming
  • Needs docker 19.03 Pulls the image cache from the remote registry, improves build speed How it works:
export DOCKER_BUILDKIT=1 # enables BUILDKIT instead of Docker default engine
docker build --build-arg BUILDKIT_INLINE_CACHE=1
Build Secrets
  • Buildkit can mount secrets using tmpfs
  • Needs to be told on the docker cli and the Dockerfile
  • Dockerfile: #syntax=docker/dockerfile:1.0.0-experimental
export DOCKER_BUILDKIT=1 # enables BUILDKIT instead of Docker default engine
docker build --secret id=,ysite.key,src=mysite.key --progress=plain
RUN --mount=type=secret,id=mysite.key # only available in this stage
Mount SSH socket into Dockerfile
ssh-add -l
# Forward into build
export DOCKER_BUILDKIT=1 # enables BUILDKIT instead of Docker default engine
docker build --ssh default --progress=plain
# Dockerfile
RUN --mount=type=ssh printenv  grep ssh && ssh-add -l

Docker CLI Plugins

like kubectl, helm…

  • Extend docker with new Subcommands via shell scripts
  • Located in '/.docker/cli...
  • “Docker Client Plugin Manager” Lukas Lach clip
    • docker clip add lukaslach/clips:dive
    • Docker dive ncurse Oberfläche

Troubleshooiting from scratch

Modern containers only have a binary, copy roots into conatainer

docker run -d --name traefik traefik
docker create alpine
docker cp alpine:/ roots
# Copy into the the container
cd rootfs && docker cp . traefik:/ && cd ..

Troubleshooting Networking

Containers are based on 3 Kernel features

  • chroot
  • cgroups
  • namespaces
docker run -d --name nginx nginx
docker run -it --rm --net container:nginx --pid container:nginx alpine # Connect to the same process namespace!! looks like a

Lukas Lach building registry tools

docker run -it --rm cmd.cat/curl/netstat

Kubernetes introduces ephemeral containers

Dependencies

  • Security issues will be discovered
  • Scanner Clair, jFrog XRay

Trivy, simply container scanning

made by Aqua security https://github.com/aquasecurity/trivy upselling

Multi-Arch Image, docker buildx

Able to build images for different platforms, using Kernel Quemu

docker run mplatform/mquery hello-world

export DOCKER_CLI_EXPIREMENTAL=enabled
docker buildx # cli Plugin
docker buildx imagetools inspect image:tag

Docker Context

  • Docker machine without the deployments
  • Define new context onto another VM
  • Enables connect to another docker daemon on another host

Cloud native storage

ORAS - OCI Registry as storage, store any data in a registry

oras push registry:test artifacts.txt
oras pull registry_test --output out
cat out/artifacts.txt

Windows subsystem for linux v2

  • Desing
    • WSL v2 runs a linux kernel maintained by microsoft in a tiny VM, docker for linux in WSL!
    • Root file system is stored in vhdx image

Rootless Docker

Reduce attack surface It is not…

  • Running as non-root in container
  • enabling
  • docker vpnkit

systemd in user context to run a docker daemon in userspace

Image building

docker run --rm -it \
    --name img \
    --volume $(pwd):/home/user/src:ro \ # for the build context and dockerfile, can be read-only since we won't modify it
    --workdir /home/user/src \ # set the builder working directory
    --volume "${HOME}/.docker:/root/.docker:ro" \ # for credentials to push to docker hub or a registry
    --security-opt seccomp=unconfined --security-opt apparmor=unconfined \ # required by runc
    r.j3ss.co/img build -t useclear

nsenter breakout and change into another namespace, nsenter -t 1 -m -u -n -i sh Link https://github.com/jpetazzo/nsenter

Cloud native applicateion bundle CNAB

standard packaing format for multi-component distributed applications, installer for docker applications

  • easly create bundles with Duffle
  • Containerized install script
  • Most accept parameter install/uninstall/upgrade/downgrade/status
  • Porter interface with Helm and Terraform

K3s lightweight K8s

K3s is meant to be run on several machines start local K3s server

K3D: K3s in docker

k3d is mady by rancher running a cluster on docker

K8s: Do not do Bare-Metal

Bootstrapping is hard

  • use a project like kubespray
  • “We don’t have to know everything”
  • Clusternetzwerk in eine DMZ erweitern

Folien

https://dille.name/slides/2019-11-13/ContainerConf%202019%20-%20Container%20Universum.html#/5

“Richtige Integrationstests mit Containern”- Philipp Krenn

  • https://testcontainers.org Testcontainers is a Java library that supports JUnit tests, providing lightweight, throwaway instances of common databases, Selenium web browsers, or anything else that can run in a Docker container. Funktioniert mit Python…
  • Docker for Mac API: socat https://hub.docker.com/r/alpine/socat/
  • ryuk (Death Note), drops a death note, and cleans up my docker containers

Test containers in container

Wormhole Pattern

docker run -it --rm -v $PWD:$PWD -w $PWD -v /var/run/docker.sock:/var/run/docker.sock maven:3 mvn --projects parent,4_testcontainers test

“Werkzeuge zum bauen von Docker Images” - Patrick Harbäck

OCI - Open Container Initiative

Build a container Image from the ground up

Security: Docker build

  • Docker needs elevated priviliges
  • Build pipelines / developers can access Docker

Only trusted users should be use your docker daemon.

Demo: Host access va privileged container

Needs priviliges to promote prviliges to build environment

docker run -it --privileged alpine ls /dev
docker run -it  alpine ls /dev
# Mount host root fs into docker
docker run -it --privileged alpine /bin/sh mount

Caching Build Pipeline kubernetes + Jenkins

Issues:

  • one docker images does not scale for parallel build
  • no distributed caching

Flexibility

  • Docker build tools based on Dockerfile

Tools

Buildkit - docker buildx

Kaniko - Kubernetes

  • Designed for Kubernetes
  • Compatible with AppArmor and SELinux
  • Focus on security and performance
  • Reproducible builds
  • Beherrscht fast komplette Dockerfile syntax

Tools

  • Makisu from Uber
  • buildah
  • Jib
  • Bazel
  • source-to-image (openshift)
  • CNF Buildack - buildpack cli

Should we use docker

Decide based on the base of your projects if you really need Dockerfile

“Infrastructure as Code with Pulumi, better than all the others (like Ansible)?” - Jonas Hecht

Tools Comparison

Tools shouldn’t suck! (no master, no agent)

Howto they handle Configuration Drift

  • Mutable Infrastructure
  • Immutable Infrastructure

Difference does not matter if we disallow configuration changes by hand! https://github.com/jonashackt/ansible-pulumi

  • using pytest and testinfra

Procedural vs Declarative

100x Jenkins und kein bisschen müde - Frederic Gurr

Presentation of the eclipse foundation how they use Jenkins. They moved away from Jenkins Core to Jenkins OSS. Due to cost reasons.

Infrastructure Common Build Infrastructure

  • Best Practices
  • Continuous integration build farm
    • Jenknis Instanzen
  • Nexus OSS Server
  • Sonarqube
  • Gerrit/GitHub

Jiro - Eigenentwicklung

https://github.com/eclipse/eclipse-cbi/jiro

  • Jenkins Instance Running on Openshift
  • Jenkins Configuration as Code (JCasC) Plugin
  • Templates (Jsonnet)
  • Shell scripts

“Tragedy of the commons!”

  • max 2 parallele Builds
  • 2 vCPU, 8GB RAM
  • einnbinden externer BuildAgents

Jenkins Pipeline

Monitoring

ToDos further Notes

  • Jsonnet als Templating !?
  • Watch Office Space
  • Kubernetes the Hardway
  • Fail whale
  • Docker nsenter
  • “Docker Client Plugin Manager” Lukas Lach clip
    • docker clip add lukaslach/clips:dive
    • Docker dive ncurse Oberfläche
  • aquasecurity: https://github.com/aquasecurity/kube-hunter
    • Pod, This option lets you discover what running a malicious container can do/discover on your cluster. This gives a perspective on what an attacker could do if they were able to compromise a pod, perhaps through a software vulnerability. This may reveal significantly more vulnerabilities. The job.yaml file defines a Job that will run kube-hunter in a pod, using default Kubernetes pod access settings.
    • Run the job with kubectl create -f ./job.yaml
    • Find the pod name with kubectl describe job kube-hunter
    • View the test results with kubectl logs <pod name>
  • Checkout kube-bench from Aquasecurity https://github.com/aquasecurity/kube-bench
  • Testcontainer für Selenium IDE Tests
  • Hype driven development https://blog.daftcode.pl/hype-driven-development-3469fc2e9b22
  • Kubecon in Amsterdam 450$, 30.3.-2.4.2019 https://events19.linuxfoundation.org/events/kubecon-cloudnativecon-europe-2020/schedule-at-a-glance/
  • Stackshare.io https://stackshare.io
  • Python Install packages: pipenv install
  • Howto implement playing around with software!? => Outcome, there has to be some form of outcome…
  • Change ansible callbacks https://docs.ansible.com/ansible/latest/plugins/callback.html
  • Cloudbees Core einführen
  • Jenkins Operator
  • “Release Engineers pro Team”
  • Testinfra Python testframework for infrastructure, using for deployment tests, https://testinfra.readthedocs.io/en/latest/modules.html using with testcontainers
  • img, based on buildkit img build --tag my:tag . building without daemon https://github.com/genuinetools/img same cli as docker