Back to Blog Kubernetes Security

Kubernetes Security Best Practices: Securing Your Clusters in Production

15 min read

Kubernetes has become the de facto standard for container orchestration, but its flexibility comes with security complexity. A misconfigured cluster can expose your entire infrastructure to attackers. This guide covers the essential security practices for running Kubernetes in production.

The Kubernetes Attack Surface

Before implementing security controls, understand what you're protecting:

  • API Server: The gateway to your cluster - if compromised, attackers control everything
  • etcd: Stores all cluster state including secrets - a prime target
  • Kubelet: Runs on every node and can execute arbitrary containers
  • Container Runtime: A container escape gives access to the host
  • Network: By default, all pods can communicate with each other

1. Cluster Hardening

Secure the API Server

  • Never expose the API server to the public internet
  • Use private endpoints (EKS private cluster, GKE private cluster)
  • Enable audit logging for all API requests
  • Restrict API access using authorized networks

Protect etcd

etcd contains all cluster secrets and configuration:

  • Enable encryption at rest for etcd data
  • Use TLS for etcd peer and client communication
  • Restrict network access to etcd (control plane only)
  • Regular backups with encryption

Enable Audit Logging

# audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Log all requests to secrets
  - level: Metadata
    resources:
    - group: ""
      resources: ["secrets"]

  # Log pod exec/attach/portforward
  - level: Request
    resources:
    - group: ""
      resources: ["pods/exec", "pods/attach", "pods/portforward"]

  # Log all other requests at metadata level
  - level: Metadata

2. Authentication and Authorization

Use OIDC for User Authentication

Never use static tokens or basic auth. Integrate with your identity provider:

  • Configure OIDC authentication with your IdP (Okta, Azure AD, Google)
  • Use short-lived tokens with automatic refresh
  • Implement MFA at the IdP level

Implement RBAC Properly

Follow least privilege principles with Role-Based Access Control:

# Namespace-scoped role for developers
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: developer
  namespace: my-app
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "services", "configmaps"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch"]
---
# Bind role to group from OIDC
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: developer-binding
  namespace: my-app
subjects:
- kind: Group
  name: "developers"  # From OIDC groups claim
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: developer
  apiGroup: rbac.authorization.k8s.io

Avoid Dangerous RBAC Permissions

These permissions can lead to privilege escalation:

  • * on any resource (wildcard verbs)
  • create on pods, deployments (can run arbitrary code)
  • create on roles/rolebindings (can grant themselves more permissions)
  • exec on pods (shell access to containers)
  • get on secrets (access to credentials)

3. Pod Security

Use Pod Security Standards

Kubernetes 1.25+ uses Pod Security Admission to enforce security standards:

# Enforce restricted standard on namespace
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Never Run as Root

apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    fsGroup: 1000
  containers:
  - name: app
    image: my-app:latest
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop:
          - ALL

Use Read-Only Root Filesystem

Prevent attackers from writing malicious files:

securityContext:
  readOnlyRootFilesystem: true
volumeMounts:
- name: tmp
  mountPath: /tmp
- name: cache
  mountPath: /app/cache
volumes:
- name: tmp
  emptyDir: {}
- name: cache
  emptyDir: {}

Drop All Capabilities

Linux capabilities grant specific privileges. Drop all and add only what's needed:

securityContext:
  capabilities:
    drop:
      - ALL
    add:
      - NET_BIND_SERVICE  # Only if binding to ports < 1024

4. Network Security

Implement Network Policies

By default, all pods can communicate. Use NetworkPolicies to restrict traffic:

# Default deny all ingress traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
---
# Allow frontend to talk to backend
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

Use a Service Mesh for mTLS

Service meshes like Istio or Linkerd provide automatic mTLS between pods:

  • Encrypt all pod-to-pod traffic
  • Strong identity for each workload
  • Fine-grained authorization policies
  • Traffic observability

5. Secrets Management

Don't Use Kubernetes Secrets Alone

Kubernetes secrets are base64 encoded, not encrypted. Enhance security with:

Enable Encryption at Rest

# EncryptionConfiguration
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: 
      - identity: {}

Use External Secrets Operators

Sync secrets from external providers like AWS Secrets Manager, HashiCorp Vault, or Azure Key Vault:

# ExternalSecret syncs from AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: db-credentials
  data:
  - secretKey: password
    remoteRef:
      key: production/database
      property: password

Use Workload Identity

Avoid storing cloud credentials in secrets. Use workload identity to let pods assume cloud IAM roles:

  • EKS: IAM Roles for Service Accounts (IRSA)
  • GKE: Workload Identity
  • AKS: Azure AD Workload Identity

6. Image Security

Scan Images for Vulnerabilities

Integrate scanning into your CI/CD pipeline:

# Example: Trivy scan in CI
trivy image --severity HIGH,CRITICAL \
  --exit-code 1 \
  my-registry/my-app:latest

Use Signed Images

Implement image signing and verification with Sigstore/Cosign:

# Sign image
cosign sign --key cosign.key my-registry/my-app:latest

# Verify in admission controller
# Kyverno policy example
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signature
spec:
  validationFailureAction: enforce
  rules:
  - name: verify-signature
    match:
      resources:
        kinds:
        - Pod
    verifyImages:
    - image: "my-registry/*"
      key: |-
        -----BEGIN PUBLIC KEY-----
        ...
        -----END PUBLIC KEY-----

Use Minimal Base Images

Reduce attack surface with distroless or scratch images:

# Distroless image
FROM gcr.io/distroless/static-debian11
COPY --from=builder /app/binary /app/binary
ENTRYPOINT ["/app/binary"]

7. Runtime Security

Deploy Runtime Protection

Use tools like Falco to detect suspicious runtime behavior:

# Falco rule: Detect shell in container
- rule: Shell Spawned in Container
  desc: Detect shell being spawned in a container
  condition: >
    spawned_process and container and
    shell_procs and not shell_allowed
  output: >
    Shell spawned in container
    (user=%user.name container=%container.name
    shell=%proc.name parent=%proc.pname)
  priority: WARNING

Common Runtime Detections

  • Shell spawned in container
  • Sensitive file access (/etc/shadow, /etc/passwd)
  • Network tools executed (curl, wget, nc)
  • Crypto mining processes
  • Container escape attempts

8. Supply Chain Security

Pin Image Digests

Don't use mutable tags like latest. Pin to immutable digests:

# Bad: Tag can be overwritten
image: nginx:1.21

# Good: Immutable digest
image: nginx@sha256:abc123...

Use Private Registries

  • Mirror public images to your private registry
  • Scan images before allowing them in your registry
  • Implement registry access controls

Security Checklist

Quick reference for Kubernetes security hardening:

  • API server not exposed publicly
  • RBAC enabled with least privilege
  • Pod Security Standards enforced
  • Network policies implemented (default deny)
  • Secrets encrypted at rest
  • External secrets manager integrated
  • Workload identity configured (no static cloud credentials)
  • Image scanning in CI/CD
  • Image signing and verification
  • Runtime protection deployed
  • Audit logging enabled
  • Containers running as non-root
  • Read-only root filesystem where possible
  • All capabilities dropped

Next Steps

Kubernetes security requires continuous attention. Start with the fundamentals - RBAC, pod security, and network policies - then layer on advanced protections like runtime security and supply chain controls.

Need help securing your Kubernetes clusters? Contact us for expert guidance on implementing these best practices.

Need Help Securing Your Kubernetes Clusters?

We help teams implement Kubernetes security best practices and build secure container platforms.

Schedule a Consultation