Kubernetes Best Practices for Production Deployments

On this page

Running applications in production on Kubernetes is challenging, but following proven best practices can make the difference between a reliable system and a maintenance nightmare. This guide covers the essential strategies that successful teams use to deploy and maintain production workloads.

What You’ll Learn

By the end of this guide, you’ll understand:

How to design resilient deployments that handle failures gracefully
Security practices that protect your applications from common threats
Monitoring strategies that give you visibility into your system’s health
Scaling patterns that keep your applications performant under load
Backup and recovery procedures that protect your data

1. Resource Management: The Foundation of Stability

Why Resource Management Matters

Resource management is the cornerstone of stable Kubernetes deployments. Without proper resource allocation, your applications can experience:

Resource starvation when one pod consumes all available CPU/memory
OOM kills when containers exceed memory limits
Poor performance due to CPU throttling
Unpredictable scaling behavior

Best Practice: Always Set Resource Limits

apiVersion: apps/v1
kind: Deployment
metadata:
  name: production-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: production-app
  template:
    metadata:
      labels:
        app: production-app
    spec:
      containers:
      - name: app
        image: production-app:v1.2.0
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 3

Key Points:

Requests: Guaranteed resources your pod will receive
Limits: Maximum resources your pod can use
Health Checks: Essential for Kubernetes to know when pods are healthy
Realistic Values: Base limits on actual usage patterns, not guesses

Pro Tip: Use Resource Quotas

apiVersion: v1
kind: ResourceQuota
metadata:
  name: production-quota
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi
    pods: "20"

2. Security: Protect Your Applications

The Security-First Approach

Security in Kubernetes requires careful attention. The principle of least privilege should guide every security decision.

Best Practice: Run as Non-Root

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-app
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 2000
      containers:
      - name: app
        image: secure-app:latest
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop:
            - ALL
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: varlog
          mountPath: /var/log
        - name: app-config
          mountPath: /app/config
          readOnly: true
      volumes:
      - name: tmp
        emptyDir: {}
      - name: varlog
        emptyDir: {}
      - name: app-config
        configMap:
          name: app-config

Security Benefits:

Non-root execution prevents privilege escalation attacks
Read-only filesystem prevents malicious file modifications
Dropped capabilities remove unnecessary privileges
ConfigMap mounting keeps configuration separate and secure

Pro Tip: Use Network Policies

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: app-network-policy
spec:
  podSelector:
    matchLabels:
      app: production-app
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: frontend
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: database
    ports:
    - protocol: TCP
      port: 5432

3. Monitoring and Observability: Know Your System

The Three Pillars of Observability

Metrics: Quantitative data about your system’s performance
Logs: Detailed records of events and errors
Traces: Request flow through your distributed system

Best Practice: Comprehensive Monitoring Setup

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s

    rule_files:
      - "alert_rules.yml"

    scrape_configs:
    - job_name: 'kubernetes-pods'
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__

Essential Metrics to Monitor

# Example alerting rules
groups:
- name: kubernetes.rules
  rules:
  - alert: HighCPUUsage
    expr: container_cpu_usage_seconds_total{container!=""} > 0.8
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High CPU usage detected"
      description: "Container {{ $labels.container }} is using {{ $value }} CPU"

  - alert: HighMemoryUsage
    expr: container_memory_usage_bytes{container!=""} / container_spec_memory_limit_bytes{container!=""} > 0.85
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High memory usage detected"
      description: "Container {{ $labels.container }} is using {{ $value | humanizePercentage }} memory"

4. Scaling Strategies: Handle Traffic Spikes

Horizontal vs Vertical Scaling

Horizontal scaling (adding more pods) is generally preferred in Kubernetes because it’s more resilient and can handle traffic spikes better.

Best Practice: Implement HPA with Multiple Metrics

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: production-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: production-app
  minReplicas: 3
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  - type: Object
    object:
      metric:
        name: requests-per-second
      describedObject:
        apiVersion: networking.k8s.io/v1
        kind: Ingress
        name: production-app-ingress
      target:
        type: Value
        value: 1000
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 10
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15

Key Features:

Multiple metrics: CPU, memory, and custom metrics
Stabilization windows: Prevent rapid scaling oscillations
Conservative scale-down: Avoid scaling down too aggressively
Aggressive scale-up: Respond quickly to traffic spikes

Pro Tip: Use VPA for Vertical Scaling

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: production-app-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: production-app
  updatePolicy:
    updateMode: "Off"  # Use "Auto" for automatic updates
  resourcePolicy:
    containerPolicies:
    - containerName: '*'
      minAllowed:
        cpu: 100m
        memory: 50Mi
      maxAllowed:
        cpu: 1
        memory: 500Mi
      controlledValues: RequestsAndLimits

5. Backup and Disaster Recovery: Protect Your Data

The 3-2-1 Backup Rule

3 copies of your data
2 different storage types
1 off-site backup

Best Practice: Automated Backup Strategy

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: production-daily-backup
spec:
  schedule: "0 2 * * *"  # Daily at 2 AM
  template:
    includedNamespaces:
    - production
    includedResources:
    - deployments
    - services
    - configmaps
    - secrets
    - persistentvolumeclaims
    - persistentvolumes
    storageLocation: production-backup
    volumeSnapshotLocations:
    - production-snapshot
    ttl: 720h  # Keep backups for 30 days

Backup Verification Strategy

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: backup-verification
spec:
  schedule: "0 4 * * 0"  # Weekly on Sunday at 4 AM
  template:
    includedNamespaces:
    - backup-test
    includedResources:
    - deployments
    - services
    - configmaps
    - secrets
    - persistentvolumeclaims
    storageLocation: production-backup
    volumeSnapshotLocations:
    - production-snapshot
    ttl: 24h

6. Deployment Strategies: Zero-Downtime Updates

Rolling Updates with Health Checks

apiVersion: apps/v1
kind: Deployment
metadata:
  name: production-app
spec:
  replicas: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    spec:
      containers:
      - name: app
        image: production-app:v1.2.0
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 3
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3

Benefits:

Zero downtime: New pods are ready before old ones are terminated
Rollback capability: Easy to revert to previous version
Health verification: Only healthy pods serve traffic

Key Takeaways

Immediate Actions You Can Take

Set resource limits on all your containers today
Implement health checks for every application
Run containers as non-root users
Set up basic monitoring with Prometheus
Configure HPA for your critical applications
Implement automated backups with Velero

Long-term Strategy

Gradually implement security policies
Build comprehensive monitoring dashboards
Test disaster recovery procedures regularly
Optimize resource usage based on monitoring data
Automate everything possible

Most importantly, keep it simple! Overcomplicating your infrasturcutre will result in unimaginable growing pains.

Remember: Production Kubernetes is a journey, not a destination. Start with these fundamentals and continuously improve based on your specific needs and challenges.

Need help implementing these best practices? Join us on Slack