Automating k8s cluster security with kyverno polices for governance and compliance

In a microservices architecture, teams work within dedicated namespaces to ensure service isolation. The DevOps engineer is responsible for creating CI/CD pipelines to deploy these services into the Kubernetes cluster. While the deployment and development of microservices are managed by the teams, the DevOps engineer also oversees cluster management and administrative tasks, including scaling, governance, compliance, high availability, security, cost optimization, day-to-day operations, and observability.

In k8s, we have an admission controller which implement governance and security.

Problem: writing admission controllers need expertise in GO and also if organization compliance rules keeps growing or modifying with time, its hard for managing and creation of admission controllers and Day to Day operations.

The Solution: Kyverno - A Dynamic Admission Controller

Kyverno provides a solution as a dynamic admission controller that simplifies the enforcement of governance and security policies without the need for custom Go code. It allows organizations to define policies in YAML files and automatically apply them within the Kubernetes cluster.

How Kyverno Works

Once deployed in the Kubernetes cluster, Kyverno functions as a dynamic admission controller. It reads the policy rules defined in the YAML files and validates whether the incoming resource creation or modification complies with these rules. The policies are flexible, making it easy to define and enforce governance and compliance requirements across the cluster.

Examples:

Pod Resource Requests and Limits: Every pod created must have resource requests and limits set.
Namespace Resource Quotas and Limit Ranges: Each namespace must have Resource Quotas and Limit Ranges defined.
Deny All Traffic by Default: When a new namespace is created, it will automatically have all traffic denied by default.
Eviction Policy for Host Path and EmptyDir Volumes: Ensure that no pod uses HostPath or EmptyDir volumes unless specified.
Mandatory Pod Labels: Every pod must include specific labels for tracking and organization.
Health Probes: Ensure that every pod includes Liveness (LP), Readiness (RP), and Startup Probes (SP) to manage lifecycle and availability.

Steps for Implementing Kyverno

Ensure the cluster is set up (Kind for local development). Then, deploy Kyverno to the cluster using a manifest or Helm chart
```
 kubectl create -f https://github.com/kyverno/kyverno/releases/download/v1.8.5/install.yaml
```
All resources have been deployed in the kyverno Namespace.

Create a policy YAML to enforce resource requests and limits (RR/RL) on pods

 apiVersion: kyverno.io/v1
 kind: ClusterPolicy
 metadata:
   name: require-requests-limits
   annotations:
     policies.kyverno.io/title: Require Limits and Requests
     policies.kyverno.io/category: Best Practices, EKS Best Practices
     policies.kyverno.io/severity: medium
     policies.kyverno.io/subject: Pod
     policies.kyverno.io/minversion: 1.6.0
     policies.kyverno.io/description: >-
       As application workloads share cluster resources, it is important to limit resources
       requested and consumed by each Pod. It is recommended to require resource requests and
       limits per Pod, especially for memory and CPU. If a Namespace level request or limit is specified,
       defaults will automatically be applied to each Pod based on the LimitRange configuration.
       This policy validates that all containers have something specified for memory and CPU
       requests and memory limits.
 spec:
   validationFailureAction: audit
   background: true
   rules:
   - name: validate-resources
     match:
       any:
       - resources:
           kinds:
           - Pod
     validate:
       message: "CPU and memory resource requests and memory limits are required for containers."
       pattern:
         spec:
           containers:
           - resources:
               requests:
                 memory: "?*"
                 cpu: "?*"
               limits:
                 memory: "?*"
           =(initContainers):
           - resources:
               requests:
                 memory: "?*"
                 cpu: "?*"
               limits:
                 memory: "?*"
           =(ephemeralContainers):
           - resources:
               requests:
                 memory: "?*"
                 cpu: "?*"
               limits:
                 memory: "?*"

Apply the clusterpolicy and verify the status
Verify the logs of the kyverno pods to identify that kyverno pod recognizes the cluster policy deployed
Now, Try creating a Nginx pod and identify the events

Although we created a ClusterPolicy, If the policy is set to Audit, non-compliant pods will still be created, but violations will be logged:

Kubernetes Events (resource-level logs)
Policy Reports (compliance overview)

If the policy is set to Enforce, and try creating a new pod and observe the request will be blocked, and a message will indicate why the pod creation was denied.
Now, try deleting the policy and deploy redis again, now it will be deployed as no policy to enforce

Use the generate action to implement namespace creation with predefined Resource Quotas and Limit Ranges

 apiVersion: kyverno.io/v1
 kind: ClusterPolicy
 metadata:
   name: add-ns-quota
   annotations:
     policies.kyverno.io/title: Add Quota
     policies.kyverno.io/category: Multi-Tenancy, EKS Best Practices
     policies.kyverno.io/subject: ResourceQuota, LimitRange
     policies.kyverno.io/minversion: 1.6.0
     policies.kyverno.io/description: >-
       To better control the number of resources that can be created in a given
       Namespace and provide default resource consumption limits for Pods,
       ResourceQuota and LimitRange resources are recommended.
       This policy will generate ResourceQuota and LimitRange resources when
       a new Namespace is created.
 spec:
   rules:
   - name: generate-resourcequota
     match:
       any:
       - resources:
           kinds:
           - Namespace
     generate:
       apiVersion: v1
       kind: ResourceQuota
       name: default-resourcequota
       synchronize: true
       namespace: "{{request.object.metadata.name}}"
       data:
         spec:
           hard:
             requests.cpu: '4'
             requests.memory: '16Gi'
             limits.cpu: '4'
             limits.memory: '16Gi'
   - name: generate-limitrange
     match:
       any:
       - resources:
           kinds:
           - Namespace
     generate:
       apiVersion: v1
       kind: LimitRange
       name: default-limitrange
       synchronize: true
       namespace: "{{request.object.metadata.name}}"
       data:
         spec:
           limits:
           - default:
               cpu: 500m
               memory: 1Gi
             defaultRequest:
               cpu: 200m
               memory: 256Mi
             type: Container

ResourceQuota(RQ) defines the total resources (CPU, memory, storage, etc.) allocated to a namespace as a whole.

LimitRanges(LR)specifies default resource requests and limits for individual containers running within the namespace.

This above picture also shows that when a namespace is created, the policy will automatically enforce default Resource Quotas and Limit Ranges.