Ensure that Audit Log Errors Emit Alerts

Description

OpenShift audit works at the API server level, logging all requests coming to the server. However, if API server instance is unable to write errors, an alert must be issued in order for the organization to take a relevant action. e.g. shutting down that instance.

Kubernetes by default has metrics that enable one to write such alerts:

• apiserver_audit_event_total
• apiserver_audit_error_total

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: audit-errors
  namespace: openshift-kube-apiserver
spec:
  groups:
  - name: apiserver-audit
    rules:
    - alert: AuditLogError
      annotations:
        summary: |-
          An API Server instance was unable to write audit logs. This could be
          triggered by the node running out of space, or a malicious actor
          tampering with the audit logs.
        description: An API Server had an error writing to an audit log.
      expr: |
        sum by (apiserver,instance)(rate(apiserver_audit_error_total{apiserver=~".+-apiserver"}[5m])) / sum by (apiserver,instance) (rate(apiserver_audit_event_total{apiserver=~".+-apiserver"}[5m])) > 0
      for: 1m
      labels:
        severity: warning

For more information, consult the official Kubernetes documentation.

Rationale

When there are errors writing audit logs, security events will not be logged by that specific API Server instance. Security Incident Response teams use these audit logs, amongst other artifacts, to determine the impact of security breaches or events. Without these logs, it becomes very difficult to assess a situation and do appropriate root cause analysis in such incidents.