PCI-DSS v3.2.1 Control Baseline for Red Hat OpenShift Container Platform 4

$ oc create sa service_account_name
       

$ oc -n openshift-kube-apiserver get configmap config -o json | jq -r '.data."config.yaml"' | jq '.apiServerArguments."enable-admission-plugins"'

$ oc -n openshift-kube-apiserver get configmap config -o json | jq -r '.data."config.yaml"' | jq '.apiServerArguments."enable-admission-plugins"'

$ oc describe clusterrolebindings

$ oc get --as="system:anonymous" --raw='/readyz?verbose'

$ oc get --as="system:anonymous" projects

"apiServerArguments":{
  ...
  "audit-log-maxbackup": [10],
  ...

"apiServerArguments":{
  ...
  "audit-log-maxsize": ["100"],
  ...

"apiServerArguments":{
  ...
  "audit-log-path":"/var/log/kube-apiserver/audit.log",
  ...

"apiServerArguments":{
  ...
  "basic-auth-file":[
    "/path/to/any/file"
  ],
  ...

"apiServerArguments": {
  ...
    "client-ca-file": [
      "/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt"
    ],
  ...

When you enable etcd encryption, the following OpenShift API server and Kubernetes API server resources are encrypted:

• Secrets
• ConfigMaps
• Routes
• OAuth access tokens
• OAuth authorize tokens

When you enable etcd encryption, encryption keys are created. These keys are rotated on a weekly basis. You must have these keys in order to restore from an etcd backup.

To ensure the correct cipher, set the encryption type to aescbc or aesgcm in the apiserver object which configures the API server itself.

spec:
  encryption:
    type: aescbc

For more information, follow the relevant documentation.

"apiServerArguments": {
  ...
    "etcd-cafile": [
        "/etc/kubernetes/static-pod-resources/configmaps/etcd-serving-ca/ca-bundle.crt"
    ],
  ...

...
"etcd-certfile": [
    "/etc/kubernetes/static-pod-resources/secrets/etcd-client/tls.crt"
],
...

...
"etcd-keyfile": [
    "/etc/kubernetes/static-pod-resources/secrets/etcd-client/tls.key"
],
...

"apiServerArguments":{
  ...
  "insecure-bind-address":[
    "127.0.0.1"
  ],
  ...

"apiServerArguments":{
  ...
  "kubelet-certificate-authority":"/etc/kubernetes/static-pod-resources/configmaps/kubelet-serving-ca/ca-bundle.crt",
  ...

"apiServerArguments":{
...
  "kubelet-client-certificate":"/etc/kubernetes/static-pod-resources/secrets/kubelet-client/tls.crt",
...
}

"apiServerArguments":{
...
  "kubelet-client-certificate":"/etc/kubernetes/static-pod-resources/secrets/kubelet-client/tls.crt",
...
}

Note that this particular rule is only valid for OCP releases up to and
including 4.8

"apiServerArguments":{
...
  "kubelet-client-key":"/etc/kubernetes/static-pod-resources/secrets/kubelet-client/tls.key",
...
}

"apiServerArguments":{
...
  "kubelet-client-key":"/etc/kubernetes/static-pod-resources/secrets/kubelet-client/tls.key",
...
}

Note that this particular rule is only valid for OCP releases up to and
including 4.8

"apiServerArguments":{
  ...
  "min-request-timeout":[
    
  ],
  ...

...
"serviceAccountPublicKeyFiles":[
  "/etc/kubernetes/static-pod-resources/configmaps/sa-token-signing-certs"
],
...

"apiServerArguments":{
...
"tls-cert-file": [
  "/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt"
],
...
}

"servingInfo":{
  ...
  "cipherSuites": [
    "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
    "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
    "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
    "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
    "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",
    "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256",
  ],
  ...

"apiServerArguments":{
...
"tls-private-key-file": [
  "/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key"
],
...
}

The configuration tlsSecurityProfile specifies TLS configurations to be used while establishing connections with the externally exposed servers. Though secure transport mode is used for establishing connections, the protocols used may not always be strong enough to avoid interception and manipulation of the data in transport. TLS Security profile configured should not make use of any protocols, ciphers, and algorithms with known security vulnerabilities.

tlsSecurityProfile can be configured to use one of custom, intermediate, modern, or old profile. Profile Old should be avoided at all times and when using custom profile one should be extremely careful as invalid configurations can be catastrophic. It is always advised to use highly secure intermediate or modern profiles and if unset a default is chosen.

Update tlsSecurityProfile to Intermediate using the following command:

oc patch apiservers.config.openshift.io cluster --type 'json' --patch '[{"op": "add", "path": "/spec/tlsSecurityProfile/intermediate", "value": {}}, {"op": "replace", "path": "/spec/tlsSecurityProfile/type", "value": "Intermediate"}'

For more information, follow OpenShift documentation: the relevant documentation.

"apiServerArguments":{
  ...
  "token-auth-file":[
    "/path/to/any/file"
  ],
  ...

ClusterLogForwarders

For users to interact with OpenShift Container Platform, they must first authenticate to the cluster. The authentication layer identifies the user associated with requests to the OpenShift Container Platform API. The authorization layer then uses information about the requesting user to determine if the request is allowed. Understanding authentication | Authentication | OpenShift Container Platform

The OpenShift Container Platform includes a built-in OAuth server for token-based authentication. Developers and administrators obtain OAuth access tokens to authenticate themselves to the API. It is recommended for an administrator to configure OAuth to specify an identity provider after the cluster is installed. User access to the cluster is managed through the identity provider. Understanding identity provider configuration | Authentication | OpenShift Container Platform

OpenShift includes built-in role based access control (RBAC) to determine whether a user is allowed to perform a given action within the cluster. Roles can have cluster scope or local (i.e. project) scope. Using RBAC to define and apply permissions | Authentication | OpenShift Container Platform

You can configure OAuth tokens to expire after a set period of inactivity. By default, no token inactivity timeout is set.

The inactivity timeout can be either set in the OAuth server configuration or in any of the OAuth clients. The client settings override the OAuth server setting.

To set the OAuth server inactivity timeout, edit the OAuth server object: oc edit oauth cluster and set the .spec.tokenConfig.accessTokenInactivityTimeout parameter to the desired value:

apiVersion: config.openshift.io/v1
kind: OAuth
metadata:
...
spec:
   tokenConfig:
     accessTokenInactivityTimeout: 10m0s 
Please note that the OAuth server converts the value internally to a human-readable format,
so that e.g. setting accessTokenInactivityTimeout=600s would be converted by the OAuth
server to accessTokenInactivityTimeout=10m0s.

        

To edit the OAuth client inactivity timeout, edit the OAuth client object: oc edit oauthclient $clientname and set the top-level accessTokenInactivityTimeoutSeconds attribute.

apiVersion: oauth.openshift.io/v1
grantMethod: auto
kind: OAuthClient
metadata:
...
accessTokenInactivityTimeoutSeconds: 600 

For users to interact with OpenShift Container Platform, they must first authenticate to the cluster. The authentication layer identifies the user associated with requests to the OpenShift Container Platform API. The authorization layer then uses information about the requesting user to determine if the request is allowed. Understanding authentication | Authentication | OpenShift Container Platform

The OpenShift Container Platform includes a built-in OAuth server for token-based authentication. Developers and administrators obtain OAuth access tokens to authenticate themselves to the API. It is recommended for an administrator to configure OAuth to specify an identity provider after the cluster is installed. User access to the cluster is managed through the identity provider. Understanding identity provider configuration | Authentication | OpenShift Container Platform

However, not all Identity Providers supported by OpenShift provide the same level of capabilities. As an example, the htpasswd Identity Provider only checks the username and password match and provides no means of 2FA, account lockout or notification mechanism. This rule therefore only allows a subset of identity providers.

For users to interact with OpenShift Container Platform, they must first authenticate to the cluster. The authentication layer identifies the user associated with requests to the OpenShift Container Platform API. The authorization layer then uses information about the requesting user to determine if the request is allowed. Understanding authentication | Authentication | OpenShift Container Platform

The OpenShift Container Platform includes a built-in OAuth server for token-based authentication. Developers and administrators obtain OAuth access tokens to authenticate themselves to the API. It is recommended for an administrator to configure OAuth to specify an identity provider after the cluster is installed. User access to the cluster is managed through the identity provider. Understanding identity provider configuration | Authentication | OpenShift Container Platform

If the identity provider is LDAP, setting the insecure flag to true would mean that passwords, such as the one used to authenticate the OAuth proxy to the LDAP server would be transmitted in the clear, potentially allowing an attacker to read the password if they captured the network traffic.

"extendedArguments": {
...
  "port": ["0"],
...

"extendedArguments": {
...
...

"extendedArguments": {
...
  "secure-port": ["10257"],
...

"extendedArguments": {
...
  "root-ca-file": [
    "/etc/kubernetes/static-pod-resources/configmaps/serviceaccount-ca/ca-bundle.crt"
  ],
...

"extendedArguments": {
...
  "service-account-private-key-file": [
    "/etc/kubernetes/static-pod-resources/secrets/service-account-private-key/service-account.key"
  ],
...

"extendedArguments": {
...
  "use-service-account-credentials": [
    "true"
  ],
...

$ oc get cm/etcd-pod -n openshift-etcd -o yaml

--cert-file=/etc/kubernetes/static-pod-certs/secrets/etcd-all-[a-z]+/etcd-serving-NODE_NAME.crt

[a-z]+

oc get -nopenshift-etcd cm etcd-pod -oyaml | grep "\-\-client-cert-auth="

oc get -nopenshift-etcd cm etcd-pod -oyaml | grep "\-\-key-file=/etc/kubernetes/static-pod-certs/secrets/etcd-all-[a-z]+/etcd-serving-NODE_NAME.key"

[a-z]+

$ oc get cm/etcd-pod -n openshift-etcd -o yaml

--peer-cert-file=/etc/kubernetes/static-pod-certs/secrets/etcd-all-[a-z]+/etcd-peer-NODE_NAME.crt

[a-z]+

oc get -nopenshift-etcd cm etcd-pod -oyaml | grep "\-\-peer-client-cert-auth="

oc get -nopenshift-etcd cm etcd-pod -oyaml | grep "\-\-peer-key-file=/etc/kubernetes/static-pod-certs/secrets/etcd-all-[a-z]+/etcd-peer-NODE_NAME.key"

[a-z]+

tlsCertFile: /path/to/TLS/cert.key

tlsPrivateKeyFile: /path/to/TLS/private.key

"apiServerArguments":{
  ...
  "kubelet-read-only-port":[
    "0"
  ],
  ...

OpenShift audit works at the API server level, logging all requests coming to the server. However, if API server instance is unable to write errors, an alert must be issued in order for the organization to take a relevant action. e.g. shutting down that instance.

Kubernetes by default has metrics that enable one to write such alerts:

• apiserver_audit_event_total
• apiserver_audit_error_total

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: audit-errors
  namespace: openshift-kube-apiserver
spec:
  groups:
  - name: apiserver-audit
    rules:
    - alert: AuditLogError
      annotations:
        summary: |-
          An API Server instance was unable to write audit logs. This could be
          triggered by the node running out of space, or a malicious actor
          tampering with the audit logs.
        description: An API Server had an error writing to an audit log.
      expr: |
        sum by (apiserver,instance)(rate(apiserver_audit_error_total{apiserver=~".+-apiserver"}[5m])) / sum by (apiserver,instance) (rate(apiserver_audit_event_total{apiserver=~".+-apiserver"}[5m])) > 0
      for: 1m
      labels:
        severity: warning

For more information, consult the official Kubernetes documentation.

OpenShift can audit the details of requests made to the API server through the standard Kubernetes audit capabilities.

In OpenShift, auditing of the API Server is on by default. Audit provides a security-relevant chronological set of records documenting the sequence of activities that have affected system by individual users, administrators, or other components of the system. Audit works at the API server level, logging all requests coming to the server. Each audit log contains two entries:

The request line containing:

• A Unique ID allowing to match the response line (see #2)
• The source IP of the request
• The HTTP method being invoked
• The original user invoking the operation
• The impersonated user for the operation (self meaning himself)
• The impersonated group for the operation (lookup meaning user's group)
• The namespace of the request or none
• The URI as requested

The response line containing:

• The aforementioned unique ID
• The response code

For more information on how to configure the audit profile, please visit the documentation

The configuration tlsSecurityProfile specifies TLS configurations to be used while establishing connections with the externally exposed servers. Though secure transport mode is used for establishing connections, the protocols used may not always be strong enough to avoid interception and manipulation of the data in transport. TLS Security profile configured should not make use of any protocols, ciphers, and algorithms with known security vulnerabilities.

tlsSecurityProfile can be configured to use one of custom, intermediate, modern, or old profile. Profile Old should be avoided at all times and when using custom profile one should be extremely careful as invalid configurations can be catastrophic. It is always advised to use highly secure intermediate or modern profiles and if unset profile configured in apiservers.config.openshift.io/cluster resource will be used as default.

To update tlsSecurityProfile to Intermediate use the following command:

oc patch -n openshift-ingress-operator ingresscontrollers.operator.openshift.io default --type 'json' --patch '[{"op": "add", "path": "/spec/tlsSecurityProfile/intermediate", "value": {}}, {"op": "replace", "path": "/spec/tlsSecurityProfile/type", "value": "Intermediate"}'

For more information, follow OpenShift documentation: the relevant documentation.

"apiServerArguments":{
  ...
  "audit-log-maxbackup": [10],
  ...

"apiServerArguments":{
  ...
  "audit-log-maxsize": ["100"],
  ...

"apiServerArguments":{
  ...
  "audit-log-path":"/var/log/openshift-apiserver/audit.log",
  ...

RBAC is a critical feature in terms of security for Kubernetes and OpenShift. It enables administrators to segment the privileges granted to a service account, and thus allows us to limit the access to resources that they get. By defining cluster roles appropriately one is able to codify organizational policy. [1]

[1] https://docs.openshift.com/container-platform/latest/authentication/using-rbac.html

RBAC is a critical feature in terms of security for Kubernetes and OpenShift. It enables administrators to segment the privileges granted to a service account, and thus allows us to limit the access to resources that they get. By defining roles appropriately one is able to codify organizational policy. [1]

[1] https://docs.openshift.com/container-platform/latest/authentication/using-rbac.html

oc patch

/tmp/allowed-registries-patch.yaml

spec:
  registrySources:
    allowedRegistries:
    - my-trusted-registry.internal.example.com

oc patch image.config.openshift.io cluster --patch="$(cat /tmp/allowed-registries-patch.yaml)" --type=merge

oc patch

/tmp/allowed-import-registries-patch.yaml

spec:
  allowedRegistriesForImport:
  - domainName: my-trusted-registry.internal.example.com
    insecure: false

oc patch image.config.openshift.io cluster --patch="$(cat /tmp/allowed-import-registries-patch.yaml)" --type=merge

oc get image.config.openshift.io/cluster -o json | jq '.spec | (.allowedRegistriesForImport[])? | select(.insecure==true)'

oc edit image.config.openshift.io/cluster

oc patch image.config.openshift.io cluster --type=json -p "[{'op': 'remove', 'path': '/spec/registrySources/insecureRegistries'}]"

Using the Red Hat Quay Container Security Operator, you can access vulnerability scan results from the OpenShift Container Platform web console for container images used in active pods on the cluster. The Red Hat Quay Container Security Operator:

• Watches containers associated with pods on all or specified namespaces
• Queries the container registry where the containers came from for vulnerability information, provided an image’s registry is running image scanning (such as Quay.io or a Red Hat Quay registry with Clair scanning)
• Exposes vulnerabilities via the ImageManifestVuln object in the Kubernetes API

For more information on the Container Security Operator, follow the OpenShift documentation: https://docs.openshift.com/container-platform/latest/security/pod-vulnerability-scan.html

Containers should not enable more capabilites than needed as this opens the door for malicious use. To enable only the required capabilities, the appropriate Security Context Constraints (SCCs) should set capabilities as a list in allowedCapabilities.

In case an SCC outside the default allow list in the variable var-sccs-with-allowed-capabilities-regex is being flagged, create a TailoredProfile and add the additional SCC to the regular expression in the variable var-sccs-with-allowed-capabilities-regex. An example allowing an SCC named additional follows:

apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
  name: cis-additional-scc
spec:
  description: Allows an additional scc
  setValues:
  - name: ocp4-var-sccs-with-allowed-capabilities-regex
    rationale: Allow our own custom SCC
    value: ^privileged$|^hostnetwork-v2$|^restricted-v2$|^nonroot-v2$|^additional$
  extends: ocp4-cis
  title: Modified CIS allowing one more SCC

Finally, reference this TailoredProfile in a ScanSettingBinding For more information on Tailoring the Compliance Operator, please consult the OpenShift documentation: https://docs.openshift.com/container-platform/latest/security/compliance_operator/co-scans/compliance-operator-tailor.html

 oc get -nopenshift-sdn ds sdn -ojson | jq -r '.spec.template.spec.containers[] | select(.name == "sdn")'

 oc get -nopenshift-sdn ds sdn -ojson | jq -r '.spec.template.spec.containers[] | select(.name == "sdn")'

{
"configMap": {
  "defaultMode": 420,
  "name": "sdn-config"
  },
"name": "config"
}