DISA STIG for Red Hat OpenShift Container Platform 4 - Node level

The OpenShift Platform allows for verifying the signature of a container image before pulling it. this is done via the policy.json file [1] which needs to be configured via the Machine Config Operator [2].

Ensure that the default policy is "reject" in /etc/containers/policy.json, which should look as follows:

  {
    "default": [{"type": "reject"}],
    "transports":
    ...
  }

[1] https://github.com/containers/image/blob/main/docs/containers-policy.json.5.md

[2] https://docs.openshift.com/container-platform/latest/security/container_security/security-container-signature.html

All logs must be owned by root user and group and have permissions 755. By default, the path for the Kubernetes pod logs is

/var/log/pods/

If any of the files have incorrect permissions or ownerhship, change the permissions and ownership of files located under "/var/log/pods" to protect from unauthorized access. 1. Execute the following to set the output of pods readable only by the owner: for node in $(oc get node -oname); do oc debug $node -- chroot /host /bin/bash -c 'echo -n "$HOSTNAME "; find /var/log/pods/ -type f \( -perm /022 -o -perm /044 \) | xargs -r chmod 600' 2>/dev/null; done 2. Execute the following to set the group and group-ownership to root for files that store the output of pods: for node in $(oc get node -oname); do oc debug $node -- chroot /host /bin/bash -c 'echo -n "$HOSTNAME "; find /var/log/pods/ -type f \! -user 0 | xargs -r chown root:root' 2>/dev/null; done

authentication:
  ...
  anonymous:
    enabled: false
  ...

authorization:
  mode: Webhook
  ...

authentication:
...
  x509:
    clientCAFile: /etc/kubernetes/kubelet-ca.crt
...

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
   name: kubelet-config-$pool
spec:
    machineConfigPoolSelector:
        matchLabels:
            pools.operator.machineconfiguration.openshift.io/$pool_name: ""
    kubeletConfig:
        eventRecordQPS: 

  apiVersion: machineconfiguration.openshift.io/v1
  kind: KubeletConfig
  metadata:
     name: kubelet-config-$pool
  spec:
      machineConfigPoolSelector:
          matchLabels:
              pools.operator.machineconfiguration.openshift.io/$pool_name: ""
      kubeletConfig:
        tlsCipherSuites:
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
  

The configuration tlsSecurityProfile specifies TLS configurations to be used while establishing connections with the externally exposed servers. Though secure transport mode is used for establishing connections, the protocols used may not always be strong enough to avoid interception and manipulation of the data in transport. TLS Security profile configured should not make use of any protocols, ciphers, and algorithms with known security vulnerabilities.

tlsSecurityProfile can be configured to use one of custom, intermediate, modern, or old profile. Profile Old should be avoided at all times and when using custom profile one should be extremely careful as invalid configurations can be catastrophic. It is always advised to configure minimum TLS version to TLSv1.2 or latest when using Custom profile or to use predefined profiles Intermediate or modern. If a TLS security profile is not configured, the default TLS security profile is Intermediate.

To configure Custom tlsSecurityProfile for the Kubelet with TLSv1.2 as minimum TLS version, create a new or modify existing KubeletConfig object along these lines, one for every MachineConfigPool:

  apiVersion: machineconfiguration.openshift.io/v1
  kind: KubeletConfig
  metadata:
      name: kubelet-tls-config-$pool
  spec:
      tlsSecurityProfile:
          type: Custom
          custom:
              ciphers:
              - ECDHE-ECDSA-CHACHA20-POLY1305
              - ECDHE-RSA-CHACHA20-POLY1305
              - ECDHE-RSA-AES128-GCM-SHA256
              - ECDHE-ECDSA-AES128-GCM-SHA256
              minTLSVersion: 
      machineConfigPoolSelector:
          matchLabels:
              pools.operator.machineconfiguration.openshift.io/$pool_name: ""
  

For more information, follow OpenShift documentation: the relevant documentation.

...
rotateCertificates: true
...

featureGates:
...
  RotateKubeletClientCertificate: true
...

makeIPTablesUtilChains: true

Protect tuned kernel parameters from being overwritten by the kubelet.

Before enabling this kernel parameter, it's important and necessary to first create a MachineConfig object that persist the required sysctl's. The required sysctl's are the following:

kernel.keys.root_maxbytes=25000000
kernel.keys.root_maxkeys=1000000
kernel.panic=10
kernel.panic_on_oops=1
vm.overcommit_memory=1
vm.panic_on_oom=0

The these need to be enabled via MachineConfig since they need to be available as soon as the node starts and before the Kubelet does. The manifest may look as follows:

---
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 75-master-kubelet-sysctls
spec:
  config:
    ignition:
      version: 3.1.0
    storage:
      files:
      - contents:
          source: data:,vm.overcommit_memory%3D1%0Avm.panic_on_oom%3D0%0Akernel.panic%3D10%0Akernel.panic_on_oops%3D1%0Akernel.keys.root_maxkeys%3D1000000%0Akernel.keys.root_maxbytes%3D25000000%0A
        mode: 0644
        path: /etc/sysctl.d/90-kubelet.conf
        overwrite: true

This will need to be done for each relevant MachineConfigPool in the cluster.

After enabling this and after the changes have successfully rolled out to the whole cluster, it will now be possible to set the protectKernelDefaults parameter.

To configure, follow the directions in the documentation

Setup required tuned kernel parameters before enabling overwritten protection. Note that depending on the Linux distribution and its version that your cluster nodes are running, these parameters might be already set up for you. Please refer to the rule instructions for a check.

Before enabling kernel parameter overwritten protection default, it's important to check if these values are already set to the required values. If not, it is necessary to first create a MachineConfig object that persist the required sysctl's. The required sysctl's are the following:

kernel.keys.root_maxbytes=25000000
kernel.keys.root_maxkeys=1000000
kernel.panic=10
kernel.panic_on_oops=1
vm.overcommit_memory=1
vm.panic_on_oom=0

The these need to be enabled via MachineConfig since they need to be available as soon as the node starts and before the Kubelet does. The manifest may look as follows:

---
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 75-master-kubelet-sysctls
spec:
  config:
    ignition:
      version: 3.1.0
    storage:
      files:
      - contents:
          source: data:,vm.overcommit_memory%3D1%0Avm.panic_on_oom%3D0%0Akernel.panic%3D10%0Akernel.panic_on_oops%3D1%0Akernel.keys.root_maxkeys%3D1000000%0Akernel.keys.root_maxbytes%3D25000000%0A
        mode: 0644
        path: /etc/sysctl.d/90-kubelet.conf
        overwrite: true

This will need to be done for each relevant MachineConfigPool in the cluster.

To configure, follow the directions in the documentation

Setup required tuned kernel parameters before enabling overwritten protection.

Before enabling kernel parameter overwritten protection default, it's important and necessary to first create a MachineConfig object that persist the required sysctl's. The required sysctl's are the following:

kernel.keys.root_maxbytes=25000000
kernel.keys.root_maxkeys=1000000
kernel.panic=10
kernel.panic_on_oops=1
vm.overcommit_memory=1
vm.panic_on_oom=0

The these need to be enabled via MachineConfig since they need to be available as soon as the node starts and before the Kubelet does. The manifest may look as follows:

---
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 75-master-kubelet-sysctls
spec:
  config:
    ignition:
      version: 3.1.0
    storage:
      files:
      - contents:
          source: data:,vm.overcommit_memory%3D1%0Avm.panic_on_oom%3D0%0Akernel.panic%3D10%0Akernel.panic_on_oops%3D1%0Akernel.keys.root_maxkeys%3D1000000%0Akernel.keys.root_maxbytes%3D25000000%0A
        mode: 0644
        path: /etc/sysctl.d/90-kubelet.conf
        overwrite: true

This will need to be done for each relevant MachineConfigPool in the cluster.

To configure, follow the directions in the documentation

Setup required tuned kernel parameters before enabling overwritten protection. Note that depending on the Linux distribution and its version that your cluster nodes are running, these parameters might be already set up for you. Please refer to the rule instructions for a check.

Before enabling kernel parameter overwritten protection default, it's important to check if these values are already set to the required values. If not, it is necessary to first create a MachineConfig object that persist the required sysctl's. The required sysctl's are the following:

kernel.keys.root_maxbytes=25000000
kernel.keys.root_maxkeys=1000000
kernel.panic=10
kernel.panic_on_oops=1
vm.overcommit_memory=1
vm.panic_on_oom=0

The these need to be enabled via MachineConfig since they need to be available as soon as the node starts and before the Kubelet does. The manifest may look as follows:

---
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 75-master-kubelet-sysctls
spec:
  config:
    ignition:
      version: 3.1.0
    storage:
      files:
      - contents:
          source: data:,vm.overcommit_memory%3D1%0Avm.panic_on_oom%3D0%0Akernel.panic%3D10%0Akernel.panic_on_oops%3D1%0Akernel.keys.root_maxkeys%3D1000000%0Akernel.keys.root_maxbytes%3D25000000%0A
        mode: 0644
        path: /etc/sysctl.d/90-kubelet.conf
        overwrite: true

This will need to be done for each relevant MachineConfigPool in the cluster.

To configure, follow the directions in the documentation

Setup required tuned kernel parameters before enabling overwritten protection. Note that depending on the Linux distribution and its version that your cluster nodes are running, these parameters might be already set up for you. Please refer to the rule instructions for a check.

Before enabling kernel parameter overwritten protection default, it's important to check if these values are already set to the required values. If not, it is necessary to first create a MachineConfig object that persist the required sysctl's. The required sysctl's are the following:

kernel.keys.root_maxbytes=25000000
kernel.keys.root_maxkeys=1000000
kernel.panic=10
kernel.panic_on_oops=1
vm.overcommit_memory=1
vm.panic_on_oom=0

The these need to be enabled via MachineConfig since they need to be available as soon as the node starts and before the Kubelet does. The manifest may look as follows:

---
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 75-master-kubelet-sysctls
spec:
  config:
    ignition:
      version: 3.1.0
    storage:
      files:
      - contents:
          source: data:,vm.overcommit_memory%3D1%0Avm.panic_on_oom%3D0%0Akernel.panic%3D10%0Akernel.panic_on_oops%3D1%0Akernel.keys.root_maxkeys%3D1000000%0Akernel.keys.root_maxbytes%3D25000000%0A
        mode: 0644
        path: /etc/sysctl.d/90-kubelet.conf
        overwrite: true

This will need to be done for each relevant MachineConfigPool in the cluster.

To configure, follow the directions in the documentation

Setup required tuned kernel parameters before enabling overwritten protection. Note that depending on the Linux distribution and its version that your cluster nodes are running, these parameters might be already set up for you. Please refer to the rule instructions for a check.

Before enabling kernel parameter overwritten protection default, it's important to check if these values are already set to the required values. If not, it is necessary to first create a MachineConfig object that persist the required sysctl's. The required sysctl's are the following:

kernel.keys.root_maxbytes=25000000
kernel.keys.root_maxkeys=1000000
kernel.panic=10
kernel.panic_on_oops=1
vm.overcommit_memory=1
vm.panic_on_oom=0

The these need to be enabled via MachineConfig since they need to be available as soon as the node starts and before the Kubelet does. The manifest may look as follows:

---
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 75-master-kubelet-sysctls
spec:
  config:
    ignition:
      version: 3.1.0
    storage:
      files:
      - contents:
          source: data:,vm.overcommit_memory%3D1%0Avm.panic_on_oom%3D0%0Akernel.panic%3D10%0Akernel.panic_on_oops%3D1%0Akernel.keys.root_maxkeys%3D1000000%0Akernel.keys.root_maxbytes%3D25000000%0A
        mode: 0644
        path: /etc/sysctl.d/90-kubelet.conf
        overwrite: true

This will need to be done for each relevant MachineConfigPool in the cluster.

To configure, follow the directions in the documentation

Setup required tuned kernel parameters before enabling overwritten protection. Note that depending on the Linux distribution and its version that your cluster nodes are running, these parameters might be already set up for you. Please refer to the rule instructions for a check.

Before enabling kernel parameter overwritten protection default, it's important to check if these values are already set to the required values. If not, it is necessary to first create a MachineConfig object that persist the required sysctl's. The required sysctl's are the following:

kernel.keys.root_maxbytes=25000000
kernel.keys.root_maxkeys=1000000
kernel.panic=10
kernel.panic_on_oops=1
vm.overcommit_memory=1
vm.panic_on_oom=0

The these need to be enabled via MachineConfig since they need to be available as soon as the node starts and before the Kubelet does. The manifest may look as follows:

---
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 75-master-kubelet-sysctls
spec:
  config:
    ignition:
      version: 3.1.0
    storage:
      files:
      - contents:
          source: data:,vm.overcommit_memory%3D1%0Avm.panic_on_oom%3D0%0Akernel.panic%3D10%0Akernel.panic_on_oops%3D1%0Akernel.keys.root_maxkeys%3D1000000%0Akernel.keys.root_maxbytes%3D25000000%0A
        mode: 0644
        path: /etc/sysctl.d/90-kubelet.conf
        overwrite: true

This will need to be done for each relevant MachineConfigPool in the cluster.

To configure, follow the directions in the documentation

Setup required tuned kernel parameters before enabling overwritten protection. Note that depending on the Linux distribution and its version that your cluster nodes are running, these parameters might be already set up for you. Please refer to the rule instructions for a check.

Before enabling kernel parameter overwritten protection default, it's important to check if these values are already set to the required values. If not, it is necessary to first create a MachineConfig object that persist the required sysctl's. The required sysctl's are the following:

kernel.keys.root_maxbytes=25000000
kernel.keys.root_maxkeys=1000000
kernel.panic=10
kernel.panic_on_oops=1
vm.overcommit_memory=1
vm.panic_on_oom=0

The these need to be enabled via MachineConfig since they need to be available as soon as the node starts and before the Kubelet does. The manifest may look as follows:

---
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 75-master-kubelet-sysctls
spec:
  config:
    ignition:
      version: 3.1.0
    storage:
      files:
      - contents:
          source: data:,vm.overcommit_memory%3D1%0Avm.panic_on_oom%3D0%0Akernel.panic%3D10%0Akernel.panic_on_oops%3D1%0Akernel.keys.root_maxkeys%3D1000000%0Akernel.keys.root_maxbytes%3D25000000%0A
        mode: 0644
        path: /etc/sysctl.d/90-kubelet.conf
        overwrite: true

This will need to be done for each relevant MachineConfigPool in the cluster.

To configure, follow the directions in the documentation

Setup required tuned kernel parameters before enabling overwritten protection. Note that depending on the Linux distribution and its version that your cluster nodes are running, these parameters might be already set up for you. Please refer to the rule instructions for a check.

Before enabling kernel parameter overwritten protection default, it's important to check if these values are already set to the required values. If not, it is necessary to first create a MachineConfig object that persist the required sysctl's. The required sysctl's are the following:

kernel.keys.root_maxbytes=25000000
kernel.keys.root_maxkeys=1000000
kernel.panic=10
kernel.panic_on_oops=1
vm.overcommit_memory=1
vm.panic_on_oom=0

The these need to be enabled via MachineConfig since they need to be available as soon as the node starts and before the Kubelet does. The manifest may look as follows:

---
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 75-master-kubelet-sysctls
spec:
  config:
    ignition:
      version: 3.1.0
    storage:
      files:
      - contents:
          source: data:,vm.overcommit_memory%3D1%0Avm.panic_on_oom%3D0%0Akernel.panic%3D10%0Akernel.panic_on_oops%3D1%0Akernel.keys.root_maxkeys%3D1000000%0Akernel.keys.root_maxbytes%3D25000000%0A
        mode: 0644
        path: /etc/sysctl.d/90-kubelet.conf
        overwrite: true

This will need to be done for each relevant MachineConfigPool in the cluster.

To configure, follow the directions in the documentation

serverTLSBootstrap: true

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
   name: kubelet-config-$pool
spec:
    machineConfigPoolSelector:
        matchLabels:
            pools.operator.machineconfiguration.openshift.io/$pool_name: ""
    kubeletConfig:
        streamingConnectionIdleTimeout: 

Two types of garbage collection are performed on an OpenShift Container Platform node:

• Container garbage collection: Removes terminated containers.
• Image garbage collection: Removes images not referenced by any running pods.

Container garbage collection can be performed using eviction thresholds. Image garbage collection relies on disk usage as reported by cAdvisor on the node to decide which images to remove from the node.

The OpenShift administrator can configure how OpenShift Container Platform performs garbage collection by creating a kubeletConfig object for each Machine Config Pool using any combination of the following:

• soft eviction for containers
• hard eviction for containers
• eviction for images

To configure, follow the directions in the documentation

This rule pertains to the imagefs.available setting of the evictionHard section.

Two types of garbage collection are performed on an OpenShift Container Platform node:

• Container garbage collection: Removes terminated containers.
• Image garbage collection: Removes images not referenced by any running pods.

Container garbage collection can be performed using eviction thresholds. Image garbage collection relies on disk usage as reported by cAdvisor on the node to decide which images to remove from the node.

The OpenShift administrator can configure how OpenShift Container Platform performs garbage collection by creating a kubeletConfig object for each Machine Config Pool using any combination of the following:

• soft eviction for containers
• hard eviction for containers
• eviction for images

To configure, follow the directions in the documentation

This rule pertains to the memory.available setting of the evictionHard section.

Two types of garbage collection are performed on an OpenShift Container Platform node:

• Container garbage collection: Removes terminated containers.
• Image garbage collection: Removes images not referenced by any running pods.

Container garbage collection can be performed using eviction thresholds. Image garbage collection relies on disk usage as reported by cAdvisor on the node to decide which images to remove from the node.

The OpenShift administrator can configure how OpenShift Container Platform performs garbage collection by creating a kubeletConfig object for each Machine Config Pool using any combination of the following:

• soft eviction for containers
• hard eviction for containers
• eviction for images

To configure, follow the directions in the documentation

This rule pertains to the nodefs.available setting of the evictionHard section.

Two types of garbage collection are performed on an OpenShift Container Platform node:

• Container garbage collection: Removes terminated containers.
• Image garbage collection: Removes images not referenced by any running pods.

Container garbage collection can be performed using eviction thresholds. Image garbage collection relies on disk usage as reported by cAdvisor on the node to decide which images to remove from the node.

The OpenShift administrator can configure how OpenShift Container Platform performs garbage collection by creating a kubeletConfig object for each Machine Config Pool using any combination of the following:

• soft eviction for containers
• hard eviction for containers
• eviction for images

To configure, follow the directions in the documentation

This rule pertains to the nodefs.inodesFree setting of the evictionHard section.

$ sudo chgrp root /etc/cni/net.d/*

$ sudo chgrp root /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-*/configmaps/controller-manager-kubeconfig/kubeconfig

$ sudo chgrp root /var/lib/etcd/member/

$ sudo chgrp root /var/lib/etcd/member/wal/*

$ sudo chgrp root /etc/kubernetes/manifests/etcd-pod.yaml

$ sudo chgrp root /etc/kubernetes/static-pod-resources/*/*/*/*.crt

$ sudo chgrp root /var/lib/cni/networks/openshift-sdn/.*

$ sudo chgrp root /etc/kubernetes/static-pod-resources/kube-apiserver-pod-*/kube-apiserver-pod.yaml

$ sudo chgrp root /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-*/kube-controller-manager-pod.yaml

$ sudo chgrp root /etc/kubernetes/static-pod-resources/kube-scheduler-pod-*/kube-scheduler-pod.yaml

$ sudo chgrp root /etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/node-kubeconfigs/*.kubeconfig

$ sudo chgrp root /var/run/multus/cni/net.d/*

$ sudo chgrp root /etc/kubernetes/static-pod-resources/*/*/*/tls.crt

$ sudo chgrp root /etc/kubernetes/static-pod-resources/*/*/*/*.key

$ sudo chgrp root /var/run/openshift-sdn/cniserver/config.json

$ sudo chgrp root /run/ovn-kubernetes/cni/ovn-cni-server.sock

$ sudo chgrp root /var/lib/ovn/etc/*.db

/var/run/openvswitch/ovs-vswitchd.pid

openvswitch

hugetlbfs

/run/openvswitch/ovs-vswitchd.pid

openvswitch

hugetlbfs

/run/openvswitch/ovsdb-server.pid

openvswitch

hugetlbfs

$ sudo chgrp root /etc/kubernetes/static-pod-resources/kube-scheduler-pod-*/configmaps/scheduler-kubeconfig/kubeconfig

$ sudo chown root /etc/cni/net.d/* 

$ sudo chown root /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-*/configmaps/controller-manager-kubeconfig/kubeconfig 

$ sudo chown root /var/lib/etcd/member/ 

$ sudo chown root /var/lib/etcd/member/wal/* 

$ sudo chown root /etc/kubernetes/manifests/etcd-pod.yaml 

$ sudo chown root /etc/kubernetes/static-pod-resources/*/*/*/*.crt 

$ sudo chown root /var/lib/cni/networks/openshift-sdn/.* 

$ sudo chown root /etc/kubernetes/static-pod-resources/kube-apiserver-pod-*/kube-apiserver-pod.yaml 

$ sudo chown root /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-*/kube-controller-manager-pod.yaml 

$ sudo chown root /etc/kubernetes/static-pod-resources/kube-scheduler-pod-*/kube-scheduler-pod.yaml 

$ sudo chown root /etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/node-kubeconfigs/*.kubeconfig 

$ sudo chown root /var/run/multus/cni/net.d/* 

$ sudo chown root /etc/kubernetes/static-pod-resources/*/*/*/tls.crt 

$ sudo chown root /etc/kubernetes/static-pod-resources/*/*/*/*.key 

$ sudo chown root /var/run/openshift-sdn/cniserver/config.json 

$ sudo chown root /run/ovn-kubernetes/cni/ovn-cni-server.sock 

$ sudo chown root /var/lib/ovn/etc/*.db 

$ sudo chown openvswitch /etc/openvswitch/conf.db 

$ sudo chown openvswitch /etc/openvswitch/.conf.db.~lock~ 

$ sudo chown openvswitch /var/run/openvswitch/ovs-vswitchd.pid 

$ sudo chown openvswitch /etc/openvswitch/system-id.conf 

$ sudo chown openvswitch /run/openvswitch/ovs-vswitchd.pid 

$ sudo chown openvswitch /run/openvswitch/ovsdb-server.pid 

$ sudo chown root /etc/kubernetes/static-pod-resources/kube-scheduler-pod-*/configmaps/scheduler-kubeconfig/kubeconfig 

$ sudo chmod 0600 /etc/cni/net.d/*

$ sudo chmod 0600 /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-*/configmaps/controller-manager-kubeconfig/kubeconfig

$ sudo chmod 0700 /var/lib/etcd

$ sudo chmod 0600 /var/lib/etcd/member/wal/*

$ sudo chmod 0600 /etc/kubernetes/manifests/etcd-pod.yaml

$ sudo chmod 0600 /etc/kubernetes/static-pod-resources/etcd-*/secrets/*/*.crt

$ sudo chmod 0644 /var/lib/cni/networks/openshift-sdn/*

$ sudo chmod 0600 /etc/kubernetes/static-pod-resources/kube-apiserver-pod-*/kube-apiserver-pod.yaml

$ sudo chmod 0600 /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-*/kube-controller-manager-pod.yaml

$ sudo chmod 0600 /etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/node-kubeconfigs/*.kubeconfig

$ sudo chmod 0644 /var/run/multus/cni/net.d/*

$ sudo chmod 0600 /etc/kubernetes/static-pod-resources/kube-*/secrets/*/tls.crt

$ sudo chmod 0600 /etc/kubernetes/static-pod-resources/*/*/*/*.key

$ sudo chmod 0600 /run/ovn-kubernetes/cni/ovn-cni-server.sock

$ sudo chmod 0640 /var/lib/ovn/etc/*.db

$ sudo chmod 0640 /etc/openvswitch/conf.db

$ sudo chmod 0600 /etc/openvswitch/.conf.db.~lock~

$ sudo chmod 0644 /var/run/openvswitch/ovs-vswitchd.pid

$ sudo chmod 0644 /etc/openvswitch/system-id.conf

$ sudo chmod 0644 /run/openvswitch/ovs-vswitchd.pid

$ sudo chmod 0644 /run/openvswitch/ovsdb-server.pid

$ sudo chmod 0644 /etc/kubernetes/static-pod-resources/kube-scheduler-pod-*/kube-scheduler-pod.yaml

$ sudo chmod 0600 /etc/kubernetes/static-pod-resources/kube-scheduler-pod-*/configmaps/scheduler-kubeconfig/kubeconfig

$ sudo chmod 0444 /var/run/openshift-sdn/cniserver/config.json