Kubernetes Cluster Security Model

Question

I am being forced to run my app in a Kubernetes cluster that is shared with another team. The team is not security minded. What is the same cluster security model? Will the other team be able to get root in my containers? Login to my containers? DoS my service? Other risk I haven't even considered. I can't find many good docs on K8S security. Any good docs, pages, or books?

Answer 1

What you're looking for here is whether Kubernetes is secure for Multi-Tenancy, and that's an interesting question which depends on a number of factors.

Out of the box Kubernetes is not secure for multi-tenancy as any user who has the ability to create workloads in a cluster can specify parameters which allow them privileged access to the underlying node.

Another challenge is the networking which, in most cases, is a flat network where all workloads running in the cluster can directly communicate with all others.

Then you have things like CVE-2020-8554 which will (in many clusters) allow any user who can create service objects to hijack traffic from other workloads in the cluster.

Then there's the risk of container breakout, there have beeen CVEs over the last couple of years that allow for a user in one container to break out to the underlying host (some details here)

It's possible to mitigate all of these, but requires quite a bit of work on the part of the cluster owner. Typically my recommendation for people who are worried about attacks from other users is to run separate clusters.

On the other topic about good Kubernetes security resources, there are quite a few for you to look at.

There's an O'Reilly book on container security here, then there's one on Kubernetes Hacking here

There are some training resources that I maintain the list for here and a reading list if you want more details on container technologies here.

Then if you're interested in Hardening Kubernetes, there's the CIS benchmark or the NSA Hardening guide and a long series of posts which detail some basic Kubernetes security concerns, that relate to the PCI guidance for container orchestration securiy here

Answer 2

Shared clusters are the native operation model for Kubernetes. It's part of the solution, it reduces infrastructure overhead and scaling to multiple thousands of containers is possible.

I am not a k8s expert, still very much in the beginning, so whoever knows better, feel free to correct me.

A k8s is nothing else as a container orchestration engine, managing the lifecycle of all containers running in the cluster, including network and storage. Regarding security we now must look at different layers to be able to model our threats and risks properly.

Layer 1 Infrastructure

A cluster consists of at least 3 nodes, either physical or virtual. Anyone with physical or network access to the nodes can at least access everything that is running on that specific node. For the sake of this answer, we trust this layer.

Layer 2 Networking

Besides the physical networking, k8s provides a software defined network to connect all the containers, much like in a docker based network. To handle INGRESS and EGRESS, typically a load balancer is running on a public IP; In a multi-tenant-model mostly one LB for each namespace, which gives us the public IP.

In networking terms, we can think of a Namespace as a private subnet, where the Loadbalancer acts as gateway to the internet.

Inter-Namespace routing is disabled by default, but it can be activated by the cluster admin.

So to make your app world accessible in your namespace, you must deploy a INGRESS controller like Traefik(there are many others, but I only know Traefik)

This ingress then communicates with the pods(aka containers) in your deployment, mostly load balanced by a Service-IP. There are tools that enable HTTPS for intra-cluster communication, but you need to adapt your App accordingly, using those certificates etc. This prevents eavesdropping on the infrastructure layer when your cluster is spread over multiple datacenters etc.

Layer 3 Access control

k8ws follows a RBAC approach, role-based access control. The Cluster Admin can access everything in the cluster and can create new roles, like Your-App-Namespace-Admin and hand it to you.

This role defines what a user can do in wich namespace, another user in a different namespace cannot access a ressource in your namespace by default unless defined in his role. Here is a good example for a role that can read all secrets in the cluster:

Recommendations

For running an application securely in a shared cluster I would address the following things:

• Ask the Cluster Admin his security concept and role model. Maybe let you show the roles and ask about a written down concept.
• Use a dedicated Ingress Controller in your namespace to be in control of the Network flow
• Setup HTTPS for intra-namespace communication properly

Some other recommendations, not specifically limited to shared clusters:

• use inmutable containers
• use rootless containers
• strip your containers from all tools not necessary to run your application

This will reduce the amount of damage an attacker that gains access to your deployments can do.

More information on Namespacee More information on RBAC

Hope this helps.