Container-based Virtual Machines

⚠⚠ Released behind an alpha admin flag in v1.14.0

By default, a standard Coder Environment runs inside a normal Linux container. Unfortunately, this carries limitations as to what applications a user can run inside of their Environment. For example, system-level applications like Docker, Kubernetes, and systemd cannot run as nested processes within a standard Linux container.

To solve this problem, Coder offers an alternative deployment strategy for Environments: Container-based Virtual Machines (CVM). Rather than creating a single isolated container, Coder leverages the sysbox container runtime to offer a VM-like experience without the performance overhead of typical virtualization solutions. CVM Environments have all the same image, rebuild, start, and stop semantics as with the standard deployment strategy, and the cost structure is identical given their ability to use shared CPU and memory. 

Users are able to run Docker, systemd, and other system-level applications inside of CVM Environments.

Coder first launches a supervising container with privileged. This container is standard and included as part of the Coder release package. As part of the Environment build process, the supervising container launches an inner container using the sysbox container runtime. This inner container is the user’s Environment. The user cannot gain access to the supervising container at any point during the lifecycle of their Environment. The isolation between the user’s Environment container and it’s outer, supervising container is what provides strong isolation.

Allowing use of Container-based Virtual Machines

Infrastructure requirements

1. The Kubernetes nodes must be running a kernel version >= 5.4 (released Nov 24th, 2019).

Admins can verify that the kernel is sufficient by running the following inside a Kubernetes node, where the output should show >= 5.4.

$ uname -r
5.4.0-1024-gcp

For deployments running on AWS EKS, this requirement can be satisfied by using an AMI family of Ubuntu2004.

2. The cluster must allow privileged containers and hostPath mounts. Remember, this only provides increased control to the supervisor container, not the user Environment.

Known incompatibilities

During the alpha period, the Container-based Virtual Machine environment variant has known incompatibilities with certain versions of the Istio proxy service. 

Creating an Environment with the CVM deployment strategy