When developing or using Coder workspaces that depend on Sysbox, you might encounter the following Pod Warning from a kubelet Error event being emitted as the template attempts to start.
Error: container create failed: time="«timestamp»" level=error msg="container_linux.go:439: starting container process caused: process_linux.go:608: container init caused: process_linux.go:579: handleReqOp caused: failed to mount /var/lib/containers/storage/overlay/«container_layer_id»/merged: invalid argument"For example
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 77s default-scheduler Successfully assigned coder/coder-tiwarpa-test-devcontainer-648fc4b78b-b5b5g to ip-172-16-9-139.eu-central-1.compute.internal
Normal Pulled 77s kubelet Successfully pulled image "154453013990.dkr.ecr.eu-central-1.amazonaws.com/coder/envbuilder:latest" in 142ms (142ms including waiting)
Warning Failed 77s kubelet Error: container create failed: time="2024-10-09T14:33:17Z" level=error msg="container_linux.go:439: starting container process caused: process_linux.go:608: container init caused: process_linux.go:579: handleReqOp caused: failed to mount /var/lib/containers/storage/overlay/1a2cd7d4b7ebbfab35235475f41c999dd7c0dac5c0cc0e1a845fe030de7f7b47/merged: invalid argument"
Normal Pulled 76s kubelet Successfully pulled image "154453013990.dkr.ecr.eu-central-1.amazonaws.com/coder/envbuilder:latest" in 97ms (97ms including waiting)
Warning Failed 76s kubelet Error: container create failed: time="2024-10-09T14:33:18Z" level=error msg="container_linux.go:439: starting container process caused: process_linux.go:608: container init caused: process_linux.go:579: handleReqOp caused: failed to mount /var/lib/containers/storage/overlay/a1ed38ade613e4f17443170a788c90d8c0d26b2677a7221faaf91d5ed22ce4b8/merged: invalid argument"
Warning Failed 63s kubelet Error: container create failed: time="2024-10-09T14:33:31Z" level=error msg="container_linux.go:439: starting container process caused: process_linux.go:608: container init caused: process_linux.go:579: handleReqOp caused: failed to mount /var/lib/containers/storage/overlay/c23927709136d2e93475ebb65f601bf35ae9355a25786b27e6e47d506f933bb4/merged: invalid argument"
Normal Pulled 63s kubelet Successfully pulled image "154453013990.dkr.ecr.eu-central-1.amazonaws.com/coder/envbuilder:latest" in 109ms (109ms including waiting)
Normal Pulled 52s kubelet Successfully pulled image "154453013990.dkr.ecr.eu-central-1.amazonaws.com/coder/envbuilder:latest" in 104ms (104ms including waiting)
Warning Failed 52s kubelet Error: container create failed: time="2024-10-09T14:33:42Z" level=error msg="container_linux.go:439: starting container process caused: process_linux.go:608: container init caused: process_linux.go:579: handleReqOp caused: failed to mount /var/lib/containers/storage/overlay/e72b16aed86f1ea0ddfd2419f286eb2318bf3ec3a2cf6582f1b8a5be818f4a31/merged: invalid argument"
Normal Pulled 40s kubelet Successfully pulled image "154453013990.dkr.ecr.eu-central-1.amazonaws.com/coder/envbuilder:latest" in 108ms (108ms including waiting)
Warning Failed 40s kubelet Error: container create failed: time="2024-10-09T14:33:54Z" level=error msg="container_linux.go:439: starting container process caused: process_linux.go:608: container init caused: process_linux.go:579: handleReqOp caused: failed to mount /var/lib/containers/storage/overlay/adf313de6df2879b3f95e58c428efe310147376b5cbc42e45f59194cfad36651/merged: invalid argument"
Normal Pulled 25s kubelet Successfully pulled image "154453013990.dkr.ecr.eu-central-1.amazonaws.com/coder/envbuilder:latest" in 124ms (124ms including waiting)
Warning Failed 25s kubelet Error: x: time="2024-10-09T14:34:09Z" level=error msg="container_linux.go:439: starting container process caused: process_linux.go:608: container init caused: process_linux.go:579: handleReqOp caused: failed to mount /var/lib/containers/storage/overlay/42f0c7f463d395ead70babeef02d1ac92f6c2160042087f68c39a96e29208e29/merged: invalid argument"
Normal Pulled 13s kubelet Successfully pulled image "154453013990.dkr.ecr.eu-central-1.amazonaws.com/coder/envbuilder:latest" in 110ms (110ms including waiting)
Warning Failed 13s kubelet Error: container create failed: time="2024-10-09T14:34:21Z" level=error msg="container_linux.go:439: starting container process caused: process_linux.go:608: container init caused: process_linux.go:579: handleReqOp caused: failed to mount /var/lib/containers/storage/overlay/bf3b1c045f34f115b625b79ea2382b4efb9b2558e271181a268f201a4d305399/merged: invalid argument"
Normal Pulling 2s (x8 over 77s) kubelet Pulling image "154453013990.dkr.ecr.eu-central-1.amazonaws.com/coder/envbuilder:latest"
Normal Pulled 2s kubelet Successfully pulled image "154453013990.dkr.ecr.eu-central-1.amazonaws.com/coder/envbuilder:latest" in 100ms (100ms including waiting)
Warning Failed 2s kubelet Error: container create failed: time="2024-10-09T14:34:32Z" level=error msg="container_linux.go:439: starting container process caused: process_linux.go:608: container init caused: process_linux.go:579: handleReqOp caused: failed to mount /var/lib/containers/storage/overlay/930f190641c0fa9991fedce131b2df1ce52c83e81f0729447cbd2e535f13ae39/merged: invalid argument"Troubleshooting
This can be caused by omitting a Pod annotation required by sysbox. This is most easily done when converting a Pod to a Deployment. The example for sysbox shows an annotated Pod specification; however in a Deployment specification, the annotation should be made on the Pod template, not the deployment itself.
This kubectl command can check if the Deployment's Pod spec template contains the correct annotation.
kubectl get deployment/$CODER_WORKSPACE_DEPLOYMENT_NAME \
--template='
{{- with .spec.template.metadata.annotations -}}
{{- "io.kubernetes.cri-o.userns-mode" | index . | printf "%q\n" -}}
{{- else -}}
{{- print "Missing required annotation!" -}}
{{- end -}}'