Defining a VM Pod

The basic idea of a VM pod is that it's a plain Kubernetes pod definition with the following conditions satisfied:

It has kubernetes.io/target-runtime: virtlet.cloud annotation so it can be recognized by CRI Proxy.
The pod has exactly one container.
The container's image has virtlet.cloud/ prefix followed by the image name which is recognized according to Virtlet's image handling rules and used to download the QCOW2 image for the VM.

If you have Virtlet running only on some of the nodes in the cluster, you also need to specify either nodeSelector or nodeAffinity for the pod to have it land on a node with Virtlet. If a VM pod lands on a node which doesn't have Virtlet or where Virtlet and CRI Proxy aren't configured properly, you can see the following messages in kubectl describe output for the VM pod (the message can be printed as a single line):

Warning  Failed     5s (x2 over 17s)  kubelet, kubemaster
Failed to pull image "virtlet.cloud/cirros":
rpc error: code = Unknown desc = Error response from daemon:
Get https://virtlet.cloud/v2/: dial tcp 50.63.202.10:443:
connect: connection refused

This means that kubelet is trying to pull the VM image from a Docker registry.

It's also possible to construct higher-level Kubernetes objects such as Deployment, StatefulSet or DaemonSet out of VM pods, in which case the template section of the object must follow the above rules for VM pods.

Below is an example of a Virtlet pod. The comments describe the particular parts of the pod spec. The following sections will give more details on each part of the spec.

# Standard k8s pod header
apiVersion: v1
kind: Pod
metadata:
  # the name of the pod
  name: cirros-vm
  # See 'Annotations recognized by Virtlet' below
  annotations:
    # This tells CRI Proxy that this pod belongs to Virtlet runtime
    kubernetes.io/target-runtime: virtlet.cloud
    # An optional annotation specifying the count of virtual CPUs.
    # Defaults to "1".
    VirtletVCPUCount: "1"
    # CirrOS doesn't load nocloud data from SCSI CD-ROM for some reason
    VirtletDiskDriver: virtio
    # inject ssh keys via cloud-init
    VirtletSSHKeys: |
      ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCaJEcFDXEK2ZbX0ZLS1EIYFZRbDAcRfuVjpstSc0De8+sV1aiu+dePxdkuDRwqFtCyk6dEZkssjOkBXtri00MECLkir6FcH3kKOJtbJ6vy3uaJc9w1ERo+wyl6SkAh/+JTJkp7QRXj8oylW5E20LsbnA/dIwWzAF51PPwF7A7FtNg9DnwPqMkxFo1Th/buOMKbP5ZA1mmNNtmzbMpMfJATvVyiv3ccsSJKOiyQr6UG+j7sc/7jMVz5Xk34Vd0l8GwcB0334MchHckmqDB142h/NCWTr8oLakDNvkfC1YneAfAO41hDkUbxPtVBG5M/o7P4fxoqiHEX+ZLfRxDtHB53 [email protected]
    # set root volume size
    VirtletRootVolumeSize: 1Gi
spec:
  # This nodeAffinity specification tells Kubernetes to run this
  # pod only on the nodes that have extraRuntime=virtlet label.
  # This label is used by Virtlet DaemonSet to select nodes
  # that must have Virtlet runtime
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: extraRuntime
            operator: In
            values:
            - virtlet
  containers:
  - name: cirros-vm
    # This specifies the image to use.
    # virtlet.cloud/ prefix is used by CRI proxy, the remaining part
    # of the image name is prepended with https:// and used to download the image
    image: virtlet.cloud/cirros
    imagePullPolicy: IfNotPresent
    # tty and stdin required for `kubectl attach -t` to work
    tty: true
    stdin: true
    resources:
      limits:
        # This memory limit is applied to the libvirt domain definition
        memory: 160Mi

Annotations recognized by Virtlet

The annotations can be specified under annotations key of the metadata part of the pod spec. Note that the values are always strings, but they can be parsed in different way by Virtlet (see Type column).

For boolean keys, the string "true" (lowercase) is interpreted as a true value. All other values are interpreted as false.

Several keys belong to the Cloud-Init settings which are described in detail in the corresponding section.

Key	Description	Value	Default
_{kubernetes.io/target-runtime}	CRI runtime setting for CRI Proxy	`virtlet.cloud`	`virtlet.cloud`
_{VirtletChown9pfsMounts}	Recursively chown 9pfs mounts	boolean	`""`
_{VirtletCloudInitImageType}	Cloud-Init image type to use	`"nocloud"` `"configdrive"`	`""`
_{VirtletCloudInitMetaData}	The contents of Cloud-Init metadata	json / yaml	`""`
_{VirtletCloudInitUserData}	The contents of Cloud-Init user-data (mergeable)	json / yaml	`""`
_{VirtletCloudInitUserDataOverwrite}	Disable merging of Cloud-Init user-data keys	boolean	`""`
_{VirtletCloudInitUserDataScript}	The contents of Cloud-Init user-data as a script	text	`""`
_{VirtletCloudInitUserDataSource}	Data source for Cloud-Init user-data	`"configmap/..."` `"secret/..."`	`""`
_{VirtletCloudInitUserDataSourceEncoding}	Encoding to use for loading Cloud-Init user-data from a ConfigMap key	`"plain"`	`"base\|4"`
_{VirtletCloudInitUserDataSourceKey}	ConfigMap key to load Cloud-Init user-data from		`""`
_{VirtletCPUModel}	CPU model to use	`""` `"host-model"`	`""`
_{VirtletDiskDriver}	Disk driver to use	`"scsi"` `"virtio"`	`"scsi"`
_{VirtletFilesFromDataSource}	Inject files from a ConfigMap or a Secret into the image	`"configmap/..."` `"secret/..."`	`""`
_{VirtletLibvirtCPUSetting}	libvirt CPU model setting	yaml	`""`
_{VirtletRootVolumeSize}	Root volume size	quantity	`""`
_{VirtletSSHKeys}	SSH keys to add to the VM injected via Cloud-Init	a list of strings	`""`
_{VirtletSSHKeySource}	Data source for ssh keys injected via Cloud-Init	`"configmap/..."` `"secret/..."`	`""`
_{VirtletVCPUCount}	The number of vCPUs to assign to the VM pod	integer	`"1"`

CRI Proxy annotation

Besides Virtlet annotations, there's kubernetes.io/target-runtime: virtlet.cloud annotation which is handled by CRI Proxy. It's important to specify it as well as virtlet.cloud prefix for CRI Proxy to be able to direct requests to Virtlet.

Chowning 9pfs mounts

Setting VirtletChown9pfsMounts to true causes 9pfs mounts to chown their volume contents to make it readable and writable by the VM.

CPU Model

VirtletCPUModel: host-model annotation enables nested virtualization. VirtletLibvirtCPUSetting is an expert-only annotation that sets the CPU options for libvirt. The YAML keys correspond to the XML elements and attributes in the libvirt XML definition, but are capitalized. The value of Model field goes into the Value key. For example:

Match: exact
Model:
  Fallback: allow
  Value: core2duo

Disk driver

The driver is set using VirtletDiskDriver annotation which may have the value of scsi (the default) or virtio. Some OS images may have problem with the default scsi driver, for example, CirrOS can't handle Cloud-Init data unless virtio driver is used.

Injecting files into the image

By using VirtletFilesFromDataSource annotation, it's possible to place the contents of a ConfigMap or a Secret on the image before booting the VM. For more information, refer to Injecting files into the VM.

vCPU count

Virtlet defaults to using just one vCPU per VM. You can change this value by setting VirtletVCPUCount annotation to the desired value, for example, VirtletVCPUCount: "2".

Volume handling

Virtlet can recognize and handle pod's volumes and container's volumeMounts sections. This can also be used to make the VM use a persistent root filesystem which will survive pod removal and re-creation. For more information on working with volumes, please refer to the Volumes section.

Environment variables

Virtlet supports passing environment variables to the VM using the standard env settings in the container definition:

...
spec:
...
containers:
  - name: cirros-vm
    ...
    env:
    - name: MY_FOO_VAR
      value: foo
    - name: MY_FOOBAR_VAR
      value: foobar

Virtlet uses Cloud-Init mechanisms to write the values into /etc/cloud/environment file inside the VM which has the same key=value per line format as /etc/environment and can be either read by an application or sourced by a shell:

MY_FOO_VAR=foo
MY_FOOBAR_VAR=foobar

For this environment mechanism to work, the cloud-init implementation inside the VM must be able to handle write_files inside the Cloud-Init user-data.