kubevirt创建虚拟机问题记录

环境

Ubuntu 20.04.6 LTS

k8s 1.28.0 【1个master，2个node】

kubevirt 0.57.0

问题复现

​ 1. 在k8s集群上通过以下命令安装kubevirt

kubectl create -f https://github.com/kubevirt/kubevirt/releases/download/v1.0.0/kubevirt-operator.yaml

kubectl create -f https://github.com/kubevirt/kubevirt/releases/download/v1.0.0/kubevirt-cr.yaml

​ 2. 安装virtctl

wget https://github.com/kubevirt/kubevirt/releases/download/v0.57.0/virtctl-v0.57.0-linux-amd64 -O virtctl

chmod +x virtctl
sudo install virtctl /usr/local/bin

上述安装完成后相应的pod

​ 3. 创建虚拟机

vm.yaml文件（来自官网: https://kubevirt.io/labs/manifests/vm.yaml)

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: testvm
spec:
  running: false
  template:
    metadata:
      labels:
        kubevirt.io/size: small
        kubevirt.io/domain: testvm
    spec:
      domain:
        devices:
          disks:
            - name: containerdisk
              disk:
                bus: virtio
            - name: cloudinitdisk
              disk:
                bus: virtio
          interfaces:
          - name: default
            masquerade: {}
        resources:
          requests:
            memory: 64M
      networks:
      - name: default
        pod: {}
      volumes:
        - name: containerdisk
          containerDisk:
            image: quay.io/kubevirt/cirros-container-disk-demo
        - name: cloudinitdisk
          cloudInitNoCloud:
            userDataBase64: SGkuXG4=

创建vm: kubectl create -f ym.yaml

​ 4. 查看vm状态

​ 5. 启动vm: virtctl start test-vm

再次查看状态

​ 6. 查看描述信息

 Message:               0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling..

#

#

问题1

这里虚拟机无法正常启动，上述红框中所示问题。我在网上查询了资料，但是也没有找到解决问题的方法，况哥之前有遇到过这种问题吗？

==========================排查过程=========================

自己补充了点关于k8s污点、容忍、亲和力相关的知识点。知道了这个报错可能是因为对应的pod通过nodeSelector或者污点导致找不到对应的node进行调度。

​ 1. 找到虚拟机对应的pod，进行describe描述

这里声明了节点选择器,必须有节点满足kubevirt.io/schedulable=true (opens new window) 才能进行调度

然后我们通过以下命令查看节点的lebal

kubectl  get  node openstacktest-1 --show-labels | grep "kubevirt.io/schedulable"

预期：我们把这个标签改为true，那么对应的vm节点的nodeSelector可以正常调度到对应的节点

我们使用label命令进行改变

kubectl label node openstacktest-1 kubevirt.io/schedulable=true --overwrite

命令指令后，我们通过查看标签没有问题

kubectl  get  node openstacktest-1 --show-labels | grep "kubevirt.io/schedulable"

但是我们隔一分钟过后，再看标签，发现标签又变回了false

问题2

是什么原因导致我使用label命令更改node的label为kubevirt.io/schedulable=true (opens new window)，过一段时间后，对应label变为了kubevirt.io/schedulable=false (opens new window)

根本原因是virt-handler对node节点的label进行了更改

Annotations and labels - KubeVirt user guide (opens new window)

Unresponsive nodes - KubeVirt user guide (opens new window)

更新于：2023.9.12

什么是嵌套虚拟化？：

嵌套虚拟化是指在虚拟机中运行另一个虚拟机的技术。它允许在一台物理机器上运行多个虚拟机，每个虚拟机都可以运行自己的操作系统。这种技术可以用于在生产环境中测试新的操作系统和软件，或者在教育和研究环境中模拟复杂的网络环境。嵌套虚拟化需要在物理机器上安装虚拟化软件，并在虚拟机中启用虚拟化支持。

解决方法：

重新安装了k8s 1.28和网络组件，并使用的kubeVirt版本为v1.0.0

过程中需要开启嵌套虚拟化功能[如果虚拟化不可用，则需要手动开启软件仿真（虚拟机必须开）]

kubectl create   configmap kubevirt-config -n kubevirt --from-literal=debug.useEmulation=true --from-literal=feature-gates=Macvtap,LiveMigration,Snapshot
# LiveMigration 开启迁移功能
# Snapshot 开启快照功能

k8s集群搭建这里不做介绍，主要看看kubevirt安装过程

kubevirt安装与基本使用 - 掘金 (opens new window)

按照上述创建并启动虚拟机后，可以通过virtctl console命令连接到对应的虚拟机

虚拟机默认用户和密码：

user: cirros

password: gocubsgo

附录

完整描述信息

root@openstackTest-1:~/kubevirt# kubectl describe vm test-vm
Name:         test-vm
Namespace:    default
Labels:       <none>
Annotations:  kubevirt.io/latest-observed-api-version: v1
              kubevirt.io/storage-observed-api-version: v1alpha3
API Version:  kubevirt.io/v1
Kind:         VirtualMachine
Metadata:
  Creation Timestamp:  2023-09-05T02:30:06Z
  Generation:          2
  Resource Version:    518769
  UID:                 1e119181-78cc-4e0b-9ea7-b1688c94e7c0
Spec:
  Running:  true
  Template:
    Metadata:
      Creation Timestamp:  <nil>
      Labels:
        kubevirt.io/domain:  testvm
        kubevirt.io/size:    small
    Spec:
      Domain:
        Devices:
          Disks:
            Disk:
              Bus:  virtio
            Name:   containerdisk
          Interfaces:
            Masquerade:
            Name:  default
        Machine:
          Type:  q35
        Resources:
          Requests:
            Memory:  64M
      Networks:
        Name:  default
        Pod:
      Node Selector:
        Role:  worker
      Volumes:
        Container Disk:
          Image:  quay.io/kubevirt/cirros-container-disk-demo
        Name:     containerdisk
Status:
  Conditions:
    Last Probe Time:       2023-09-05T02:31:05Z
    Last Transition Time:  2023-09-05T02:31:05Z
    Message:               Guest VM is not reported as running
    Reason:                GuestNotRunning
    Status:                False
    Type:                  Ready
    Last Probe Time:       <nil>
    Last Transition Time:  2023-09-05T02:31:05Z
    Message:               0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling..
    Reason:                Unschedulable
    Status:                False
    Type:                  PodScheduled
  Created:                 true
  Printable Status:        ErrorUnschedulable
  Volume Snapshot Statuses:
    Enabled:  false
    Name:     containerdisk
    Reason:   Snapshot is not supported for this volumeSource type [containerdisk]
Events:
  Type    Reason            Age   From                       Message
  ----    ------            ----  ----                       -------
  Normal  SuccessfulCreate  51s   virtualmachine-controller  Started the virtual machine by creating the new virtual machine instance test-vm