Вот перевод текста сообщения на русский язык:

`root@gpu01:~# ceph versions`
`{
   "mon": {
       "ceph version 14.2.2 (a887fe9a5d3d97fe349065d3c1c9dbd7b8870855) nautilus (stable)": 3
   },
   "mgr": {
       "ceph version 14.2.2 (a887fe9a5d3d97fe349065d3c1c9dbd7b8870855) nautilus (stable)": 1
   },
   "osd": {
       "ceph version 14.2.2 (a887fe9a5d3d97fe349065d3c1c9dbd7b8870855) nautilus (stable)": 96
   },
   "mds": {},
   "overall": {
       "ceph version 14.2.2 (a887fe9a5d3d97fe349065d3c1c9dbd7b8870855) nautilus (stable)": 1
   }
}`

`Oct  2 20:37:39 gpu01-la3 pvedaemon[508130]: <mquery@pam> end task UPID:gpu01:0033252C:0467553A:5D956CE3:qmstart:103:mquery@pam: start failed: command '/usr/bin/kvm -id 103 -name winclient103 -chardev 'socket,id=qmp,path=/var/run/qemu-server/103.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/103.pid -daemonize -smbios 'type=1,uuid=e5190607-223f-4886-a9eb-1650123a9b66' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/rbd/rbd/vm-103-disk-0' -smp '8,sockets=1,cores=8,maxcpus=8' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,hv_ipi,kvm=off' -m 16384 -object 'memory-backend-ram,id=ram-node0,size=16384M' -numa 'node,nodeid=0,cpus=0-7,memdev=ram-node0' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=b2:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on' -device 'vfio-pci,host=b2:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1' -device 'vfio-pci,host=b2:00.2,id=hostpci0.2,bus=ich9-pcie-port-1,addr=0x0.2' -device 'vfio-pci,host=b2:00.3,id=hostpci0.3,bus=ich9-pcie-port-1,addr=0x0.3' -chardev 'socket,path=/var/run/qemu-server/103.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:cb85eb190f0' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/rbd/rbd/vm-103-disk-1,if=none,id=drive-scsi0,cache=writeback,discard=on,throttling.bps-read=419430400,throttling.iops-read=8000,throttling.bps-write=419430400,throttling.iops-write=8000,format=raw,aio=threads,detect-zeroes=unmap' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -drive 'file=/dev/rbd/rbd/vm-103-disk-2,if=none,id=drive-scsi1,cache=writeback,discard=on,throttling.bps-read=419430400,throttling.iops-read=8000,throttling.bps-write=419430400,throttling.iops-write=8000,format=raw,aio=threads,detect-zeroes=unmap' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1' -netdev 'type=tap,id=net0,ifname=tap103i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=3A:C2:94:31:A4:C4,netdev=net0,bus=pci.0,addr=0x12,id=net0' -rtc 'driftfix=slew,base=localtime' -machine 'type=pc-q35-3.1' -global 'kvm-pit.lost_tick_policy=discard'' failed: got timeout`

`Oct  2 20:37:39 gpu01-la3 pvedaemon[508130]: <mquery@pam> end task UPID:gpu01:0033252C:0467553A:5D956CE3:qmstart:103:mquery@pam: start failed: command '/usr/bin/kvm -id 103 -name winclient103 -chardev 'socket,id=qmp,path=/var/run/qemu-server/103.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/103.pid -daemonize -smbios 'type=1,uuid=e5190607-223f-4886-a9eb-1650123a9b66' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/rbd/rbd/vm-103-disk-0' -smp '8,sockets=1,cores=8,maxcpus=8' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,hv_ipi,kvm=off' -m 16384 -object 'memory-backend-ram,id=ram-node0,size=16384M' -numa 'node,nodeid=0,cpus=0-7,memdev=ram-node0' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=b2:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on' -device 'vfio-pci,host=b2:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1' -device 'vfio-pci,host=b2:00.2,id=hostpci0.2,bus=ich9-pcie-port-1,addr=0x0.2' -device 'vfio-pci,host=b2:00.3,id=hostpci0.3,bus=ich9-pcie-port-1,addr=0x0.3' -chardev 'socket,path=/var/run/qemu-server/103.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:cb85eb190f0' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/rbd/rbd/vm-103-disk-1,if=none,id=drive-scsi0,cache=writeback,discard=on,throttling.bps-read=419430400,throttling.iops-read=8000,throttling.bps-write=419430400,throttling.iops-write=8000,format=raw,aio=threads,detect-zeroes=unmap' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -drive 'file=/dev/rbd/rbd/vm-103-disk-2,if=none,id=drive-scsi1,cache=writeback,discard=on,throttling.bps-read=419430400,throttling.iops-read=8000,throttling.bps-write=419430400,throttling.iops-write=8000,format=raw,aio=threads,detect-zeroes=unmap' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1' -netdev 'type=tap,id=net0,ifname=tap103i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=3A:C2:94:31:A4:C4,netdev=net0,bus=pci.0,addr=0x12,id=net0' -rtc 'driftfix=slew,base=localtime' -machine 'type=pc-q35-3.1' -global 'kvm-pit.lost_tick_policy=discard'' failed: got timeout`

**Summary:**

The output shows the results of a failed attempt to start a virtual machine (VM) named `winclient103`. The error is a "timeout," indicating that the VM failed to initialize within the expected timeframe.  The log message `<mquery@pam> end task UPID:gpu01:0033252C:0467553A:5D956CE3:qmstart:103:mquery@pam: start failed: command ... failed: got timeout` suggests that the issue is with the command executed by Proxmox to start the VM using KVM. The command itself is very long and lists all the hardware configuration details for the VM.

**Possible Causes:**

*   **Resource Contention:** The host system might be under heavy load, preventing the VM from acquiring necessary resources.
*   **Storage Issues:** Problems accessing the VM's storage (e.g., RBD storage) could lead to a timeout.
*   **Network Issues:** Network configuration problems could also cause a timeout.
*   **Hardware Compatibility:** There might be compatibility issues between the VM's hardware configuration and the host system.
*   **KVM Issues:** A problem with the KVM hypervisor itself could be causing the timeout.

**Further Investigation:**

*   **Check Host Resources:** Monitor the host system's CPU, memory, and disk usage.
*   **Examine Storage:** Verify the health of the storage devices used by the VM.
*   **Review Network Configuration:** Double-check the VM's network settings and the host system's network configuration.
*   **Check KVM Logs:** Examine the KVM logs for more detailed error messages.
*   **Simplify Configuration:** Try starting the VM with a minimal configuration to see if that resolves the timeout.

И еще, что покажет #rbd info rbd/vm-104-disk-1?

В данный момент Proxmox отключает некоторые функции /usr/share/perl5/PVE/Storage/RBDPlugin.pm.  У меня тут такой код: my $krbd_feature_blacklist = ['deep-flatten', 'fast-diff', 'object-map', 'exclusive-lock']; Это было сделано для старого ядра на Proxmox 5, но обновление не проводилось. Я думаю об exclusive-lock (с версии ядра 4.10), может, это поможет в твоем случае. #rbd feature enable rbd/vm-104-disk-1 exclusive-lock

Code: root@gpu01:~# rbd showmapped
id pool namespace image         snap device
0  rbd            vm-101-disk-1 -    /dev/rbd0
1  rbd            vm-101-disk-2 -    /dev/rbd1
2  rbd            vm-101-disk-0 -    /dev/rbd2
3  rbd            vm-104-disk-1 -    /dev/rbd3
4  rbd            vm-104-disk-2 -    /dev/rbd4
5  rbd            vm-104-disk-0 -    /dev/rbd5
6  rbd            vm-102-disk-1 -    /dev/rbd6
7  rbd            vm-102-disk-2 -    /dev/rbd7
8  rbd            vm-102-disk-0 -    /dev/rbd8

root@gpu01:~# rbd map rbd/vm-103-disk-2
/dev/rbd9

root@gpu01:~# rbd unmap rbd/vm-103-disk-2 Это странно, потому что через CLI всё работает, но когда PVE пытается запустить VM с RBD диском, возникает ошибка.

Ну, блин, CephX отключен.

Нет, я не.... Код: root@gpu01:/etc/pve/priv/ceph# ls -al
всего: 0
drwx------ 2 root www-data 0 Авг 12 05:10 .
drwx------ 2 root www-data 0 3 мая 05:02 ..

Временное решение этой проблемы – перезагрузить хост pve. Тогда он снова перестанет работать, выдавая ошибку записи sysfs при подключении дополнительного rbd-диска через некоторое время.