Краткое описание: Создал кластер, добавил узлы (работает только через `--use_ssh`). При попытке посмотреть статус другого узла – запрос к `https://node-a:8006/api2/json/nodes/node-b/storage/local/status` возвращает ошибку: 596 таймаут соединения. Но если зайти на каждый узел напрямую – статус возвращается. Соединение прямое, без прокси. Браузеры очищены в приватном режиме и без расширений.
Версии:
* proxmox-ve: 5.2-2 (running kernel: 4.15.18-1-pve)
* pve-manager: 5.2-5 (running version: 5.2-5/eb24855a)
* pve-kernel-4.15: 5.2-4
* pve-kernel-4.15.18-1-pve: 4.15.18-15
* pve-kernel-4.15.17-3-pve: 4.15.17-14
* corosync: 2.4.2-pve5
* criu: 2.11.1-1~bpo90
* glusterfs-client: 3.8.8-1
* ksm-control-daemon: not correctly installed
* libjs-extjs: 6.0.1-2
* libpve-access-control: 5.0-8
* libpve-apiclient-perl: 2.0-5
* libpve-common-perl: 5.0-35
* libpve-guest-common-perl: 2.0-17
* libpve-http-server-perl: 2.0-9
* libpve-storage-perl: 5.0-24
* libqb0: 1.0.1-1
* lvm2: 2.02.168-pve6
* lxc-pve: 3.0.0-3
* lxcfs: 3.0.0-1
* novnc-pve: 1.0.0-1
* proxmox-widget-toolkit: 1.0-19
* pve-cluster: 5.0-28
* pve-container: 2.0-24
* pve-docs: 5.2-4
* pve-firewall: 3.0-13
* pve-firmware: 2.0-5
* pve-ha-manager: 2.0-5
* pve-i18n: 1.0-6
* pve-libspice-server1: 0.12.8-3
* pve-qemu-kvm: 2.11.1-5
* pve-xtermjs: 1.0-5
* qemu-server: 5.0-29
* smartmontools: 6.5+svn4324-1
* spiceterm: 3.0-5
* vncterm: 1.5-3
* zfsutils-linux: 0.7.9-pve1~bpo90
Причина: При использовании `curl` возникает следующая ошибка:
```
$ curl -k -d "username=root@pam&password=......"
$ curl -v -k -b "PVE:root@pam:5B477CF0::e......"
* Trying 10.1.12.224...
* TCP_NODELAY set
* Connected to 10.1.12.224 (10.1.12.224) port 8006 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: OU=PVE Cluster Node; O=Proxmox Virtual Environment; CN=node-a
* start date: Jul 10 16:39:52 2018 GMT
* expire date: Jul 7 16:39:52 2028 GMT
* issuer: CN=Proxmox Virtual Environment; OU=eff7585c-aafd-4f32-972b-f5453b85cd1a; O=PVE Cluster Manager CA
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET /api2/json/nodes/node-b/storage/local/status HTTP/1.1
> Host: 10.1.12.224:8006
> User-Agent: curl/7.58.0
> Accept: */*
> Cookie: PVE:root@pam:5B477CF0::e......
>
< HTTP/1.1 401 No ticket
< Cache-Control: max-age=0
< Connection: close
< Date: Thu, 12 Jul 2018 16:09:32 GMT
< Pragma: no-cache
< Server: pve-api-daemon/3.0
< Expires: Thu, 12 Jul 2018 16:09:32 GMT
<
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
```
Решение: Проблема в MTU 9000. После возврата MTU на bond интерфейсах к 1500 - проблема решена.
Вывод: Я использую bonding и VLAN поверх bonding для всех сетей – LAN, межсоединение узлов, внешние сети. И MTU 9000 было установлено на всех bonding. Но MTU 9000 необходимо только VLAN для межсоединения узлов.
Версии:
* proxmox-ve: 5.2-2 (running kernel: 4.15.18-1-pve)
* pve-manager: 5.2-5 (running version: 5.2-5/eb24855a)
* pve-kernel-4.15: 5.2-4
* pve-kernel-4.15.18-1-pve: 4.15.18-15
* pve-kernel-4.15.17-3-pve: 4.15.17-14
* corosync: 2.4.2-pve5
* criu: 2.11.1-1~bpo90
* glusterfs-client: 3.8.8-1
* ksm-control-daemon: not correctly installed
* libjs-extjs: 6.0.1-2
* libpve-access-control: 5.0-8
* libpve-apiclient-perl: 2.0-5
* libpve-common-perl: 5.0-35
* libpve-guest-common-perl: 2.0-17
* libpve-http-server-perl: 2.0-9
* libpve-storage-perl: 5.0-24
* libqb0: 1.0.1-1
* lvm2: 2.02.168-pve6
* lxc-pve: 3.0.0-3
* lxcfs: 3.0.0-1
* novnc-pve: 1.0.0-1
* proxmox-widget-toolkit: 1.0-19
* pve-cluster: 5.0-28
* pve-container: 2.0-24
* pve-docs: 5.2-4
* pve-firewall: 3.0-13
* pve-firmware: 2.0-5
* pve-ha-manager: 2.0-5
* pve-i18n: 1.0-6
* pve-libspice-server1: 0.12.8-3
* pve-qemu-kvm: 2.11.1-5
* pve-xtermjs: 1.0-5
* qemu-server: 5.0-29
* smartmontools: 6.5+svn4324-1
* spiceterm: 3.0-5
* vncterm: 1.5-3
* zfsutils-linux: 0.7.9-pve1~bpo90
Причина: При использовании `curl` возникает следующая ошибка:
```
$ curl -k -d "username=root@pam&password=......"
$ curl -v -k -b "PVE:root@pam:5B477CF0::e......"
* Trying 10.1.12.224...
* TCP_NODELAY set
* Connected to 10.1.12.224 (10.1.12.224) port 8006 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: OU=PVE Cluster Node; O=Proxmox Virtual Environment; CN=node-a
* start date: Jul 10 16:39:52 2018 GMT
* expire date: Jul 7 16:39:52 2028 GMT
* issuer: CN=Proxmox Virtual Environment; OU=eff7585c-aafd-4f32-972b-f5453b85cd1a; O=PVE Cluster Manager CA
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET /api2/json/nodes/node-b/storage/local/status HTTP/1.1
> Host: 10.1.12.224:8006
> User-Agent: curl/7.58.0
> Accept: */*
> Cookie: PVE:root@pam:5B477CF0::e......
>
< HTTP/1.1 401 No ticket
< Cache-Control: max-age=0
< Connection: close
< Date: Thu, 12 Jul 2018 16:09:32 GMT
< Pragma: no-cache
< Server: pve-api-daemon/3.0
< Expires: Thu, 12 Jul 2018 16:09:32 GMT
<
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
```
Решение: Проблема в MTU 9000. После возврата MTU на bond интерфейсах к 1500 - проблема решена.
Вывод: Я использую bonding и VLAN поверх bonding для всех сетей – LAN, межсоединение узлов, внешние сети. И MTU 9000 было установлено на всех bonding. Но MTU 9000 необходимо только VLAN для межсоединения узлов.
