+7 (495) 320-70-49
- Назад
- Телефоны
- +7 (495) 320-70-49
- Заказать звонок
info@proxmox.su
Москва, Бакунинская улица, 69с1
Пн-Пт: 09-00 до 18-00
Сб-Вс: выходной

Ядро 6.5.11 — ошибка с сетевой картой HP 631FLR-SFP28., Proxmox Виртуальная Среда

dominiaz

Guest

25.12.2023 19:44:00

У меня есть сетевой контроллер Ethernet 5d:00.0: Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller (rev 01) на HP DL380G10. После свежей установки Proxmox 8.1 появилась такая ошибка. Я попытался обновить ядро до версии 6.5.11-7-pve с помощью apt, но ошибка сохраняется:

```
[ 5.123860] ================================================================================
[ 5.123862] UBSAN: shift-out-of-bounds in ./include/linux/log2.h:57:13
[ 5.123865] shift exponent 64 is too large for 64-bit type 'long unsigned int'
[ 5.123867] CPU: 60 PID: 935 Comm: (udev-worker) Tainted: P O 6.5.11-7-pve #1
[ 5.123871] Hardware name: HPE ProLiant DL380 Gen10/ProLiant DL380 Gen10, BIOS U30 07/20/2023
[ 5.123872] Call Trace:
[ 5.123875] <TASK>
[ 5.123878] dump_stack_lvl+0x48/0x70
[ 5.123892] dump_stack+0x10/0x20
[ 5.123894] ...
[ 5.124656] ================================================================================
[ 5.125035] bnxt_en 0000:5d:00.0: QPLIB: cmdq[0xe]=0x3 status 0x3
[ 5.125042] bnxt_en 0000:5d:00.0 bnxt_re0: Failed to modify HW QP
[ 5.125045] infiniband bnxt_re0: Couldn't change QP1 state to INIT: -14
[ 5.125048] infiniband bnxt_re0: Couldn't start port
[ 5.125371] infiniband bnxt_re0: Couldn't open port 1
[ 5.125654] infiniband bnxt_re0: Device registered with IB successfully
```

Я скатил ядро до версии 6.2.16-20-pve, и ошибка пропала, так что проблема в ядре 6.5.11. Буду признателен за любую помощь!

confusing-charger

Guest

30.04.2024 15:22:00

У меня та же проблема. В моём случае, это, вероятно, связано с контроллером Ethernet: Broadcom Inc. и дочерние компании BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01). Также установлены 4 контроллера Ethernet: Broadcom Inc. и дочерние компании BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller (rev 01). Сегодня я обновился до PVE 8.2. С ядром 6.8.4-2-pve сервер больше не загружается корректно и зависает на каком-то этапе загрузки. Сейчас использую старое 6.5.13-5-pve, которое все еще работает. Требуется дальнейшая отладка и тестирование. Код:
kernel: bnxt_en 0000:c6:00.0 bnxt_re0: Failed to modify HW QP
kernel: infiniband bnxt_re0: Couldn't change QP1 state to INIT: -110
kernel: infiniband bnxt_re0: Couldn't start port
kernel: bnxt_en 0000:c6:00.0 bnxt_re0: Failed to destroy HW QP

# lspci -v
c6:00.0 Ethernet controller: Broadcom Inc. и дочерние компании BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)
DeviceName: Broadcom 10G Ethernet #1
Subsystem: Super Micro Computer Inc BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller
Flags: bus master, fast devsel, latency 0, IRQ 57, IOMMU group 18
Memory at 18020f10000 (64-bit, prefetchable) [size=64K]
Memory at 18020e00000 (64-bit, prefetchable) [size=1M]
Memory at 18020f22000 (64-bit, prefetchable) [size=8K]
Expansion ROM at b9440000 [disabled] [size=256K]
Capabilities: [48] Power Management version 3
Capabilities: [a0] MSI-X: Enable+ Count=255 Masked-
Capabilities: [ac] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [13c] Device Serial Number 3c-ec-ef-ff-fe-9e-74-04
Capabilities: [150] Power Budgeting <?>
Capabilities: [160] Virtual Channel
Capabilities: [180] Vendor Specific Information: ID=0000 Rev=0 Len=020 <?>
Capabilities: [1b0] Latency Tolerance Reporting
Capabilities: [1b8] Alternative Routing-ID Interpretation (ARI)
Capabilities: [230] Transaction Processing Hints
Capabilities: [300] Secondary PCI Express
Capabilities: [200] Precision Time Measurement
Kernel driver in use: bnxt_en
Kernel modules: bnxt_en

# ethtool -i eno1np0
driver: bnxt_en
version: 6.5.13-5-pve
firmware-version: 224.0.158.0/pkg 224.1.102.0
expansion-rom-version:
bus-info: 0000:c6:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
Edit*: Могу подтвердить, что это работает с blacklist. Сервер успешно загрузился с 6.8.4-2-pve. Код: # cat /etc/modprobe.d/pve-blacklist.conf
blacklist bnxt_re

# ethtool -i eno1np0
driver: bnxt_en
version: 6.8.4-2-pve
firmware-version: 224.0.158.0/pkg 224.1.102.0
expansion-rom-version:
bus-info: 0000:c6:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

eracerxrs

Guest

13.05.2024 21:37:00

Только что наткнулся на эту проблему на 6.8.4-3-pve. Чёрный список работает. # ethtool -i enp193s0f1np1 driver: bnxt_en version: 6.8.4-3-pve firmware-version: 220.0.59.0/pkg 220.0.83.0 expansion-rom-version: bus-info: 0000:c1:00.1 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: no Моя прошивка довольно старая, но она работала на предыдущем ядре 6.5. Кто-нибудь смог обновить прошивку сетевой карты от Supermicro?

antonin.chadima Guest	#4 0 14.05.2024 09:03:00 Скачай свежую прошивку сетевой карты от Broadcom.

BKS-IT Guest	#5 0 24.07.2024 12:36:00 У меня тоже была эта ошибка, оказалось, нужно просто скачать и установить утилиту для прошивки (она скачивает прошивку для сетевой карты). После этого проблема решилась.

dodancs

Guest

04.04.2025 21:30:00

Привет всем! У меня тоже возникают похожие проблемы. Это на Supermicro AS-2015SV-WTNRT, с Broadcom BCM57416 10G ethernet PHYs:

Код: 05:00.0 Ethernet controller: Broadcom Inc. и дочерние компании BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)
DeviceName: Broadcom Ethernet BCM57416/5720L #1
Subsystem: Super Micro Computer Inc BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 55
NUMA node: 0
Region 0: Memory at 30020e10000 (64-bit, prefetchable) [size=64K]
Region 2: Memory at 30020d00000 (64-bit, prefetchable) [size=1M]
Region 4: Memory at 30020e22000 (64-bit, prefetchable) [size=8K]
Expansion ROM at f0440000 [disabled] [size=256K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [a0] MSI-X: Enable+ Count=255 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00000ff0
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 512 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr+ TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4 (downgraded)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp- 10BitTagReq- OBFF Via WAKE#, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Via WAKE#,
AtomicOpsCtl: ReqEn+
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO+ CmpltAbrt- UnxCmplt+ RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 04000001 0000200f 05020000 00000000
Capabilities: [13c v1] Device Serial Number 7c-c2-55-ff-fe-87-77-9c
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [160 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Status: InProgress-
Capabilities: [1b0 v1] Latency Tolerance Reporting
Max snoop latency: 1048576ns
Max no snoop latency: 1048576ns
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [230 v1] Transaction Processing Hints
Interrupt vector mode supported
Device specific mode supported
Steering table in MSI-X table
Capabilities: [300 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [200 v1] Precision Time Measurement
PTMCap: Requester:+ Responder:- Root:-
PTMClockGranularity: Unimplemented
PTMControl: Enabled:- RootSelected:-
PTMEffectiveGranularity: Unknown
Kernel driver in use: bnxt_en
Kernel modules: bnxt_en Firmware version: Code: driver: bnxt_en
version: 6.8.0-57-generic
firmware-version: 226.0.145.0/pkg 226.1.107.1
expansion-rom-version:
bus-info: 0000:05:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no Dmesg logs attached.

RaHu Guest	#7 0 16.04.2025 18:40:00 Тут та же самая ситуация, тоже Supermicro, похожая прошивка. Началось это после обновления хоста с версии 8.3.0 до 8.4.1.

Titin Guest	#8 0 03.02.2024 14:16:00 Привет, проблема, похоже, все еще существует и связана с использованием bonding, смотри: https://bugzilla.redhat.com/show_bug.cgi?id=2020532

antonin.chadima Guest	#9 0 12.02.2024 14:43:00 Решено: проблема заключается не в каком-либо взаимодействии, а в версии ядра + модуле bnxt_re + прошивке Broadcom + инициализации RDMA RoCE. У меня была та же проблема. В моем стойке всего 25 сетевых карт Broadcom BCM57504 4x25G SFP28 PCIe. У большинства нет этой проблемы, но у некоторых есть. Я выяснил, что проблема вызвана более старой прошивкой, которая заставляет ядро думать, что RDMA включен на карте. После этого, из-за этого, и только на серверах со старой прошивкой, дополнительно к стандартному bnxt_en загружается модуль bnxt_re. Этот модуль на картах с включенным RDMA или старой прошивкой вылетает. Он загружается нормально на серверах с новой прошивкой, но при этом RDMA-функции на этих картах не включаются – значит, ошибка не срабатывает. Решение – занести bnxt_re в черный список в /etc/modprobe.d/pve-blacklist.conf или обновить прошивку и сбросить прошивку до заводских настроек (это невозможно при новой версии ядра, мне пришлось использовать RHEL с ядром 5.14). Пример старой прошивки, вызывающей проблему: ``` #lspci -v 01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet (rev 11) Subsystem: Broadcom Inc. and subsidiaries BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet (NetXtreme-E Quad-port 25G SFP28 Ethernet PCIe4.0 x16 Adapter (BCM957504-P425G)) Flags: bus master, fast devsel, latency 0, IRQ 150 Memory at f8f30000 (64-bit, non-prefetchable) [size=64K] Memory at 380b0000000 (64-bit, prefetchable) [size=1M] Memory at 380b0106000 (64-bit, prefetchable) [size=8K] Expansion ROM at f8e80000 [disabled] [size=512K] Capabilities: [48] Power Management version 3 Capabilities: [50] Vital Product Data Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+ Capabilities: [a0] MSI-X: Enable+ Count=74 Masked- Capabilities: [ac] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [13c] Device Serial Number 5c-6f-69-ff-fe-88-ec-b0 Capabilities: [150] Power Budgeting <?> Capabilities: [160] Virtual Channel Capabilities: [180] Vendor Specific Information: ID=0000 Rev=0 Len=020 <?> Capabilities: [1b0] Latency Tolerance Reporting Capabilities: [1b8] Alternative Routing-ID Interpretation (ARI) Capabilities: [230] Transaction Processing Hints Capabilities: [300] Secondary PCI Express Capabilities: [200] Precision Time Measurement Capabilities: [358] Physical Layer 16.0 GT/s <?> Capabilities: [388] Lane Margining at the Receiver <?> Kernel driver in use: bnxt_en Kernel modules: bnxt_en ``` ``` #ethtool -i enp1s0f0np0 driver: bnxt_en version: 6.5.11-8-pve firmware-version: 216.16.8/pkg 21.0.333.11 expansion-rom-version: bus-info: 0000:01:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: no ``` ``` #rmmod bnxt_re && modprobe bnxt_re [ 4794.513390] bnxt_re: Broadcom NetXtreme-C/E RoCE Driver [ 4794.514285] infiniband (null): Low latency framework is enabled [ 4794.514539] bnxt_re: firmware revision 21.85.21.91 [ 4794.514553] bnxt_re: device 0000:01:00.0: 24 port channels [ 4794.514577] bnxt_re: device 0000:01:00.0: loaded [ 4794.515276] bnxt_re: firmware revision 21.85.21.91 [ 4794.515427] bnxt_re: device 0000:01:00.0: 24 port channels [ 4794.515443] bnxt_re: device 0000:01:00.0: loaded [ 4794.515510] bnxt_re: firmware revision 21.85.21.91 [ 4794.515543] bnxt_re: device 0000:01:00.0: 24 port channels [ 4794.515573] bnxt_re: device 0000:01:00.0: loaded [ 4794.515597] bnxt_re: firmware revision 21.85.21.91 [ 4794.515620] bnxt_re: device 0000:01:00.0: 24 port channels [ 4794.515642] bnxt_re: device 0000:01:00.0: loaded [ 4794.515668] bnxt_re: firmware revision 21.85.21.91 [ 4794.515693] bnxt_re: device 0000:01:00.0: 24 port channels [ 4794.515680] bnxt_re: device 0000:01:00.0: loaded [ 4794.515705] bnxt_re: firmware revision 21.85.21.91 [ 4794.516072] bnxt_re: device 0000:01:00.0: 24 port channels [ 4794.516092] bnxt_re: device 0000:01:00.0: loaded [ 4794.516090] bnxt_re: firmware revision 21.85.21.91 [ 4794.516520] bnxt_re: device 0000:01:00.0: 24 port channels [ 4794.516567] bnxt_re: device 0000:01:00.0: loaded [ 4794.516589] bnxt_re: firmware revision 21.85.21.91 [ 4794.517573] bnxt_re: device 0000:01:00.0: 24 port channels [ 4794.517594] bnxt_re: device 0000:01:00.0: loaded [ 4794.517614] bnxt_re: firmware revision 21.85.21.91 [ 4794.517644] bnxt_re: device 0000:01:00.0: 24 port channels [ 4794.517666] bnxt_re: device 0000:01:00.0: loaded [ 4794.517686] bnxt_re: firmware revision 21.85.21.91 [ 4794.517714] bnxt_re: device 0000:01:00.0: 24 port channels [ 4794.517736] bnxt_re: device 0000:01:00.0: loaded [ 4794.517758] bnxt_re: firmware revision 21.85.21.91 [ 4794.517786] bnxt_re: device 0000:01:00.0: 24 port channels [ 4794.517807] bnxt_re: device 0000:01:00.0: loaded [ 4794.517829] bnxt_re: firmware revision 21.85.21.91 [ 4794.517859] bnxt_re: device 0000:01:00.0: 24 port channels [ 4794.517881] bnxt_re: device 0000:01:00.0: loaded ``` ``` #ibv_devices device node GUID ------ ---------------- bnxt_re0 5e6f69fffe88ecb0 bnxt_re1 5e6f69fffe88ecb1 bnxt_re2 5e6f69fffe88ecb2 bnxt_re3 5e6f69fffe88ecb3 ``` Пример новой прошивки без проблем: ``` #lspci -v 41:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet (rev 11) Subsystem: Broadcom Inc. and subsidiaries NetXtreme-E P425D BCM57504 4x25G SFP28 PCIE Flags: bus master, fast devsel, latency 0, IRQ 54 Memory at 28080130000 (64-bit, prefetchable) [size=64K] Memory at 28080000000 (64-bit, prefetchable) [size=1M] Memory at 28080158000 (64-bit, prefetchable) [size=32K] Expansion ROM at f26c0000 [disabled] [size=256K] Capabilities: [48] Power Management version 3 Capabilities: [50] Vital Product Data Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+ Capabilities: [a0] MSI-X: Enable+ Count=74 Masked- Capabilities: [ac] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [13c] Device Serial Number 5c-6f-69-ff-fe-cf-93-00 Capabilities: [150] Power Budgeting <?> Capabilities: [160] Virtual Channel Capabilities: [180] Vendor Specific Information: ID=0000 Rev=0 Len=020 <?> Capabilities: [1b0] Latency Tolerance Reporting Capabilities: [1b8] Alternative Routing-ID Interpretation (ARI) Capabilities: [230] Transaction Processing Hints Capabilities: [300] Secondary PCI Express Capabilities: [200] Precision Time Measurement Capabilities: [358] Physical Layer 16.0 GT/s <?> Capabilities: [388] Lane Margining at the Receiver <?> Kernel driver in use: bnxt_en Kernel modules: bnxt_en ``` ``` #ethtool -i enp65s0f0np0 driver: bnxt_en version: 6.5.11-8-pve firmware-version: 2185.21.91 ``` Не используйте RDMA на этой машине! Я надеюсь, что эта информация поможет.

Читают тему

Главная Каталог 0 Корзина 0 Избранные Кабинет 0 Сравнение Акции Контакты Услуги Бренды Отзывы Компания Лицензии Документы Реквизиты Поиск Блог Обзоры