mirror of
https://git.proxmox.com/git/pve-docs
synced 2025-07-13 18:57:55 +00:00
sysadmin: revise firmware chapter, add firmware repo section
Chapter "Firmware Updates": * improve the structure and clarity of information provided * mention which update methods are when available/recommended * add information about the already pre-installed pve-firmware package * emphasise the importance of CPU microcode updates, how to interpret versions and how to recover a possibly unbootable system * move info about non-free-firmware repo to "Package Repositories" Chapter "Package Repositories": * add new section "Debian Firmware Repository" Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
This commit is contained in:
Alexander Zeidler
Thomas Lamprecht
parent
c6284866ef
commit
48ae572140
@ -4,103 +4,192 @@ Firmware Updates
|
|||||||
ifdef::wiki[]
|
ifdef::wiki[]
|
||||||
:pve-toplevel:
|
:pve-toplevel:
|
||||||
endif::wiki[]
|
endif::wiki[]
|
||||||
|
|
||||||
Firmware updates from this chapter should be applied when running {pve} on a
|
Firmware updates from this chapter should be applied when running {pve} on a
|
||||||
bare-metal server. Whether configuring firmware updates is appropriate within
|
bare-metal server. Whether configuring firmware updates is appropriate within
|
||||||
guests, e.g. when using device pass-through, depends strongly on your setup and
|
guests, e.g. when using device pass-through, depends strongly on your setup and
|
||||||
is therefore out of scope.
|
is therefore out of scope.
|
||||||
|
|
||||||
Regular firmware updates for devices are just as important for proper operation
|
In addition to regular software updates, firmware updates are also important
|
||||||
as regular software updates. There are several ways to obtain and apply those
|
for reliable and secure operation.
|
||||||
updates. The methods listed in this chapter can also be combined to minimize the
|
|
||||||
chance of missing an important update.
|
|
||||||
|
|
||||||
TIP: When a firmware was updated, a system reboot is the safest way to apply the
|
When obtaining and applying firmware updates, a combination of available options
|
||||||
new version.
|
is recommended to get them as early as possible or at all.
|
||||||
|
|
||||||
|
The term firmware is usually divided linguistically into microcode (for CPUs)
|
||||||
|
and firmware (for other devices).
|
||||||
|
|
||||||
|
|
||||||
[[sysadmin_firmware_persistent]]
|
[[sysadmin_firmware_persistent]]
|
||||||
Persistent Firmware
|
Persistent Firmware
|
||||||
~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~
|
||||||
The following methods write the new firmware permanently to the respective
|
This section is suitable for all devices. Updated microcode, which is usually
|
||||||
device. The firmware therefore remains up to date regardless of the booted
|
included in a BIOS/UEFI update, is stored on the motherboard, whereas other
|
||||||
operating system.
|
firmware is stored on the respective device. This persistent method is
|
||||||
|
especially important for the CPU, as it enables the earliest possible regular
|
||||||
|
loading of the updated microcode at boot time.
|
||||||
|
|
||||||
TIP: When using a user space application or 'fwupd', the hardware must usually
|
CAUTION: With some updates, such as for BIOS/UEFI or storage controller, the
|
||||||
have been manufactured after 2014, the system must have been booted with UEFI
|
device configuration could be reset. Please follow the vendor's instructions
|
||||||
and the EFI partition manually mounted.
|
carefully and back up the current configuration.
|
||||||
|
|
||||||
CAUTION: When updating the BIOS/UEFI itself, its settings are usually reset. Be
|
Please check with your vendor which update methods are available.
|
||||||
prepared to reconfigure them afterwards.
|
|
||||||
|
|
||||||
|
* Convenient update methods for servers can include Dell's Lifecycle Manager or
|
||||||
|
Service Packs from HPE.
|
||||||
|
|
||||||
[[sysadmin_firmware_persistent_vendor_specific]]
|
* Sometimes there are Linux utilities available as well. Examples are
|
||||||
Vendor-specific
|
|
||||||
^^^^^^^^^^^^^^^
|
|
||||||
Firmware updates are usually available from the vendor directly. Please check
|
|
||||||
with your vendor what options are available.
|
|
||||||
|
|
||||||
Depending on the platform and vendor, there are convenient methods available.
|
|
||||||
For servers, for example, Dell's Lifecycle Manager or Service Packs from HPE.
|
|
||||||
Sometimes there are Linux utilities available as well. Examples are
|
|
||||||
https://network.nvidia.com/support/firmware/mlxup-mft/['mlxup'] for NVIDIA
|
https://network.nvidia.com/support/firmware/mlxup-mft/['mlxup'] for NVIDIA
|
||||||
ConnectX or
|
ConnectX or
|
||||||
https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/software-installation/updating-the-firmware/manually-updating-the-adapter-firmware-on-linuxesx.html['bnxtnvm'/'niccli']
|
https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/software-installation/updating-the-firmware/manually-updating-the-adapter-firmware-on-linuxesx.html['bnxtnvm'/'niccli']
|
||||||
for Broadcom network cards.
|
for Broadcom network cards.
|
||||||
|
|
||||||
|
* https://fwupd.org[LVFS] could also be an option if there is a cooperation with
|
||||||
|
a https://fwupd.org/lvfs/vendors/[vendor] and
|
||||||
|
https://fwupd.org/lvfs/devices/[supported hardware] in use. The technical
|
||||||
|
requirement for this is that the system was manufactured after 2014, is booted
|
||||||
|
via UEFI and the easiest way is to mount the EFI partition from which you boot
|
||||||
|
(`mount /dev/disk/by-partuuid/<from efibootmgr -v> /boot/efi`) before installing
|
||||||
|
'fwupd'.
|
||||||
|
|
||||||
[[sysadmin_firmware_persistent_lvfs_fwupd]]
|
TIP: If the update instructions require a host reboot, make sure that it can be
|
||||||
Linux Vendor Firmware Service (LVFS) via fwupd
|
done safely. See also xref:ha_manager_node_maintenance[Node Maintenance].
|
||||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
||||||
On https://fwupd.org['LVFS'], vendors can make their firmware updates available
|
|
||||||
in a standardized way to a wide range of Linux hosts. Here is the growing list
|
|
||||||
of participating https://fwupd.org/lvfs/vendors/[vendors] and their currently
|
|
||||||
supported https://fwupd.org/lvfs/devices/[devices].
|
|
||||||
|
|
||||||
To use 'fwupd', manually mount your
|
|
||||||
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot_installer_part_scheme[EFI System Partition]
|
|
||||||
(ESP) you booted from on `/boot/`. After installing the package 'fwupd', update
|
|
||||||
firmware with the following commands:
|
|
||||||
----
|
|
||||||
# fwupdmgr refresh
|
|
||||||
# fwupdmgr get-updates
|
|
||||||
# fwupdmgr update
|
|
||||||
# reboot
|
|
||||||
----
|
|
||||||
|
|
||||||
|
|
||||||
[[sysadmin_firmware_runtime_files]]
|
[[sysadmin_firmware_runtime_files]]
|
||||||
Runtime Firmware Files
|
Runtime Firmware Files
|
||||||
~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~
|
||||||
The following methods keep the firmware files available at the {pve} host and do
|
This method stores firmware on the {pve} operating system and will pass it to a
|
||||||
not persist it on the device itself. Whenever a device is initialized, usually
|
device if its xref:sysadmin_firmware_persistent[persisted firmware] is less
|
||||||
during the boot process, the corresponding firmware is loaded into the RAM of
|
recent. It is supported by devices such as network and graphics cards, but not
|
||||||
the respective device. These methods do not provide and can not update firmware
|
by those that rely on persisted firmware such as the motherboard and hard disks.
|
||||||
that is used in the very early boot process (e.g. BIOS/UEFI, hard disks).
|
|
||||||
|
|
||||||
In {pve} the package `pve-firmware` is already installed by default. Therefore,
|
In {pve} the package `pve-firmware` is already installed by default. Therefore,
|
||||||
with the normal system updates (APT), the included firmware of common hardware
|
with the normal xref:system_software_updates[system updates (APT)], included
|
||||||
is automatically kept up to date. Be aware that CPU microcode updates are
|
firmware of common hardware is automatically kept up to date.
|
||||||
located in a separate Debian repository component, which is not configured by
|
|
||||||
default.
|
An additional xref:sysadmin_debian_firmware_repo[Debian Firmware Repository]
|
||||||
|
exists, but is not configured by default.
|
||||||
|
|
||||||
|
If you try to install an additional firmware package but it conflicts, APT will
|
||||||
|
abort the installation. Perhaps the particular firmware can be obtained in
|
||||||
|
another way.
|
||||||
|
|
||||||
|
|
||||||
[[sysadmin_firmware_runtime_files_debian_repo]]
|
[[sysadmin_firmware_cpu]]
|
||||||
Debian Firmware Repository
|
CPU Microcode Updates
|
||||||
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
~~~~~~~~~~~~~~~~~~~~~
|
||||||
Starting with Debian Bookworm ({pve} 8) non-free firmware (as defined by
|
Microcode updates are intended to fix found security vulnerabilities and other
|
||||||
https://www.debian.org/social_contract#guidelines[DFSG]) has been moved to the
|
serious CPU bugs. While the CPU performance can be affected, a patched microcode
|
||||||
newly created Debian repository component `non-free-firmware`. It contains
|
is usually still more performant than an unpatched microcode where the kernel
|
||||||
firmware for CPUs (called microcode) as well as other firmware. In the past,
|
itself has to do mitigations. Depending on the CPU type, it is possible that
|
||||||
CPUs repeatedly had security vulnerabilities beside other issues. Using this
|
performance results of the flawed factory state can no longer be achieved
|
||||||
update method (additional) to apply microcode updates is convenient, safe and
|
without knowingly running the CPU in an unsafe state.
|
||||||
fast.
|
|
||||||
|
|
||||||
To be able to install microcode updates or other firmware from the
|
To get an overview of present CPU vulnerabilities and their mitigations, run
|
||||||
`non-free-firmware` component, edit the file `/etc/apt/sources.list`, append
|
`lscpu`. Current real-world known vulnerabilities can only show up if the
|
||||||
`non-free-firmware` to the end of each of the three Debian repository lines and
|
{pve} host is xref:system_software_updates[up to date], its version not
|
||||||
run `apt-get update`.
|
xref:faq-support-table[end of life], and has at least been rebooted since the
|
||||||
|
last kernel update.
|
||||||
|
|
||||||
To keep the CPU microcode up to date, depending on the vendor, install the
|
Besides the recommended microcode update via
|
||||||
package `intel-microcode` or `amd64-microcode` and reboot your {pve} host
|
xref:sysadmin_firmware_persistent[persistent] BIOS/UEFI updates, there is also
|
||||||
afterwards.
|
an independent method via *Early OS Microcode Updates*. It is convenient to use
|
||||||
|
and also quite helpful when the motherboard vendor no longer provides BIOS/UEFI
|
||||||
|
updates. Regardless of the method in use, a reboot is always needed to apply a
|
||||||
|
microcode update.
|
||||||
|
|
||||||
|
|
||||||
|
Set up Early OS Microcode Updates
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
After enabling the
|
||||||
|
xref:sysadmin_debian_firmware_repo[Debian Firmware Repository], run
|
||||||
|
`apt install <intel-microcode|amd64-microcode>` and reboot the {pve} host
|
||||||
|
xref:ha_manager_node_maintenance[safely] afterwards.
|
||||||
|
|
||||||
|
Infrequent future microcode updates also require a reboot to be loaded.
|
||||||
|
|
||||||
|
|
||||||
|
Microcode Version
|
||||||
|
^^^^^^^^^^^^^^^^^
|
||||||
|
To get the current running microcode revision for comparison or debugging
|
||||||
|
purposes:
|
||||||
|
|
||||||
|
----
|
||||||
|
# grep microcode /proc/cpuinfo | uniq
|
||||||
|
microcode : 0xf0
|
||||||
|
----
|
||||||
|
|
||||||
|
Since a microcode package contains microcode for different CPU types, an updated
|
||||||
|
microcode for your CPU will only be included occasionally. Therefore, date
|
||||||
|
information from the package can also not be matched to a specific release date
|
||||||
|
from the vendor.
|
||||||
|
|
||||||
|
If the microcode package is installed, the system has been rebooted, and the
|
||||||
|
microcode included in the package is more recent than that on the motherboard
|
||||||
|
and CPU, the message "microcode updated early" appears in the log:
|
||||||
|
|
||||||
|
----
|
||||||
|
# dmesg | grep microcode
|
||||||
|
[ 0.000000] microcode: microcode updated early to revision 0xf0, date = 2021-11-12
|
||||||
|
[ 0.896580] microcode: Microcode Update Driver: v2.2.
|
||||||
|
----
|
||||||
|
|
||||||
|
|
||||||
|
[[sysadmin_firmware_troubleshooting]]
|
||||||
|
Troubleshooting
|
||||||
|
^^^^^^^^^^^^^^^
|
||||||
|
For debugging purposes, the set up Early OS Microcode Update applied regularly
|
||||||
|
at system boot can be temporarily disabled as follows:
|
||||||
|
|
||||||
|
1. make sure that the host can be rebooted xref:ha_manager_node_maintenance[safely]
|
||||||
|
2. reboot the host to get to the GRUB menu (hold `SHIFT` if it is hidden)
|
||||||
|
3. at the desired {pve} boot entry press `E`
|
||||||
|
4. go to the line which starts with `linux` and append separated by a space
|
||||||
|
*`dis_ucode_ldr`*
|
||||||
|
5. press `CTRL-X` to boot this time without an Early OS Microcode Update
|
||||||
|
|
||||||
|
If a problem related to a recent microcode update is suspected, a package
|
||||||
|
downgrade should be considered instead of package removal
|
||||||
|
(`apt purge <intel-microcode|amd64-microcode>`). Otherwise, a too old
|
||||||
|
xref:sysadmin_firmware_persistent[persisted] microcode might be loaded, even
|
||||||
|
though a more recent one would run without problems.
|
||||||
|
|
||||||
|
A downgrade is possible if an earlier microcode package version is
|
||||||
|
available in the Debian repository, as shown in this example:
|
||||||
|
|
||||||
|
----
|
||||||
|
# apt list -a intel-microcode
|
||||||
|
Listing... Done
|
||||||
|
intel-microcode/stable-security,now 3.20230808.1~deb12u1 amd64 [installed]
|
||||||
|
intel-microcode/stable 3.20230512.1 amd64
|
||||||
|
----
|
||||||
|
----
|
||||||
|
# apt install intel-microcode=3.202305*
|
||||||
|
...
|
||||||
|
Selected version '3.20230512.1' (Debian:12.1/stable [amd64]) for 'intel-microcode'
|
||||||
|
...
|
||||||
|
dpkg: warning: downgrading intel-microcode from 3.20230808.1~deb12u1 to 3.20230512.1
|
||||||
|
...
|
||||||
|
intel-microcode: microcode will be updated at next boot
|
||||||
|
...
|
||||||
|
----
|
||||||
|
|
||||||
|
Make sure (again) that the host can be rebooted
|
||||||
|
xref:ha_manager_node_maintenance[safely]. To apply an older microcode
|
||||||
|
potentially included in the microcode package for your CPU type, reboot now.
|
||||||
|
|
||||||
|
[TIP]
|
||||||
|
====
|
||||||
|
It makes sense to hold the downgraded package for a while and try more recent
|
||||||
|
versions again at a later time. Even if the package version is the same in the
|
||||||
|
future, system updates may have fixed the experienced problem in the meantime.
|
||||||
|
----
|
||||||
|
# apt-mark hold intel-microcode
|
||||||
|
intel-microcode set on hold.
|
||||||
|
----
|
||||||
|
----
|
||||||
|
# apt-mark unhold intel-microcode
|
||||||
|
# apt update
|
||||||
|
# apt upgrade
|
||||||
|
----
|
||||||
|
====
|
||||||
|
|||||||
@ -208,6 +208,24 @@ See the respective
|
|||||||
https://pve.proxmox.com/wiki/Category:Ceph_Upgrade[upgrade guide] for details.
|
https://pve.proxmox.com/wiki/Category:Ceph_Upgrade[upgrade guide] for details.
|
||||||
|
|
||||||
|
|
||||||
|
[[sysadmin_debian_firmware_repo]]
|
||||||
|
Debian Firmware Repository
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Starting with Debian Bookworm ({pve} 8) non-free firmware (as defined by
|
||||||
|
https://www.debian.org/social_contract#guidelines[DFSG]) has been moved to the
|
||||||
|
newly created Debian repository component `non-free-firmware`.
|
||||||
|
|
||||||
|
Enable this repository if you want to set up
|
||||||
|
xref:sysadmin_firmware_cpu[Early OS Microcode Updates] or need additional
|
||||||
|
xref:sysadmin_firmware_runtime_files[Runtime Firmware Files] not already
|
||||||
|
included in the pre-installed package `pve-firmware`.
|
||||||
|
|
||||||
|
To be able to install packages from this component, run
|
||||||
|
`editor /etc/apt/sources.list`, append `non-free-firmware` to the end of each
|
||||||
|
`.debian.org` repository line and run `apt update`.
|
||||||
|
|
||||||
|
|
||||||
[[repos_secure_apt]]
|
[[repos_secure_apt]]
|
||||||
|
|
||||||
SecureApt
|
SecureApt
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user