proxmox-mirrors/pve-docs
1
0
Fork 0
mirror of https://git.proxmox.com/git/pve-docs synced 2025-04-28 12:12:29 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

sysadmin: revise firmware chapter, add firmware repo section

Browse Source 
Chapter "Firmware Updates":
* improve the structure and clarity of information provided
* mention which update methods are when available/recommended
* add information about the already pre-installed pve-firmware package
* emphasise the importance of CPU microcode updates, how to interpret
  versions and how to recover a possibly unbootable system
* move info about non-free-firmware repo to "Package Repositories"

Chapter "Package Repositories":
* add new section "Debian Firmware Repository"

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
This commit is contained in:
Alexander Zeidler 2023-10-30 13:10:10 +01:00 committed by Thomas Lamprecht
parent c6284866ef
commit 48ae572140
2 changed files with 175 additions and 68 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

225
firmware-updates.adoc
View File

@ -4,103 +4,192 @@ Firmware Updates
ifdef::wiki[]
:pve-toplevel:
endif::wiki[]
Firmware updates from this chapter should be applied when running {pve} on a
bare-metal server. Whether configuring firmware updates is appropriate within
guests, e.g. when using device pass-through, depends strongly on your setup and
is therefore out of scope.
Regular firmware updates for devices are just as important for proper operation
as regular software updates. There are several ways to obtain and apply those
updates. The methods listed in this chapter can also be combined to minimize the
chance of missing an important update.
In addition to regular software updates, firmware updates are also important
for reliable and secure operation.
TIP: When a firmware was updated, a system reboot is the safest way to apply the
new version.
When obtaining and applying firmware updates, a combination of available options
is recommended to get them as early as possible or at all.
The term firmware is usually divided linguistically into microcode (for CPUs)
and firmware (for other devices).
[[sysadmin_firmware_persistent]]
Persistent Firmware
~~~~~~~~~~~~~~~~~~~
The following methods write the new firmware permanently to the respective
device. The firmware therefore remains up to date regardless of the booted
operating system.
This section is suitable for all devices. Updated microcode, which is usually
included in a BIOS/UEFI update, is stored on the motherboard, whereas other
firmware is stored on the respective device. This persistent method is
especially important for the CPU, as it enables the earliest possible regular
loading of the updated microcode at boot time.
TIP: When using a user space application or 'fwupd', the hardware must usually
have been manufactured after 2014, the system must have been booted with UEFI
and the EFI partition manually mounted.
CAUTION: With some updates, such as for BIOS/UEFI or storage controller, the
device configuration could be reset. Please follow the vendor's instructions
carefully and back up the current configuration.
CAUTION: When updating the BIOS/UEFI itself, its settings are usually reset. Be
prepared to reconfigure them afterwards.
Please check with your vendor which update methods are available.
* Convenient update methods for servers can include Dell's Lifecycle Manager or
Service Packs from HPE.
[[sysadmin_firmware_persistent_vendor_specific]]
Vendor-specific
^^^^^^^^^^^^^^^
Firmware updates are usually available from the vendor directly. Please check
with your vendor what options are available.
Depending on the platform and vendor, there are convenient methods available.
For servers, for example, Dell's Lifecycle Manager or Service Packs from HPE.
Sometimes there are Linux utilities available as well. Examples are
* Sometimes there are Linux utilities available as well. Examples are
https://network.nvidia.com/support/firmware/mlxup-mft/['mlxup'] for NVIDIA
ConnectX or
https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/software-installation/updating-the-firmware/manually-updating-the-adapter-firmware-on-linuxesx.html['bnxtnvm'/'niccli']
for Broadcom network cards.
* https://fwupd.org[LVFS] could also be an option if there is a cooperation with
a https://fwupd.org/lvfs/vendors/[vendor] and
https://fwupd.org/lvfs/devices/[supported hardware] in use. The technical
requirement for this is that the system was manufactured after 2014, is booted
via UEFI and the easiest way is to mount the EFI partition from which you boot
(`mount /dev/disk/by-partuuid/<from efibootmgr -v> /boot/efi`) before installing
'fwupd'.
[[sysadmin_firmware_persistent_lvfs_fwupd]]
Linux Vendor Firmware Service (LVFS) via fwupd
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
On https://fwupd.org['LVFS'], vendors can make their firmware updates available
in a standardized way to a wide range of Linux hosts. Here is the growing list
of participating https://fwupd.org/lvfs/vendors/[vendors] and their currently
supported https://fwupd.org/lvfs/devices/[devices].
To use 'fwupd', manually mount your
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot_installer_part_scheme[EFI System Partition]
(ESP) you booted from on `/boot/`. After installing the package 'fwupd', update
firmware with the following commands:
----
# fwupdmgr refresh
# fwupdmgr get-updates
# fwupdmgr update
# reboot
----
TIP: If the update instructions require a host reboot, make sure that it can be
done safely. See also xref:ha_manager_node_maintenance[Node Maintenance].
[[sysadmin_firmware_runtime_files]]
Runtime Firmware Files
~~~~~~~~~~~~~~~~~~~~~~
The following methods keep the firmware files available at the {pve} host and do
not persist it on the device itself. Whenever a device is initialized, usually
during the boot process, the corresponding firmware is loaded into the RAM of
the respective device. These methods do not provide and can not update firmware
that is used in the very early boot process (e.g. BIOS/UEFI, hard disks).
This method stores firmware on the {pve} operating system and will pass it to a
device if its xref:sysadmin_firmware_persistent[persisted firmware] is less
recent. It is supported by devices such as network and graphics cards, but not
by those that rely on persisted firmware such as the motherboard and hard disks.
In {pve} the package `pve-firmware` is already installed by default. Therefore,
with the normal system updates (APT), the included firmware of common hardware
is automatically kept up to date. Be aware that CPU microcode updates are
located in a separate Debian repository component, which is not configured by
default.
with the normal xref:system_software_updates[system updates (APT)], included
firmware of common hardware is automatically kept up to date.
An additional xref:sysadmin_debian_firmware_repo[Debian Firmware Repository]
exists, but is not configured by default.
If you try to install an additional firmware package but it conflicts, APT will
abort the installation. Perhaps the particular firmware can be obtained in
another way.
[[sysadmin_firmware_runtime_files_debian_repo]]
Debian Firmware Repository
^^^^^^^^^^^^^^^^^^^^^^^^^^
Starting with Debian Bookworm ({pve} 8) non-free firmware (as defined by
https://www.debian.org/social_contract#guidelines[DFSG]) has been moved to the
newly created Debian repository component `non-free-firmware`. It contains
firmware for CPUs (called microcode) as well as other firmware. In the past,
CPUs repeatedly had security vulnerabilities beside other issues. Using this
update method (additional) to apply microcode updates is convenient, safe and
fast.
[[sysadmin_firmware_cpu]]
CPU Microcode Updates
~~~~~~~~~~~~~~~~~~~~~
Microcode updates are intended to fix found security vulnerabilities and other
serious CPU bugs. While the CPU performance can be affected, a patched microcode
is usually still more performant than an unpatched microcode where the kernel
itself has to do mitigations. Depending on the CPU type, it is possible that
performance results of the flawed factory state can no longer be achieved
without knowingly running the CPU in an unsafe state.
To be able to install microcode updates or other firmware from the
`non-free-firmware` component, edit the file `/etc/apt/sources.list`, append
`non-free-firmware` to the end of each of the three Debian repository lines and
run `apt-get update`.
To get an overview of present CPU vulnerabilities and their mitigations, run
`lscpu`. Current real-world known vulnerabilities can only show up if the
{pve} host is xref:system_software_updates[up to date], its version not
xref:faq-support-table[end of life], and has at least been rebooted since the
last kernel update.
To keep the CPU microcode up to date, depending on the vendor, install the
package `intel-microcode` or `amd64-microcode` and reboot your {pve} host
afterwards.
Besides the recommended microcode update via
xref:sysadmin_firmware_persistent[persistent] BIOS/UEFI updates, there is also
an independent method via *Early OS Microcode Updates*. It is convenient to use
and also quite helpful when the motherboard vendor no longer provides BIOS/UEFI
updates. Regardless of the method in use, a reboot is always needed to apply a
microcode update.
Set up Early OS Microcode Updates
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
After enabling the
xref:sysadmin_debian_firmware_repo[Debian Firmware Repository], run
`apt install <intel-microcode|amd64-microcode>` and reboot the {pve} host
xref:ha_manager_node_maintenance[safely] afterwards.
Infrequent future microcode updates also require a reboot to be loaded.
Microcode Version
^^^^^^^^^^^^^^^^^
To get the current running microcode revision for comparison or debugging
purposes:
----
# grep microcode /proc/cpuinfo | uniq
microcode : 0xf0
----
Since a microcode package contains microcode for different CPU types, an updated
microcode for your CPU will only be included occasionally. Therefore, date
information from the package can also not be matched to a specific release date
from the vendor.
If the microcode package is installed, the system has been rebooted, and the
microcode included in the package is more recent than that on the motherboard
and CPU, the message "microcode updated early" appears in the log:
----
# dmesg | grep microcode
[ 0.000000] microcode: microcode updated early to revision 0xf0, date = 2021-11-12
[ 0.896580] microcode: Microcode Update Driver: v2.2.
----
[[sysadmin_firmware_troubleshooting]]
Troubleshooting
^^^^^^^^^^^^^^^
For debugging purposes, the set up Early OS Microcode Update applied regularly
at system boot can be temporarily disabled as follows:
1. make sure that the host can be rebooted xref:ha_manager_node_maintenance[safely]
2. reboot the host to get to the GRUB menu (hold `SHIFT` if it is hidden)
3. at the desired {pve} boot entry press `E`
4. go to the line which starts with `linux` and append separated by a space
*`dis_ucode_ldr`*
5. press `CTRL-X` to boot this time without an Early OS Microcode Update
If a problem related to a recent microcode update is suspected, a package
downgrade should be considered instead of package removal
(`apt purge <intel-microcode|amd64-microcode>`). Otherwise, a too old
xref:sysadmin_firmware_persistent[persisted] microcode might be loaded, even
though a more recent one would run without problems.
A downgrade is possible if an earlier microcode package version is
available in the Debian repository, as shown in this example:
----
# apt list -a intel-microcode
Listing... Done
intel-microcode/stable-security,now 3.20230808.1~deb12u1 amd64 [installed]
intel-microcode/stable 3.20230512.1 amd64
----
----
# apt install intel-microcode=3.202305*
...
Selected version '3.20230512.1' (Debian:12.1/stable [amd64]) for 'intel-microcode'
...
dpkg: warning: downgrading intel-microcode from 3.20230808.1~deb12u1 to 3.20230512.1
...
intel-microcode: microcode will be updated at next boot
...
----
Make sure (again) that the host can be rebooted
xref:ha_manager_node_maintenance[safely]. To apply an older microcode
potentially included in the microcode package for your CPU type, reboot now.
[TIP]
====
It makes sense to hold the downgraded package for a while and try more recent
versions again at a later time. Even if the package version is the same in the
future, system updates may have fixed the experienced problem in the meantime.
----
# apt-mark hold intel-microcode
intel-microcode set on hold.
----
----
# apt-mark unhold intel-microcode
# apt update
# apt upgrade
----
====

18
pve-package-repos.adoc
View File

@ -208,6 +208,24 @@ See the respective
https://pve.proxmox.com/wiki/Category:Ceph_Upgrade[upgrade guide] for details.
[[sysadmin_debian_firmware_repo]]
Debian Firmware Repository
~~~~~~~~~~~~~~~~~~~~~~~~~~
Starting with Debian Bookworm ({pve} 8) non-free firmware (as defined by
https://www.debian.org/social_contract#guidelines[DFSG]) has been moved to the
newly created Debian repository component `non-free-firmware`.
Enable this repository if you want to set up
xref:sysadmin_firmware_cpu[Early OS Microcode Updates] or need additional
xref:sysadmin_firmware_runtime_files[Runtime Firmware Files] not already
included in the pre-installed package `pve-firmware`.
To be able to install packages from this component, run
`editor /etc/apt/sources.list`, append `non-free-firmware` to the end of each
`.debian.org` repository line and run `apt update`.
[[repos_secure_apt]]
SecureApt