mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/chenhuacai/linux-loongson
synced 2025-08-27 06:50:37 +00:00
ee4f032455
Update the Documentation/edac/scrub.rst to include use cases and policies for CXL memory device-based, CXL region-based patrol scrub control and CXL Error Check Scrub (ECS). Reviewed-by: Dave Jiang <dave.jiang@intel.com> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com> Signed-off-by: Shiju Jose <shiju.jose@huawei.com> Reviewed-by: Fan Ni <fan.ni@samsung.com> Reviewed-by: Alison Schofield <alison.schofield@intel.com> Acked-by: Dan Williams <dan.j.williams@intel.com> Link: https://patch.msgid.link/20250521124749.817-2-shiju.jose@huawei.com Signed-off-by: Dave Jiang <dave.jiang@intel.com>
343 lines
15 KiB
ReStructuredText
343 lines
15 KiB
ReStructuredText
.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.2-no-invariants-or-later
|
|
|
|
=============
|
|
Scrub Control
|
|
=============
|
|
|
|
Copyright (c) 2024-2025 HiSilicon Limited.
|
|
|
|
:Author: Shiju Jose <shiju.jose@huawei.com>
|
|
:License: The GNU Free Documentation License, Version 1.2 without
|
|
Invariant Sections, Front-Cover Texts nor Back-Cover Texts.
|
|
(dual licensed under the GPL v2)
|
|
|
|
- Written for: 6.15
|
|
|
|
Introduction
|
|
------------
|
|
|
|
Increasing DRAM size and cost have made memory subsystem reliability an
|
|
important concern. These modules are used where potentially corrupted data
|
|
could cause expensive or fatal issues. Memory errors are among the top
|
|
hardware failures that cause server and workload crashes.
|
|
|
|
Memory scrubbing is a feature where an ECC (Error-Correcting Code) engine
|
|
reads data from each memory media location, corrects if necessary and writes
|
|
the corrected data back to the same memory media location.
|
|
|
|
DIMMs can be scrubbed at a configurable rate to detect uncorrected memory
|
|
errors and attempt recovery from detected errors, providing the following
|
|
benefits:
|
|
|
|
1. Proactively scrubbing DIMMs reduces the chance of a correctable error
|
|
becoming uncorrectable.
|
|
|
|
2. When detected, uncorrected errors caught in unallocated memory pages are
|
|
isolated and prevented from being allocated to an application or the OS.
|
|
|
|
3. This reduces the likelihood of software or hardware products encountering
|
|
memory errors.
|
|
|
|
4. The additional data on failures in memory may be used to build up
|
|
statistics that are later used to decide whether to use memory repair
|
|
technologies such as Post Package Repair or Sparing.
|
|
|
|
There are 2 types of memory scrubbing:
|
|
|
|
1. Background (patrol) scrubbing while the DRAM is otherwise idle.
|
|
|
|
2. On-demand scrubbing for a specific address range or region of memory.
|
|
|
|
Several types of interfaces to hardware memory scrubbers have been
|
|
identified, such as CXL memory device patrol scrub, CXL DDR5 ECS, ACPI
|
|
RAS2 memory scrubbing, and ACPI NVDIMM ARS (Address Range Scrub).
|
|
|
|
The control mechanisms vary across different memory scrubbers. To enable
|
|
standardized userspace tooling, there is a need to present these controls
|
|
through a standardized ABI.
|
|
|
|
A generic memory EDAC scrub control allows users to manage underlying
|
|
scrubbers in the system through a standardized sysfs control interface. It
|
|
abstracts the management of various scrubbing functionalities into a unified
|
|
set of functions.
|
|
|
|
Use cases of common scrub control feature
|
|
-----------------------------------------
|
|
|
|
1. Several types of interfaces for hardware memory scrubbers have been
|
|
identified, including the CXL memory device patrol scrub, CXL DDR5 ECS,
|
|
ACPI RAS2 memory scrubbing features, ACPI NVDIMM ARS (Address Range Scrub),
|
|
and software-based memory scrubbers.
|
|
|
|
Of the identified interfaces to hardware memory scrubbers some support
|
|
control over patrol (background) scrubbing (e.g., ACPI RAS2, CXL) and/or
|
|
on-demand scrubbing (e.g., ACPI RAS2, ACPI ARS). However, the scrub control
|
|
interfaces vary between memory scrubbers, highlighting the need for
|
|
a standardized, generic sysfs scrub control interface that is accessible to
|
|
userspace for administration and use by scripts/tools.
|
|
|
|
2. User-space scrub controls allow users to disable scrubbing if necessary,
|
|
for example, to disable background patrol scrubbing or adjust the scrub
|
|
rate for performance-aware operations where background activities need to
|
|
be minimized or disabled.
|
|
|
|
3. User-space tools enable on-demand scrubbing for specific address ranges,
|
|
provided that the scrubber supports this functionality.
|
|
|
|
4. User-space tools can also control memory DIMM scrubbing at a configurable
|
|
scrub rate via sysfs scrub controls. This approach offers several benefits:
|
|
|
|
4.1. Detects uncorrectable memory errors early, before user access to affected
|
|
memory, helping facilitate recovery.
|
|
|
|
4.2. Reduces the likelihood of correctable errors developing into uncorrectable
|
|
errors.
|
|
|
|
5. Policy control for hotplugged memory is necessary because there may not
|
|
be a system-wide BIOS or similar control to manage scrub settings for a CXL
|
|
device added after boot. Determining these settings is a policy decision,
|
|
balancing reliability against performance, so userspace should control it.
|
|
Therefore, a unified interface is recommended for handling this function in
|
|
a way that aligns with other similar interfaces, rather than creating a
|
|
separate one.
|
|
|
|
Scrubbing features
|
|
------------------
|
|
|
|
CXL Memory Scrubbing features
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
CXL spec r3.1 [1]_ section 8.2.9.9.11.1 describes the memory device patrol
|
|
scrub control feature. The device patrol scrub proactively locates and makes
|
|
corrections to errors in regular cycle. The patrol scrub control allows the
|
|
userspace request to change CXL patrol scrubber's configurations.
|
|
|
|
The patrol scrub control allows the requester to specify the number of
|
|
hours in which the patrol scrub cycles must be completed, provided that
|
|
the requested scrub rate must be within the supported range of the
|
|
scrub rate that the device is capable of. In the CXL driver, the
|
|
number of seconds per scrub cycles, which user requests via sysfs, is
|
|
rescaled to hours per scrub cycles.
|
|
|
|
In addition, they allow the host to disable the feature in case it interferes
|
|
with performance-aware operations which require the background operations to
|
|
be turned off.
|
|
|
|
Error Check Scrub (ECS)
|
|
~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
CXL spec r3.1 [1]_ section 8.2.9.9.11.2 describes Error Check Scrub (ECS)
|
|
- a feature defined in the JEDEC DDR5 SDRAM Specification (JESD79-5) and
|
|
allowing DRAM to internally read, correct single-bit errors, and write back
|
|
corrected data bits to the DRAM array while providing transparency to error
|
|
counts.
|
|
|
|
The DDR5 device contains number of memory media Field Replaceable Units (FRU)
|
|
per device. The DDR5 ECS feature and thus the ECS control driver supports
|
|
configuring the ECS parameters per FRU.
|
|
|
|
ACPI RAS2 Hardware-based Memory Scrubbing
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
ACPI spec 6.5 [2]_ section 5.2.21 ACPI RAS2 describes an ACPI RAS2 table
|
|
which provides interfaces for platform RAS features and supports independent
|
|
RAS controls and capabilities for a given RAS feature for multiple instances
|
|
of the same component in a given system.
|
|
|
|
Memory RAS features apply to RAS capabilities, controls and operations that
|
|
are specific to memory. RAS2 PCC sub-spaces for memory-specific RAS features
|
|
have a Feature Type of 0x00 (Memory).
|
|
|
|
The platform can use the hardware-based memory scrubbing feature to expose
|
|
controls and capabilities associated with hardware-based memory scrub
|
|
engines. The RAS2 memory scrubbing feature supports as per spec,
|
|
|
|
1. Independent memory scrubbing controls for each NUMA domain, identified
|
|
using its proximity domain.
|
|
|
|
2. Provision for background (patrol) scrubbing of the entire memory system,
|
|
as well as on-demand scrubbing for a specific region of memory.
|
|
|
|
ACPI Address Range Scrubbing (ARS)
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
ACPI spec 6.5 [2]_ section 9.19.7.2 describes Address Range Scrubbing (ARS).
|
|
ARS allows the platform to communicate memory errors to system software.
|
|
This capability allows system software to prevent accesses to addresses with
|
|
uncorrectable errors in memory. ARS functions manage all NVDIMMs present in
|
|
the system. Only one scrub can be in progress system wide at any given time.
|
|
|
|
The following functions are supported as per the specification:
|
|
|
|
1. Query ARS Capabilities for a given address range, indicates platform
|
|
supports the ACPI NVDIMM Root Device Unconsumed Error Notification.
|
|
|
|
2. Start ARS triggers an Address Range Scrub for the given memory range.
|
|
Address scrubbing can be done for volatile or persistent memory, or both.
|
|
|
|
3. Query ARS Status command allows software to get the status of ARS,
|
|
including the progress of ARS and ARS error record.
|
|
|
|
4. Clear Uncorrectable Error.
|
|
|
|
5. Translate SPA
|
|
|
|
6. ARS Error Inject etc.
|
|
|
|
The kernel supports an existing control for ARS and ARS is currently not
|
|
supported in EDAC.
|
|
|
|
.. [1] https://computeexpresslink.org/cxl-specification/
|
|
|
|
.. [2] https://uefi.org/specs/ACPI/6.5/
|
|
|
|
Comparison of various scrubbing features
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
+--------------+-----------+-----------+-----------+-----------+
|
|
| | ACPI | CXL patrol| CXL ECS | ARS |
|
|
| Name | RAS2 | scrub | | |
|
|
+--------------+-----------+-----------+-----------+-----------+
|
|
| | | | | |
|
|
| On-demand | Supported | No | No | Supported |
|
|
| Scrubbing | | | | |
|
|
| | | | | |
|
|
+--------------+-----------+-----------+-----------+-----------+
|
|
| | | | | |
|
|
| Background | Supported | Supported | Supported | No |
|
|
| scrubbing | | | | |
|
|
| | | | | |
|
|
+--------------+-----------+-----------+-----------+-----------+
|
|
| | | | | |
|
|
| Mode of | Scrub ctrl| per device| per memory| Unknown |
|
|
| scrubbing | per NUMA | | media | |
|
|
| | domain. | | | |
|
|
+--------------+-----------+-----------+-----------+-----------+
|
|
| | | | | |
|
|
| Query scrub | Supported | Supported | Supported | Supported |
|
|
| capabilities | | | | |
|
|
| | | | | |
|
|
+--------------+-----------+-----------+-----------+-----------+
|
|
| | | | | |
|
|
| Setting | Supported | No | No | Supported |
|
|
| address range| | | | |
|
|
| | | | | |
|
|
+--------------+-----------+-----------+-----------+-----------+
|
|
| | | | | |
|
|
| Setting | Supported | Supported | No | No |
|
|
| scrub rate | | | | |
|
|
| | | | | |
|
|
+--------------+-----------+-----------+-----------+-----------+
|
|
| | | | | |
|
|
| Unit for | Not | in hours | No | No |
|
|
| scrub rate | Defined | | | |
|
|
| | | | | |
|
|
+--------------+-----------+-----------+-----------+-----------+
|
|
| | Supported | | | |
|
|
| Scrub | on-demand | No | No | Supported |
|
|
| status/ | scrubbing | | | |
|
|
| Completion | only | | | |
|
|
+--------------+-----------+-----------+-----------+-----------+
|
|
| UC error | |CXL general|CXL general| ACPI UCE |
|
|
| reporting | Exception |media/DRAM |media/DRAM | notify and|
|
|
| | |event/media|event/media| query |
|
|
| | |scan? |scan? | ARS status|
|
|
+--------------+-----------+-----------+-----------+-----------+
|
|
| | | | | |
|
|
| Support for | Supported | Supported | Supported | No |
|
|
| EDAC control | | | | |
|
|
| | | | | |
|
|
+--------------+-----------+-----------+-----------+-----------+
|
|
|
|
The File System
|
|
---------------
|
|
|
|
The control attributes of a registered scrubber instance could be
|
|
accessed in:
|
|
|
|
/sys/bus/edac/devices/<dev-name>/scrubX/
|
|
|
|
sysfs
|
|
-----
|
|
|
|
Sysfs files are documented in
|
|
`Documentation/ABI/testing/sysfs-edac-scrub`
|
|
|
|
`Documentation/ABI/testing/sysfs-edac-ecs`
|
|
|
|
Examples
|
|
--------
|
|
|
|
The usage takes the form shown in these examples:
|
|
|
|
1. CXL memory Patrol Scrub
|
|
|
|
The following are the use cases identified why we might increase the scrub rate.
|
|
|
|
- Scrubbing is needed at device granularity because a device is showing
|
|
unexpectedly high errors.
|
|
|
|
- Scrubbing may apply to memory that isn't online at all yet. Likely this
|
|
is a system wide default setting on boot.
|
|
|
|
- Scrubbing at a higher rate because the monitor software has determined that
|
|
more reliability is necessary for a particular data set. This is called
|
|
Differentiated Reliability.
|
|
|
|
1.1. Device based scrubbing
|
|
|
|
CXL memory is exposed to memory management subsystem and ultimately userspace
|
|
via CXL devices. Device-based scrubbing is used for the first use case
|
|
described in "Section 1 CXL Memory Patrol Scrub".
|
|
|
|
When combining control via the device interfaces and region interfaces,
|
|
"see Section 1.2 Region based scrubbing".
|
|
|
|
Sysfs files for scrubbing are documented in
|
|
`Documentation/ABI/testing/sysfs-edac-scrub`
|
|
|
|
1.2. Region based scrubbing
|
|
|
|
CXL memory is exposed to memory management subsystem and ultimately userspace
|
|
via CXL regions. CXL Regions represent mapped memory capacity in system
|
|
physical address space. These can incorporate one or more parts of multiple CXL
|
|
memory devices with traffic interleaved across them. The user may want to control
|
|
the scrub rate via this more abstract region instead of having to figure out the
|
|
constituent devices and program them separately. The scrub rate for each device
|
|
covers the whole device. Thus if multiple regions use parts of that device then
|
|
requests for scrubbing of other regions may result in a higher scrub rate than
|
|
requested for this specific region.
|
|
|
|
Region-based scrubbing is used for the third use case described in
|
|
"Section 1 CXL Memory Patrol Scrub".
|
|
|
|
Userspace must follow below set of rules on how to set the scrub rates for any
|
|
mixture of requirements.
|
|
|
|
1. Taking each region in turn from lowest desired scrub rate to highest and set
|
|
their scrub rates. Later regions may override the scrub rate on individual
|
|
devices (and hence potentially whole regions).
|
|
|
|
2. Take each device for which enhanced scrubbing is required (higher rate) and
|
|
set those scrub rates. This will override the scrub rates of individual devices,
|
|
setting them to the maximum rate required for any of the regions they help back,
|
|
unless a specific rate is already defined.
|
|
|
|
Sysfs files for scrubbing are documented in
|
|
`Documentation/ABI/testing/sysfs-edac-scrub`
|
|
|
|
2. CXL memory Error Check Scrub (ECS)
|
|
|
|
The Error Check Scrub (ECS) feature enables a memory device to perform error
|
|
checking and correction (ECC) and count single-bit errors. The associated
|
|
memory controller sets the ECS mode with a trigger sent to the memory
|
|
device. CXL ECS control allows the host, thus the userspace, to change the
|
|
attributes for error count mode, threshold number of errors per segment
|
|
(indicating how many segments have at least that number of errors) for
|
|
reporting errors, and reset the ECS counter. Thus the responsibility for
|
|
initiating Error Check Scrub on a memory device may lie with the memory
|
|
controller or platform when unexpectedly high error rates are detected.
|
|
|
|
Sysfs files for scrubbing are documented in
|
|
`Documentation/ABI/testing/sysfs-edac-ecs`
|