This patch introduces a hierarchical backlog mechanism to cache
user data in high-throughput encryption/decryption scenarios,
the implementation addresses packet loss issues when hardware
queues overflow during peak loads.
First, we use sec_alloc_req_id to obtain an exclusive resource
from the pre-allocated resource pool of each queue, if no resource
is allocated, perform the DMA map operation on the request memory.
When the task is ready, we will attempt to send it to the hardware,
if the hardware queue is already full, we cache the request into
the backlog list, then return an EBUSY status to the upper layer
and instruct the packet-sending thread to pause transmission.
Simultaneously, when the hardware completes a task, it triggers
the sec callback function, within this function, reattempt to send
the requests from the backlog list and wake up the sending thread
until the hardware queue becomes fully occupied again.
In addition, it handles such exceptions like the hardware is reset
when packets are sent, it will switch to the software computing
and release occupied resources.
Signed-off-by: Wenkai Lin <linwenkai6@hisilicon.com>
Signed-off-by: Chenghai Huang <huangchenghai2@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The following splat was triggered when booting the kernel built with
arm64's defconfig + CRYPTO_SELFTESTS + DMA_API_DEBUG.
------------[ cut here ]------------
DMA-API: hisi_sec2 0000:75:00.0: cacheline tracking EEXIST, overlapping mappings aren't supported
WARNING: CPU: 24 PID: 1273 at kernel/dma/debug.c:596 add_dma_entry+0x248/0x308
Call trace:
add_dma_entry+0x248/0x308 (P)
debug_dma_map_sg+0x208/0x3e4
__dma_map_sg_attrs+0xbc/0x118
dma_map_sg_attrs+0x10/0x24
hisi_acc_sg_buf_map_to_hw_sgl+0x80/0x218 [hisi_qm]
sec_cipher_map+0xc4/0x338 [hisi_sec2]
sec_aead_sgl_map+0x18/0x24 [hisi_sec2]
sec_process+0xb8/0x36c [hisi_sec2]
sec_aead_crypto+0xe4/0x264 [hisi_sec2]
sec_aead_encrypt+0x14/0x20 [hisi_sec2]
crypto_aead_encrypt+0x24/0x38
test_aead_vec_cfg+0x480/0x7e4
test_aead_vec+0x84/0x1b8
alg_test_aead+0xc0/0x498
alg_test.part.0+0x518/0x524
alg_test+0x20/0x64
cryptomgr_test+0x24/0x44
kthread+0x130/0x1fc
ret_from_fork+0x10/0x20
---[ end trace 0000000000000000 ]---
DMA-API: Mapped at:
debug_dma_map_sg+0x234/0x3e4
__dma_map_sg_attrs+0xbc/0x118
dma_map_sg_attrs+0x10/0x24
hisi_acc_sg_buf_map_to_hw_sgl+0x80/0x218 [hisi_qm]
sec_cipher_map+0xc4/0x338 [hisi_sec2]
This occurs in selftests where the input and the output scatterlist point
to the same underlying memory (e.g., when tested with INPLACE_TWO_SGLISTS
mode).
The problem is that the hisi_sec2 driver maps these two different
scatterlists using the DMA_BIDIRECTIONAL flag which leads to overlapped
write mappings which are not supported by the DMA layer.
Fix it by using the fine grained and correct DMA mapping directions. While
at it, switch the DMA directions used by the hisi_zip driver too.
Signed-off-by: Zenghui Yu <yuzenghui@huawei.com>
Reviewed-by: Longfang Liu <liulongfang@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
During encryption and decryption, user requests
must be checked first, if the specifications that
are not supported by the hardware are used, the
software computing is used for processing.
Fixes: 2f072d75d1 ("crypto: hisilicon - Add aead support on SEC2")
Signed-off-by: Wenkai Lin <linwenkai6@hisilicon.com>
Signed-off-by: Chenghai Huang <huangchenghai2@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The hardware only supports authentication sizes
that are 4-byte aligned. Therefore, the driver
switches to software computation in this case.
Fixes: 2f072d75d1 ("crypto: hisilicon - Add aead support on SEC2")
Signed-off-by: Wenkai Lin <linwenkai6@hisilicon.com>
Signed-off-by: Chenghai Huang <huangchenghai2@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
According to the HMAC RFC, the authentication key
can be 0 bytes, and the hardware can handle this
scenario. Therefore, remove the incorrect validation
for this case.
Fixes: 2f072d75d1 ("crypto: hisilicon - Add aead support on SEC2")
Signed-off-by: Wenkai Lin <linwenkai6@hisilicon.com>
Signed-off-by: Chenghai Huang <huangchenghai2@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
On the HiSilicon accelerators drivers, the PF/VFs driver can send messages
to the VFs/PF by writing hardware registers, and the VFs/PF driver receives
messages from the PF/VFs by reading hardware registers. To support this
feature, a new version id is added, different communication mechanism are
used based on different version id.
Signed-off-by: Yang Shen <shenyang39@huawei.com>
Signed-off-by: Weili Qian <qianweili@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
When the digest alg is HMAC-SHAx or another, the authsize may be less
than 4 bytes and mac_len of the BD is set to zero, the hardware considers
it a BD configuration error and reports a ras error, so the sec driver
needs to switch to software calculation in this case, this patch add a
check for it and remove unnecessary check that has been done by crypto.
Fixes: 2f072d75d1 ("crypto: hisilicon - Add aead support on SEC2")
Signed-off-by: Wenkai Lin <linwenkai6@hisilicon.com>
Signed-off-by: Chenghai Huang <huangchenghai2@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
When the AEAD algorithm is used for encryption or decryption,
the input authentication length varies, the hardware needs to
obtain the input length to pass the integrity check verification.
Currently, the driver uses a fixed authentication length,which
causes decryption failure, so the length configuration is modified.
In addition, the step of setting the auth length is unnecessary,
so it was deleted from the setkey function.
Fixes: 2f072d75d1 ("crypto: hisilicon - Add aead support on SEC2")
Signed-off-by: Wenkai Lin <linwenkai6@hisilicon.com>
Signed-off-by: Chenghai Huang <huangchenghai2@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The error detection of the data aggregation feature is separated from
the compression/decompression feature. This patch enables the error
detection and reporting of the data aggregation feature. When an
unrecoverable error occurs in the algorithm core, the device reports
the error to the driver, and the driver will reset the device.
Signed-off-by: Weili Qian <qianweili@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
If an error indicating that the device needs to be reset is reported,
disable the error reporting before device reset is complete,
enable the error reporting after the reset is complete to prevent
the same error from being reported repeatedly.
Fixes: eaebf4c3b1 ("crypto: hisilicon - Unify hardware error init/uninit into QM")
Signed-off-by: Weili Qian <qianweili@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Query the capability register status of accelerator devices
(SEC, HPRE and ZIP) through the debugfs interface, for example:
cat cap_regs. The purpose is to improve the robustness and
locability of hardware devices and drivers.
Signed-off-by: Qi Tao <taoqi10@huawei.com>
Signed-off-by: Chenghai Huang <huangchenghai2@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Ensure that the inline function contains no more than 10 lines.
move q_num_set() from hisi_acc_qm.h to qm.c.
Signed-off-by: Chenghai Huang <huangchenghai2@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Before the device is enabled again, the device may still
store the previously processed data. If an error occurs in
the previous task, the device may fail to be enabled again.
Therefore, before enabling device, reset the device to restore
the initial state.
Signed-off-by: Weili Qian <qianweili@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The offset of SEC_CORE_ENABLE_BITMAP should be 0 instead of 32,
it cause a kasan shift-out-bounds warning, fix it.
Signed-off-by: Wenkai Lin <linwenkai6@hisilicon.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Some information showed by the dump function is invalid. Mask
the unnecessary information from the dump file.
Signed-off-by: Chenghai Huang <huangchenghai2@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The AIV is one of the SEC resources. When releasing resources,
it need to release the AIV resources at the same time.
Otherwise, memory leakage occurs.
The aiv resource release is added to the sec resource release
function.
Signed-off-by: Chenghai Huang <huangchenghai2@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
There is a scenario where the file directory is created but the
file memory is not set. In this case, if a user accesses the
file, an error occurs.
So during the creation process of debugfs, memory should be
allocated first before creating the directory. In the release
process, the directory should be deleted first before releasing
the memory to avoid the situation where the memory does not
exist when accessing the directory.
In addition, the directory released by the debugfs is a global
variable. When the debugfs of an accelerator fails to be
initialized, releasing the directory of the global variable
affects the debugfs initialization of other accelerators.
The debugfs root directory released by debugfs init should be a
member of qm, not a global variable.
Signed-off-by: Chenghai Huang <huangchenghai2@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
When CONFIG_PCI_IOV is disabled, the SRIOV configuration
function is not required. An error occurs if this function is
incorrectly called.
Consistent with other modules, add the condition for
configuring the sriov function of sec_pci_driver.
Signed-off-by: Chenghai Huang <huangchenghai2@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Unused parameter of static functions should be removed.
Signed-off-by: Wenkai Lin <linwenkai6@hisilicon.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This patch fixes following cleanup issues:
- The return value of the function is
inconsistent with the actual return type.
- After the pointer type is directly converted
to the `__le64` type, the program may crash
or produce unexpected results.
Signed-off-by: Qi Tao <taoqi10@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Nested macros are integrated into a single macro,
making the code simpler.
Signed-off-by: Qi Tao <taoqi10@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
As the sec DFX function is enhanced, some RAS registers are added
to the original DFX registers to enhance the DFX positioning function.
Signed-off-by: Qi Tao <taoqi10@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Switch to raw_smp_processor_id() to prevent a number of
warnings from kernel debugging. We do not care about
preemption here, as the CPU number is only used as a
poor mans load balancing or device selection. If preemption
happens during an encrypt/decrypt operation a small performance
hit will occur but everything will continue to work, so just
ignore it. This commit is similar to e7a9b05ca4
("crypto: cavium - Fix smp_processor_id() warnings").
[ 7538.874350] BUG: using smp_processor_id() in preemptible [00000000] code: af_alg06/8438
[ 7538.874368] caller is debug_smp_processor_id+0x1c/0x28
[ 7538.874373] CPU: 50 PID: 8438 Comm: af_alg06 Kdump: loaded Not tainted 5.10.0.pc+ #18
[ 7538.874377] Call trace:
[ 7538.874387] dump_backtrace+0x0/0x210
[ 7538.874389] show_stack+0x2c/0x38
[ 7538.874392] dump_stack+0x110/0x164
[ 7538.874394] check_preemption_disabled+0xf4/0x108
[ 7538.874396] debug_smp_processor_id+0x1c/0x28
[ 7538.874406] sec_create_qps+0x24/0xe8 [hisi_sec2]
[ 7538.874408] sec_ctx_base_init+0x20/0x4d8 [hisi_sec2]
[ 7538.874411] sec_aead_ctx_init+0x68/0x180 [hisi_sec2]
[ 7538.874413] sec_aead_sha256_ctx_init+0x28/0x38 [hisi_sec2]
[ 7538.874421] crypto_aead_init_tfm+0x54/0x68
[ 7538.874423] crypto_create_tfm_node+0x6c/0x110
[ 7538.874424] crypto_alloc_tfm_node+0x74/0x288
[ 7538.874426] crypto_alloc_aead+0x40/0x50
[ 7538.874431] aead_bind+0x50/0xd0
[ 7538.874433] alg_bind+0x94/0x148
[ 7538.874439] __sys_bind+0x98/0x118
[ 7538.874441] __arm64_sys_bind+0x28/0x38
[ 7538.874445] do_el0_svc+0x88/0x258
[ 7538.874447] el0_svc+0x1c/0x28
[ 7538.874449] el0_sync_handler+0x8c/0xb8
[ 7538.874452] el0_sync+0x148/0x180
Signed-off-by: Wenkai Lin <linwenkai6@hisilicon.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Add the printf of an error message and optimized the handling
process of ret.
Signed-off-by: Chenghai Huang <huangchenghai2@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Pre-store the valid value of the sec alg support related capability
register in sec_qm_init(), which will be called by probe process.
It can reduce the number of capability register queries and avoid
obtaining incorrect values in abnormal scenarios, such as reset
failed and the memory space disabled.
Fixes: 921715b6b7 ("crypto: hisilicon/sec - get algorithm bitmap from registers")
Signed-off-by: Zhiqi Song <songzhiqi1@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Extract a public function to set qm algs and remove
the similar code for setting qm algs in each module.
Signed-off-by: Wenkai Lin <linwenkai6@hisilicon.com>
Signed-off-by: Hao Fang <fanghao11@huawei.com>
Signed-off-by: Zhiqi Song <songzhiqi1@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
When the Kunpeng accelerator executes tasks such as encryption
and decryption have minimum requirements on the number of device
queues. If the number of queues does not meet the requirement,
the process initialization will fail. Therefore, the driver checks
the number of queues on the device before registering the algorithm.
If the number does not meet the requirements, the driver does not register
the algorithm to crypto subsystem, the device is still added to the
qm_list.
Signed-off-by: Weili Qian <qianweili@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
If the queue isolation feature is enabled, the number of queues
supported by the device changes. When PF is enabled using the
current default number of queues, the default number of queues may
be greater than the number supported by the device. As a result,
the PF fails to be bound to the driver.
After modification, if queue isolation feature is enabled, when
the default queue parameter is greater than the number supported
by the device, the number of enabled queues will be changed to
the number supported by the device, so that the PF and driver
can be properly bound.
Fixes: 8bbecfb402 ("crypto: hisilicon/qm - add queue isolation support for Kunpeng930")
Signed-off-by: Longfang Liu <liulongfang@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
When sec_aead_mac_init returns an error code, sec_cipher_map
will exit abnormally, the hardware sgl should be unmmaped.
Signed-off-by: Wenkai Lin <linwenkai6@hisilicon.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
<linux/aer.h> is unused, so remove it.
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Cc: Kai Ye <yekai13@huawei.com>
Cc: Longfang Liu <liulongfang@huawei.com>
Acked-by: Longfang Liu <liulongfang@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Fix that put two or more continuous blank lines inside function.
Signed-off-by: Wenkai Lin <linwenkai6@hisilicon.com>
Signed-off-by: Kai Ye <yekai13@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
There are a couple of spelling mistakes in sec2. Fix them.
Signed-off-by: Kai Ye <yekai13@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The last register logic and different register logic are combined.
Use "u32" instead of 'int' in the regs function input parameter to
simplify some checks.
Signed-off-by: Kai Ye <yekai13@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Change the value of clock gating register to 0x7fff to enable
clock gating of the address prefetch module. When the device is
idle, the clock is turned off to save power.
Signed-off-by: Weili Qian <qianweili@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The value of qm algorithm can change dynamically according to the
value of the capability register.
Add xxx_set_qm_algs() function to obtain the algs that the
hardware device supported from the capability register and set
them into usr mode attribute files.
Signed-off-by: Zhiqi Song <songzhiqi1@huawei.com>
Signed-off-by: Wenkai Lin <linwenkai6@hisilicon.com>
Signed-off-by: Weili Qian <qianweili@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Add function 'sec_get_alg_bitmap' to get hardware algorithm bitmap
before register algorithm to crypto, instead of determining
whether to register an algorithm based on hardware platform's version.
Signed-off-by: Wenkai Lin <linwenkai6@hisilicon.com>
Signed-off-by: Weili Qian <qianweili@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Hardware V3 and later versions support get error type from
registers. To be compatible with later hardware versions,
get error type from registers instead of fixed marco.
Signed-off-by: Weili Qian <qianweili@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Hardware V3 and later versions can obtain qp num and depth supported
by the hardware from registers. To be compatible with later hardware
versions, get qp num and depth from registers instead of fixed marcos.
Signed-off-by: Weili Qian <qianweili@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Before hardware V3, hardwares do not provide the feature registers,
driver resolves hardware differences based on the hardware version.
As a result, the driver does not support the new hardware.
Hardware V3 and later versions support to obtain hardware features,
such as power-gating management and doorbell isolation, through
the hardware registers. To be compatible with later hardware versions,
the features of the current device is obtained by reading the
hardware registers instead of the hardware version.
Signed-off-by: Weili Qian <qianweili@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The authentication algorithm supports a maximum of 128-byte keys.
The allocated key memory is insufficient.
Fixes: 2f072d75d1 ("crypto: hisilicon - Add aead support on SEC2")
Signed-off-by: Kai Ye <yekai13@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The base register address of V2 and V3 are different. HW V3 not needs
to change the BD err detection.
Signed-off-by: Kai Ye <yekai13@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Before stopping the function, the driver needs to flush all the remaining
work about event irq. Therefore, accelerator drivers use a private
workqueue(qm->wq) to handle event irq instead of the system workqueue.
This patch moves alloc workqueue from sec_main.c and zip_main.c to qm.c.
Signed-off-by: Weili Qian <qianweili@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Spelling mistakes (triple letters) in comment.
Detected with the help of Coccinelle.
Signed-off-by: Julia Lawall <Julia.Lawall@inria.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Should not to uses the CRYPTO_ALG_ALLOCATES_MEMORY in SEC2. The SEC2
driver uses the pre-allocated buffers, including the src sgl pool, dst
sgl pool and other qp ctx resources. (e.g. IV buffer, mac buffer, key
buffer). The SEC2 driver doesn't allocate memory during request processing.
The driver only maps software sgl to allocated hardware sgl during I/O. So
here is fix it.
Signed-off-by: Kai Ye <yekai13@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Add last word dumping function during sec engine controller reset.
Signed-off-by: Kai Ye <yekai13@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The value of the register is changed after the task running. A debugfs
file node is added to help users to check the change of register values.
Signed-off-by: Longfang Liu <liulongfang@huawei.com>
Signed-off-by: Kai Ye <yekai13@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
