From ec07a280ba797b221d859abdd71a4d92c3a339cd Mon Sep 17 00:00:00 2001 From: Dietmar Maurer Date: Sun, 28 Feb 2021 10:32:56 +0100 Subject: [PATCH] docs: document data blob format --- docs/backup-protocol.rst | 8 +++---- docs/file-formats.rst | 48 ++++++++++++++++++++++++++++++++++++++-- 2 files changed, 50 insertions(+), 6 deletions(-) diff --git a/docs/backup-protocol.rst b/docs/backup-protocol.rst index 27adcb1d..f21c7efc 100644 --- a/docs/backup-protocol.rst +++ b/docs/backup-protocol.rst @@ -38,7 +38,7 @@ Upload Blobs ~~~~~~~~~~~~ Uploading blobs is done using ``POST /blob``. The HTTP body contains the -data encoded as ``DataBlob`` (see source code). +data encoded as :ref:`Data Blob `). The file name needs to end with ``.blob``, and is automatically added to the backup manifest. @@ -50,7 +50,7 @@ Upload Chunks Chunks belong to an index, so you first need to open an index (see below). After that, you can upload chunks using ``POST /fixed_chunk`` and ``POST /dynamic_chunk``. The HTTP body contains the chunk data -encoded as ``DataBlob`` (see source code). +encoded as :ref:`Data Blob `). Upload Fixed Indexes @@ -122,14 +122,14 @@ Download Blobs ~~~~~~~~~~~~~~ Downloading blobs is done using ``GET /download``. The HTTP body contains the -data encoded as ``DataBlob`` (see source code). +data encoded as :ref:`Data Blob `. Download Chunks ~~~~~~~~~~~~~~~ Downloading chunks is done using ``GET /chunk``. The HTTP body contains the -data encoded as ``DataBlob`` (see source code). +data encoded as :ref:`Data Blob `). Download Index Files diff --git a/docs/file-formats.rst b/docs/file-formats.rst index bb725649..71c4f468 100644 --- a/docs/file-formats.rst +++ b/docs/file-formats.rst @@ -6,7 +6,51 @@ File Formats Proxmox File Archive Format (``.pxar``) --------------------------------------- - .. graphviz:: pxar-format-overview.dot - + +.. _data-blob-format: + +Data Blob Format +---------------- + +The data blob format is used to store small binary data. The magic number decides the exact format: + +.. list-table:: + :widths: auto + + * - ``[66, 171, 56, 7, 190, 131, 112, 161]`` + - unencrypted + - uncompressed + * - ``[49, 185, 88, 66, 111, 182, 163, 127]`` + - unencrypted + - compressed + * - ``[123, 103, 133, 190, 34, 45, 76, 240]`` + - encrypted + - uncompressed + * - ``[230, 89, 27, 191, 11, 191, 216, 11]`` + - encrypted + - compressed + +Compression algorithm is ``zstd``. Encryption cipher is ``AES_256_GCM``. + +Unencrypted blobs use the following format: + +.. list-table:: + :widths: auto + + * - ``MAGIC: [u8; 8]`` + * - ``CRC32: [u8; 4]`` + * - ``Data: (max 16MiB)`` + +Encrypted blobs additionally contains a 16 byte IV, followed by a 16 +byte Authenticated Encyryption (AE) tag, followed by the encrypted +data: + +.. list-table:: + + * - ``MAGIC: [u8; 8]`` + * - ``CRC32: [u8; 4]`` + * - ``ÌV: [u8; 16]`` + * - ``TAG: [u8; 16]`` + * - ``Data: (max 16MiB)``