mirror of
https://git.proxmox.com/git/pve-common
synced 2025-04-28 15:57:11 +00:00
ef0bcc98ad
The use of `print` can be inefficient for writing larger files due to its default buffering in 8 KiB blocks. This is especially problematic on `pmxcfs` where files are written in 4 KiB blocks due to the defaults of `libfuse2`. This leads to significant write amplification on files larger than 4 KiB. Patch (fix #5728: pmxcfs: allow bigger writes than 4k for fuse) [1] addresses this by enabling `big_writes`, allowing up to 128 KiB blocks. But due to the use of `print` in `file_set_contents`, writes are still only buffered in 8 KiB blocks. To further address this, this commit switches to using `syswrite` instead of `print` to mitigate the block size limit imposed by `print`. Combined with patch [1], file writes to `/etc/pve/` are now buffered in 128 KiB blocks. The table below illustrates the drastic reduction in write amplification when writing files of different sizes to `/etc/pve/` using `file_set_contents`: print big_writes+print big_writes+syswrite file size written amplif. written amplif. written amplif. 1 KiB 48 KiB 48.0 45 KiB 45.0 41 KiB 41.0 2 KiB 48 KiB 24.0 45 KiB 22.5 62 KiB 31.0 4 KiB 82 KiB 20.5 80 KiB 20.0 73 KiB 18.3 8 KiB 121 KiB 15.1 90 KiB 11.3 89 KiB 11.1 16 KiB 217 KiB 13.6 146 KiB 9.1 113 KiB 7.1 32 KiB 506 KiB 15.8 314 KiB 9.8 158 KiB 4.9 64 KiB 1472 KiB 23.0 826 KiB 12.9 259 KiB 4.0 128 KiB 5585 KiB 43.6 3765 KiB 29.4 452 KiB 3.5 256 KiB 20424 KiB 79.8 10743 KiB 42.0 2351 KiB 9.2 512 KiB 86715 KiB 169.4 43650 KiB 85.3 3204 KiB 6.3 1024 KiB 369568 KiB 360.9 187496 KiB 183.1 15845 KiB 15.5 Since `file_set_contents` also performs a `rename` after writing, the following table shows the results when the file is written without renaming it afterwards: print big_writes+print big_writes+syswrite file size written amplif. written amplif. written amplif. 1 KiB 29 KiB 29.0 29 KiB 29.0 25 KiB 25.0 2 KiB 29 KiB 14.5 30 KiB 15.0 25 KiB 12.5 4 KiB 37 KiB 9.3 44 KiB 11.0 41 KiB 10.3 8 KiB 61 KiB 7.6 45 KiB 5.6 45 KiB 5.6 16 KiB 143 KiB 8.9 86 KiB 5.4 57 KiB 3.6 32 KiB 396 KiB 12.4 225 KiB 7.0 69 KiB 2.2 64 KiB 1281 KiB 20.0 673 KiB 10.5 105 KiB 1.6 128 KiB 4789 KiB 37.4 3478 KiB 27.2 169 KiB 1.3 256 KiB 18868 KiB 73.7 9976 KiB 39.0 572 KiB 2.2 512 KiB 79304 KiB 154.9 42714 KiB 83.4 2150 KiB 4.2 1024 KiB 347929 KiB 339.8 182483 KiB 178.2 11133 KiB 10.9 [1] https://lists.proxmox.com/pipermail/pve-devel/2024-September/065396.html Signed-off-by: Filip Schauer <f.schauer@proxmox.com> |
||
|---|---|---|
| debian | ||
| src | ||
| test | ||
| .gitignore | ||
| Makefile | ||
| README.dev | ||
= Setup PVE Development Environment =
0. Read https://pve.proxmox.com/wiki/Developer_Documentation
1. Install Debian 12 Bookworm (you can also start from a PVE installation and
skip step 2 - 5, 7 - 11)
2. Configure the network interface(s)
3. Change the IP address of your hostname for proper name resolution
in /etc/hosts
Using 127.0.1.1 will not work, so change it to an IP address from your
local network!
4: Check that the Debian repositories are set properly.
See https://wiki.debian.org/SourcesList for more information.
5. Optional: Install openssh-server and connect via ssh to the host.
run: apt-get update && apt-get install openssh-server
Connect via ssh to host and switch user to root
6. Configure 'pvetest' repository in /etc/apt/sources.list.d/:
run: echo "deb http://download.proxmox.com/debian bookworm pvetest" > /etc/apt/sources.list.d/pve-development.list
7. Add the repository key, run:
wget -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg "https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg"
8. run: apt-get update && apt-get dist-upgrade
9. run: apt-get install proxmox-ve
10. run: mv /etc/apt/sources.list.d/pve-enterprise.list /etc/apt/sources.list.d/pve-enterprise.list.bak
11. You should now have a working Proxmox VE installation.
Open a browser: https://<host_IP_address>:8006 e.g. https://10.0.0.90:8006
= Install build prerequisites for development environment =
NOTE: this is a huge and probably outdated list intended to be able to build
(almost) all packages, from the UI/API components to backend components to our
Linux Kernel. If you only want to hack on specific topics you won't need most
of those.
Instead we try to have a complete list of build dependencies in each source
repositories 'debian/control' file. If you run `make deb` dpkg-buildpackage
will stop and tell you if you miss some required packages.
12. For installing the most important, always needed, ones run:
apt-get install build-essential git git-email debhelper pve-doc-generator
Additionally, for quickly installing (almost) all build dependencies run:
WARNING: this list is almost for sure outdated! Use the build-deps definitions
defined in each package! You could install `devscripts` (huge package, but nice
helpers) and use:
# mk-build-deps --install
in the top-level directory of a git repository.
apt-get install autotools-dev autogen dh-autoreconf dkms doxygen check pkg-config \
groff quilt dpatch automake autoconf libtool lintian libdevel-cycle-perl \
libjson-perl libcommon-sense-perl liblinux-inotify2-perl libio-stringy-perl \
libstring-shellquote-perl dh-systemd rpm2cpio libsqlite3-dev sqlite3 \
libglib2.0-dev librrd-dev librrds-perl rrdcached libdigest-hmac-perl \
libxml-parser-perl gdb libcrypt-openssl-random-perl \
libcrypt-openssl-rsa-perl libnet-ldap-perl libauthen-pam-perl \
libjson-xs-perl libterm-readline-gnu-perl oathtool libmime-base32-perl \
liboath0 libpci-dev texi2html libsdl1.2-dev libgnutls28-dev \
libspice-protocol-dev xfslibs-dev libnuma-dev libaio-dev \
pve-libspice-server-dev libusbredirparser-dev glusterfs-common \
libusb-1.0-0-dev librbd-dev libpopt-dev iproute bridge-utils numactl \
glusterfs-common ceph-common python-ceph libgoogle-perftools4 \
libfile-chdir-perl lvm2 glusterfs-client liblockfile-simple-perl \
libsystemd-dev libreadline-gplv2-dev libio-multiplex-perl \
libnetfilter-log-dev libipset3 ipset socat libsasl2-dev libogg-dev \
python-pyparsing libfilesys-df-perl libcrypt-ssleay-perl \
libfile-readbackwards-perl libanyevent-perl libanyevent-http-perl \
unzip liblocale-po-perl libfile-sync-perl cstream \
lzop dtach hdparm gdisk parted ttf-dejavu-core \
liblzma-dev dosfstools mtools libxen-dev libfuse-dev libcpg-dev libquorum-dev \
libcmap-dev libuuid-perl libqb-dev libapparmor-dev docbook2x libcap-dev \
dh-apparmor graphviz libseccomp-dev libglib-perl libgtk3-perl libnss3-dev \
libdlm-dev libudev-dev asciidoc-dblatex source-highlight libiscsi-dev \
libiscsi7 librsvg2-bin libarchive-dev libgpgme-dev libcurl4-gnutls-dev \
libtest-mockmodule-perl libjemalloc-dev libjpeg-dev
= Compile PVE packages from Source =
13: Download and install git repositories as Proxmox modules:
run: mkdir /root/proxmox && cd /root/proxmox
run: git clone git://git.proxmox.com/git/pve-common.git
'pve-common.git' is some kind of starting repository and needed for some
other repositories as dependency.
Install this to get an idea of how the installation process is working.
See https://git.proxmox.com/ for all available repositories.
14: Most packages can be installed with 'make dinstall' command.
run: cd pve-common && make dinstall
15: Reboot the system.
16. Learn to use the quilt patch scripts.
17. Happy coding!
= REST vs. SOAP =
We decided to change our SOAP API (1.X) and use a REST like API. The
concept is described in [1] (Resource Oriented Architecture
(ROA)). The main advantage is that we are able to remove a lot of code
(the whole SOAP stack) to reduce software complexity.
We also moved away from server side content generation. Instead we use
the ExtJS Rich Internet Application Framework
(http://www.sencha.com).
That framework, like any other AJAX toolkit, can talk directly to the
REST API using JSON. So we were able to remove the server side
template toolkit completely.
= JSON and JSON Schema =
We use JSON as data format, because it is simple and parse-able by any
web browser.
Additionally, we use JSON Schema [2] to formally describe our API. So
we can automatically generate the whole API Documentation, and we can
verify all parameters and return values.
A great side effect was that we are able to use JSON Schema to
produce command line argument parsers automatically. In fact, the REST
API and the command line tools use the same code.
Object linkage is done using the JSON Hyper Schema (links property).
A small utility called 'pvesh' exposes the whole REST API on the command
line.
So here is a summary of the advantage:
- easy, human readable data format (native web browser format)
- automatic parameter verification (we can also verify return values)
- automatic generation of API documentation
- easy way to create command line tools (using same API).
= API Implementation (PVE::RESTHandler) =
All classes exposing methods on the API use PVE::RESTHandler as base class.
use base qw(PVE::RESTHandler);
To expose methods, one needs to call register_method():
__PACKAGE__->register_method ($schema);
Where $schema is a PVE method schema as described in
PVE::JSONSchema. It includes a description of parameters and return
values, and a reference to the actual code
__PACKAGE__->register_method ({
name => 'echo',
path => 'echo',
method => 'GET',
description => "simple return value of parameter 'text'",
parameters => {
additionalProperties => 0,
properties => {
text => {
type => 'string',
}
},
},
returns => {
type => 'string',
},
code => sub {
my ($param) = @_;
return $param->{text};
}
});
The 'name' property is only used if you want to call the method
directly from Perl. You can do that using:
print __PACKAGE__->echo({ text => "a test" });
We use Perl's AUTOLOAD feature to implement this. Note: You need to
pass parameters a HASH reference.
There is a special helper method called cli_handler(). This is used by
the CLIHandler Class for command line tools, where you want to pass
arguments as array of strings. This uses Getopt::Long to parse parameters.
There is a second way to map names to methods - using the 'path'
property. And you can register subclasses. That way you can set up a
filesystem like hierarchy to access methods.
Here is an example:
----------------------------
package C1;
__PACKAGE__->register_method ({
subclass => "C2",
path => 'sub2',
});
__PACKAGE__->register_method ({
name => 'list1',
path => 'index',
method => 'GET',
...
});
package C2;
__PACKAGE__->register_method ({
name => 'list2',
path => 'index',
method => 'GET',
...
});
-------------------------------
The utily method find_handler (in PVE::RESTHandler) can be use to do
'path' related method lookups.
C1->find_handler('GET', "/index") => C1::list1
C1->find_handler('GET', "/sub2/index") => C2::list2
The HTTP server use the URL (a path) to find the corresponding method.
= References =
[1] RESTful Web Services
Web services for the real world
By
Leonard Richardson, Sam Ruby
Publisher:
O'Reilly Media
Released:
May 2007
[2] JSON Schema links: http://json-schema.org/