Commit Graph

57 Commits

Author SHA1 Message Date
Dietmar Maurer
8619b21e4d proxmox-rrd: make rrd load callback configurable 2021-10-14 11:41:26 +02:00
Dietmar Maurer
52d5f340f2 proxmox-rrd: add more regression tests 2021-10-14 10:55:12 +02:00
Dietmar Maurer
f2e1ab2d44 proxmox-rrd: add regression tests and two minor fixes 2021-10-14 10:17:07 +02:00
Dietmar Maurer
a1eede6918 proxmox-rrd: pass time and value to update function 2021-10-14 08:12:56 +02:00
Dietmar Maurer
4231e2d5f5 proxmox-rrd: add some integration tests (file format tests) 2021-10-13 18:21:23 +02:00
Dietmar Maurer
432374d024 use complete_file_name from proxmox-router 1.1 2021-10-13 14:10:02 +02:00
Thomas Lamprecht
919ccf713a proxmox-rrd: move unshipped cli tool to examples
it's a rather low-level tool mostly useful for debugging and some of
it is rather "dumb" (by design) anyway, e.g., it does not
transparently applies journal but really only operates on the DB
files as is (which can conflict with daemon operations).

In summary, not (yet) a tool meant for end user consumption.
Move it to examples folder to avoid compilation on packaging (we do
not ship it anyway) which allows us to move the rather expensive
proxmox-router (pulls in hyper) to the dev-dependencies section.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
392d646f7b proxmox-rrd: add more commands to the rrd cli tool
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
7edea7e08c proxmox-rrd: rename last_counter to last_value
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
66dfd1f08f proxmox-rrd: protect against negative update time
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
507f19dd33 proxmox-rrd: new helpers: slot, slot_start_time & slot_end_time
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
334eb9ce48 proxmox-rrd: avoid expensive modulo (%) inside loop
Modulo is very slow, so we try to avoid it inside loops.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
4cd28918e2 proxmox-rrd: add binary to create/manage rrd files
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
2c72c6a7ba proxmox-rrd: split out load_rrd (cleanup)
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
e928c24948 proxmox-rrd: support CF::Last
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
4bf2db8666 remove proxmox-rrd-api-types crate, s/RRDTimeFrameResolution/RRDTimeFrame/
Because the types used inside the RRD have other requirements
than the API types:

- other serialization format
- the API may not support all RRD features

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
ab567561b5 proxmox-rrd: extract_data: include values from current slot
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
cf097c5a89 proxmox-rrd: remove dependency to proxmox-rrd-api-types
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
bc68dee171 proxmox-rrd: implement new CBOR based format
Storing much more data points now got get better graphs.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
0355554905 proxmox-rrd: use a journal to reduce amount of bytes written
Append pending changes in a simple text based format that allows for
lockless appends as long as we stay below 4 KiB data per write.

Apply the journal every 30 minutes and on daemon startup.

Note that we do not ensure that the journal is synced, this is a
perfomance optimization we can make as the kernel defaults to
writeback in-flight data every 30s (sysctl vm/dirty_expire_centisecs)
anyway, so we lose at max half a minute of data on a crash, here one
should have in mind that we normally expose 1 minute as finest
granularity anyway, so not really much lost.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Wolfgang Bumiller
e0ce41b03a update to first proxmox crate split
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-11 11:58:49 +02:00
Wolfgang Bumiller
fa9757e67f bump proxmox dependency to 0.14.0 and proxmox-http to 0.5.0
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-08 11:18:22 +02:00
Dietmar Maurer
071cb7aa8b proxmox-rrd: use correct directory options in create_rrdb_dir 2021-10-07 08:50:50 +02:00
Dietmar Maurer
9c64c09c92 proxmox-rrd: cleanup error handling 2021-10-07 08:01:12 +02:00
Dietmar Maurer
5165bed8c2 proxmox-rrd: use log crate instead of eprintln, avoid duplicate logs 2021-10-06 18:19:22 +02:00
Dietmar Maurer
9c7fd3c936 proxmox-rrd: fix update (do not update) when time is in the past 2021-10-06 18:01:48 +02:00
Dietmar Maurer
881d8f85ea proxmox-rrd: improve developer docs 2021-10-06 12:19:54 +02:00
Dietmar Maurer
54f7a80f97 proxmox-rrd: remove serde dependency 2021-10-06 10:55:46 +02:00
Dietmar Maurer
538e6f66f3 split out RRD api types into proxmox-rrd-api-types crate 2021-10-06 09:49:51 +02:00
Dietmar Maurer
ac17698e4a proxmox-rrd: use create_path instead of std::fs::create_dir_all
To ensure correct file ownership.
2021-10-06 08:37:14 +02:00
Dietmar Maurer
8d1a9d2ec6 move RRD code into proxmox-rrd crate 2021-10-06 08:13:28 +02:00
Dietmar Maurer
a4e56bff60 more api type cleanups: avoid re-exports 2021-09-10 12:25:32 +02:00
Dietmar Maurer
80f7bf3822 start new pbs-config workspace
moved src/config/domains.rs
2021-09-02 12:58:20 +02:00
Fabian Grünbichler
c94f367dec clippy: more misc fixes
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:55 +01:00
Fabian Grünbichler
655ceac1c6 clippy: collapse/rework nested ifs
no semantic changes (intended).

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:22:59 +01:00
Wolfgang Bumiller
4d8bd987a4 clippy fixups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-14 11:20:07 +02:00
Stefan Reiter
6eff0b289e rrd: fix integer underflow
Causes a panic if last_update is smaller than RRD_DATA_ENTRIES*reso,
which (I believe) can happen when inserting the first value for a DB.

Clamp the value to 0 in that case.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-01 14:30:32 +02:00
Dietmar Maurer
110ceff08c avoid chrono dependency, depend on proxmox 0.3.8
- remove chrono dependency

- depend on proxmox 0.3.8

- remove epoch_now, epoch_now_u64 and epoch_now_f64

- remove tm_editor (moved to proxmox crate)

- use new helpers from proxmox 0.3.8
  * epoch_i64 and epoch_f64
  * parse_rfc3339
  * epoch_to_rfc3339_utc
  * strftime_local

- BackupDir changes:
  * store epoch and rfc3339 string instead of DateTime
  * backup_time_to_string now return a Result
  * remove unnecessary TryFrom<(BackupGroup, i64)> for BackupDir

- DynamicIndexHeader: change ctime to i64

- FixedIndexHeader: change ctime to i64
2020-09-15 07:12:57 +02:00
Dominik Csapak
50fa6045b3 api2/status: use new rrd::extract_cached_data
and drop the now unused extract_lists function

this also fixes a bug, where we did not add the datastore to the list at
all when there was no rrd data

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 13:31:16 +02:00
Dominik Csapak
11cb8cd008 rrd: move creation of serde value into api
there is now a 'extract_cached_data' which just returns
the data of the specified field, and an api function that converts
a list of fields to the correct serde value

this way we do not have to create a serde value in rrd/cache.rs
(makes for a better interface)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 13:31:14 +02:00
Dominik Csapak
9a860821e4 refactor time functions to tools
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 13:31:10 +02:00
Dominik Csapak
20ada7a08c rrd: add 'extract_lists'
this is an interface to simply get the Vec<Option<f64>> out of rrd
without going through serde values

we return a list of timestamps and a HashMap with the lists we could find
(otherwise it is not in the map)

if no lists could be extracted, the time list is also empty

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-09 12:19:06 +02:00
Dietmar Maurer
1fa3341d68 rrd: reduce io by saving data only once a minute 2020-05-29 09:16:13 +02:00
Dietmar Maurer
a76b0e3c09 src/rrd/rrd.rs: do not wrap error and return ErrorKind::NotFound 2020-05-25 10:30:04 +02:00
Dietmar Maurer
9ce21e005d src/rrd/cache.rs: display/log error when RRD load fails 2020-05-25 10:18:53 +02:00
Dietmar Maurer
207d0c7714 src/rrd/rrd.rs: store/verify magic number 2020-05-25 09:21:54 +02:00
Dietmar Maurer
bfc2f8b7a2 src/rrd/rrd.rs: implement DST_COUNTER 2020-05-25 08:14:30 +02:00
Dietmar Maurer
d261516f2d src/rrd/rrd.rs: correctly compute derived values
use f64 for time.
2020-05-25 07:02:04 +02:00
Dietmar Maurer
f4561029ba src/rrd/rrd.rs: implement DST_DERIVE 2020-05-24 19:02:35 +02:00
Dietmar Maurer
7acdec12e8 src/rrd/rrd.rs: restructure whole code 2020-05-24 16:51:28 +02:00