Upgrading to 6.0

What a Rezolus 5.x setup does differently under 6.0, and what to run instead.

6.0 is not released yet.

You record with rezolus record

The output must end in .dendro. Without -o, record writes rezolus.dendro, where 5.x wrote rezolus.rez. Any other output is refused with exit code 2 before an endpoint is contacted, and the message names a 5.x recorder (5.27.0 or later), which still writes the old formats.

$rezolus record -o run.rez
error: -o run.rez: rezolus 6.x writes .dendro only (-o out.dendro); .rez output was removed in 6.0. ...
# exit 2
$rezolus record -o run.dendro
5.x6.0
record -o run.rez, -o run.parquet, -o run.rawrecord -o run.dendro
record --format parquet (or rez, raw)Leave out --format. --format dendro is the only value accepted.
record --separate --endpoint A --endpoint Brecord --endpoint A --endpoint B -o run.dendro: one archive, with each endpoint as its own recording.
record --stream -o run.rezrecord -o run.dendro. Every agent is recorded over its stream; --stream is an unknown argument (exit 2).
--endpoint URL,source=NAME--endpoint URL,alias=NAME. source= is still accepted and read as alias=, with a note on stderr.
--metadata producer_epoch=VALUE (or producer_epochs=)Refused (exit 2). The recorder writes the producer epoch itself; use another key.
output, format, separate = true in a --config fileThe same rules apply: output must end in .dendro, and format other than dendro and separate = true are refused (exit 2).

A recording's name now lives in its alias label. A recording carries a source label only when its producer declares one. A KPI template or saved query that selects source="NAME" for a name you gave with source=NAME now selects nothing; select alias="NAME" instead.

$rezolus record --endpoint http://127.0.0.1:4241,source=web -o s.dendro
note: source=web on http://127.0.0.1:4241/ is read as alias=web; source= is deprecated, use alias=

You run hindsight

The packaged /etc/rezolus/hindsight.toml changes its output from /var/lib/rezolus/rezolus.rez to /var/lib/rezolus/rezolus.dendro. A SIGHUP capture is written beside it as rezolus-<timestamp>.dendro.

  • If you never edited the file, the package upgrade replaces it and hindsight writes to the new path.
  • If you edited it, dpkg asks which version to keep (and keeps yours under --force-confold). rpm keeps yours and writes the new one beside it as hindsight.toml.rpmnew.
  • A kept file whose output does not end in .dendro is refused at startup with exit code 1, and the service does not run until you change it.
  • The files 5.x wrote stay where they are. 6.0 writes rezolus.dendro and rezolus-<timestamp>.dendro, so anything that reads /var/lib/rezolus/rezolus.rez or globs rezolus-*.rez finds no dump written after the upgrade. Point it at the .dendro names.
$rezolus hindsight /etc/rezolus/hindsight.toml
output = "/var/lib/rezolus/rezolus.rez": hindsight 6.x writes .dendro archives only; use an output ending in .dendro. ...
# exit 1. Fix the config, then restart:
$sudo sed -i 's#^output = .*#output = "/var/lib/rezolus/rezolus.dendro"#' /etc/rezolus/hindsight.toml
$sudo systemctl restart rezolus-hindsight

Hindsight reads the agent's /metrics/stream, so the agent it points at must be 5.21.0 or later. An older agent, or a rezolus exporter endpoint, is refused at startup with exit code 1. Existing .rez dumps are converted as described under You have .rez or parquet files.

  • GET /dump on hindsight's HTTP endpoint returns a .dendro archive (dump.dendro), where 5.x returned a .rez.
  • Hindsight's GET /status no longer has failed_scrapes or scrapes_failing_since. Hindsight no longer scrapes; a dropped stream is reconnected after one interval.

You have .rez or parquet files

rezolus view, rezolus mcp, the static-site viewer and every rezolus recording subcommand except upgrade refuse a .rez archive (v1, v2 or v3), a .parquet file and a .parquet.ab.tar A/B tarball, and print the command that converts the file. The exit code is 1, except for recording check, which exits 2. Nothing converts a file when it is opened.

$rezolus view old.rez
old.rez is a v3 .rez archive, which rezolus no longer opens; convert it to a .dendro archive first:
  rezolus recording upgrade old.rez
# exit 1

rezolus recording upgrade FILE writes FILE's name with a .dendro extension next to it, and leaves the input unchanged. -o names the output instead. It refuses to replace an existing file.

$rezolus recording upgrade old.rez
converted old.rez (v3 .rez archive) to old.dendro: 2 source(s), 2 segment(s), 0 live WAL row(s), 4 clock offset(s)
$rezolus recording upgrade capture.parquet -o capture.dendro
converted capture.parquet (parquet recording) to capture.dendro: 1 source(s), 1 segment(s), 5 row(s)
# Convert every .rez, parquet and A/B tarball in a directory
$for f in *.rez *.parquet *.parquet.ab.tar; do [ -e "$f" ] || continue; rezolus recording upgrade "$f"; done
  • A .parquet.ab.tar becomes one .dendro with two recordings labelled arm=baseline and arm=experiment, which the viewer opens as the comparison. x.parquet.ab.tar becomes x.dendro. The tarball's aliases and category are not carried over.
  • A combined parquet written by 5.x recording combine, or by a 5.x recorder given several endpoints, becomes one recording per input.
  • In 5.x, recording upgrade old.rez rewrote a v1 or v2 .rez in place as v3, and refused a v3 file ("already a v3 (SQLite) archive", exit 1). In 6.0 the same command converts any .rez version and writes old.dendro beside it. --to v3 is refused (exit 1); --to dendro is accepted and gives the same result as leaving it out.
  • A .dendro written by 5.x opens in 6.0 as it is. recording upgrade on it exits 1 with "already a dendro archive; there is nothing to upgrade".

You run agents older than 5.21.0

6.0 records a Rezolus agent only from its /metrics/stream route, which agents serve from 5.21.0 with snapshot_format = "v3" (the default). record, hindsight and the live viewer (rezolus view http://host:4241) refuse an older agent, a snapshot_format = "v2" agent, and a proxy that does not route /metrics/stream. Hindsight and the live viewer also refuse a rezolus exporter endpoint (exit 1).

  • At startup, record exits 1 and writes nothing.
  • An agent that comes up during the run, or that answers a reconnect with a refusal, is left out of the recording. The other endpoints are recorded, the archive is finalized, and the run names the agent it left out without failing.
$rezolus record --url http://old-host:4241 -o run.dendro
error: http://old-host:4241/ is Rezolus 5.20.0, which predates the replication stream (agents serve /metrics/stream from 5.21.0): ... Nothing was recorded. ...
# exit 1

Either upgrade the agent to 5.21.0 or later, or record it with a 5.x recorder (5.27.0 or later) and convert the result with 6.0:

# with the 5.x binary
$rezolus record --url http://old-host:4241 -o run.rez
# with the 6.0 binary
$rezolus recording upgrade run.rez

You read 6.0 recordings with 5.x

A .dendro written by 6.0 uses archive schema version 5. That includes record and hindsight output, Save as Report (named <recording>-report.dendro by default, or <baseline>_vs_<experiment>.dendro in a comparison) and the live viewer's Save (rezolus-capture.dendro by default).

# 5.x, on a file written by 6.0
$rezolus mcp describe-recording run.dendro
Failed to load recording: unsupported archive schema version 5: this build writes v4 and reads v4

Every 5.x release from 5.23.0 opens .dendro schema version 4 only and refuses the file with this message. Releases before 5.23.0 do not open .dendro files at all. No command converts a 6.0 archive to an older format, so upgrade every reader (people, scripts and pipelines) to 6.0 before relying on 6.0 recordings.

Scripts and integrations

  • File names: change *.rez and *.parquet to *.dendro in output paths, globs and artifact collection.
  • Parquet consumers: a .dendro is not a parquet file, and 6.0 has no parquet export. Read values with rezolus mcp query FILE 'PROMQL', or keep a 5.x recorder for that pipeline.
  • Exit codes: record refuses a removed output or flag with exit code 2 (a usage error). A command given a .rez or parquet input exits 1, except recording check, which exits 2.
  • A Prometheus _seconds histogram is stored in seconds, as the exporter serves it; 5.x stored its buckets in nanoseconds. histogram_quantile over a 6.0 recording reads seconds, so a recording check threshold on it is in seconds (0.25, not 250000000). Charts read the unit from the recording and draw either as times. A comparison or combined archive holding a 5.x and a 6.0 recording of one _seconds histogram reads one in nanoseconds and the other in seconds, so its charts and ratios are off by 1e9; compare recordings made by the same major version.
  • rate(x[range]) now returns the average rate over [t - range, t]. 5.x ignored the range and returned the per-step rate, the same as irate. A saved query, KPI template or recording check written for the per-step rate should use irate(x[...]), or rate(x[s]) where s is the step. Gauges, irate and rate(x[s]) (with s the step) return the same values as 5.x at every timestamp 5.x returned. 6.0 may also return a point at the recording's first or last tick.
  • =~ and !~ are anchored regular expressions, as in Prometheus. 5.x read them as lists of literal alternatives separated by |. Escape a ., +, (, [ or \ that is meant literally.
  • rezolus mcp query, describe-metrics, detect-anomalies and analyze-correlation on an archive with several recordings read them together and exit 0. 5.x exited 1 and listed the recordings. An A/B archive (recordings that carry arm) still exits 1 and needs --recording arm=baseline or --recording arm=experiment.
  • A query written for one host can now match series from every host, or nothing. Over two hosts, sum(irate(cpu_usage[5m])) / cpu_cores answers "No series matched", because cpu_cores has one series per host. Use sum by (host) (irate(cpu_usage[5m])) / on (host) cpu_cores, a matcher such as cpu_cores{host="web-01"}, or --recording host=web-01.
  • rezolus mcp query on a metric the recording has, with a selector that matches no series, prints "No series matched." and exits 0. 5.x printed "Metric not found" and exited 1.
  • Every series read from a .dendro carries its recording's labels: host, source, endpoint, alias, any record --label, and __producer__ and arm where the recording has them. A series' own label under one of those keys is renamed exported_<key>, so a matcher on the series' own value needs the exported_ name.
  • An archive made by recording combine opens as an A/B comparison only when its recordings carry an arm label. Give combine the arms explicitly: rezolus recording combine baseline.dendro experiment.dendro --ab baseline=baseline.dendro experiment=experiment.dendro -o ab.dendro. To open an older combined archive as a comparison, use rezolus view ab.dendro --baseline host=A --experiment host=B. When its two recordings have identical labels (two runs combined on one host), no selector can name either one, and view exits 1 with "matched 2 recordings". Label them by position instead: rezolus recording combine ab.dendro --ab baseline=0 experiment=1 -o ab2.dendro.
  • recording filter x.dendro -o y.dendro refuses to replace an existing y.dendro (exit 1). 5.x overwrote it.
  • The agent's /status no longer has index_resyncs.
  • rezolus-capture (the script and the Docker image) writes one capture.dendro in --output-dir in place of capture.parquet and the per-endpoint parquet files. --source rezolus is refused.
  • Tools that read the agent's /metrics/stream directly see a new content type, application/vnd.metriken.replication.v1+dendro, and x-metriken-* headers in place of x-rezolus-*. Every 5.x record --stream refuses a 6.0 agent's stream; drop --stream, and the 5.x recorder scrapes the agent instead.
  • rezolus view a.dendro b.dendro opens b.dendro as the experiment of a comparison. 5.x ignored a second file after a .rez or .dendro, with a warning.

Agent configuration and metrics

  • Three pairs of samplers are merged. [samplers.scheduler_runqueue] and [samplers.cpu_migrations] are now [samplers.scheduler]; [samplers.blockio_requests] and [samplers.blockio_latency] are [samplers.blockio]; [samplers.syscall_counts] and [samplers.syscall_latency] are [samplers.syscall]. An old section name is read as the new one with a warning. Metric names are unchanged, but the sampler label, rezolus status and rezolus_bpf_run_time use the new names.
  • hw_sensors is on by default. [samplers.hw_sensors] enabled = false turns it off. It no longer reads nvme or drivetemp hwmon chips; drivehealth reports drive temperature. It no longer publishes the ACPI Processor cooling devices (one per logical CPU). Each channel gains device and name labels.
  • tcp_connect_latency has an outcome label (established, refused, timeout, error, aborted). A query without an outcome matcher merges every outcome, timeouts included; add {outcome="established"} for the 5.x meaning. The exporter's percentiles are per outcome.
  • task_cpu_usage leaves out threads younger than task_min_lifetime (default "1s"). Set "0s" for the 5.x behaviour.
  • Setting task_attribution = true also turns on process_attribution unless it is set. Set process_attribution = false in [samplers.cpu_usage] for the 5.x behaviour.
  • memory_slabinfo publishes memory_slab_cache_bytes for every slab cache (up to 1024). caches = "fixed" in [samplers.memory_slabinfo] keeps the 5.x set of eight. Both memory_slab_cache_* metrics gain an id label, so a Prometheus server sees new series for every cache.
  • ext4_journal reports in nanoseconds instead of jiffies. phase="wait" and phase="request_delay" are removed. Its values are not comparable with 5.x recordings.
  • On macOS, cpu_usage is one series per CPU with an id label, and the nice state is gone (niced time counts as user).

Removed flags

5.x6.0 resultInstead
record --format rez|parquet|rawexit 2Leave it out; output is .dendro.
record --separateexit 2One archive with a recording per endpoint.
record --streamexit 2 (unknown argument)Leave it out; agents are always streamed.
recording upgrade --to v3exit 1recording upgrade FILE
recording combine -o x.parquet, -o x.rez, -o x.parquet.ab.tarexit 1-o x.dendro; add --ab baseline=A experiment=B for a comparison.
recording combine --pinned, --category, --bypass-time-checkexit 1 (--category without --ab: exit 2, a usage error in 5.x too)Leave them out.
recording annotate --source, --node, --filterexit 1Name a recording when it is recorded (--endpoint URL,alias=NAME or --label); shrink one with recording filter.
recording filter --queries, --templatesexit 2 (unknown argument)recording filter --samplers or --metrics.
recording metadata --schema, --file, --geometry, --fieldexit 2 (unknown argument)recording metadata -i FILE (add --json for machine-readable output).
recording filter, annotate, check --annotate editing a .rez or parquet in placeexit 1 (check: exit 2)Convert with recording upgrade, then edit the .dendro in place or write a copy with -o.
recording convert to parquetrefusedrecording convert FILE.raw writes a .dendro.
Viewer Time selector (Aligned/Raw)removedOpen a chart on its own (the open-in-new-tab link) and use SHOW SAMPLES.