From 0d2b1b39cc90500bd9eb416c00fcea8cf19da8b2 Mon Sep 17 00:00:00 2001 From: Boris Rybalkin Date: Thu, 9 Jul 2026 08:47:54 +0100 Subject: [PATCH 1/6] TODO: drop gptfdisk from 3rdparty migration list (already in-tree) --- TODO.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/TODO.md b/TODO.md index 2088bca1..2a84833e 100644 --- a/TODO.md +++ b/TODO.md @@ -11,7 +11,6 @@ repo with its own release workflow. platform CI pipeline (similar to how nginx and authelia are already built in-tree), so upgrading a dependency is just a version bump in this repo. -**Current downloads in `package.sh`:** +**Remaining downloads in `package.sh`:** - `https://github.com/syncloud/3rdparty/releases/download/openldap/openldap-${ARCH}.tar.gz` - `https://github.com/syncloud/3rdparty/releases/download/btrfs/btrfs-${ARCH}.tar.gz` -- `https://github.com/syncloud/3rdparty/releases/download/gptfdisk/gptfdisk-${ARCH}.tar.gz` From 3dd5e26ef8b99d9546bfc96a054d5fb530745dce Mon Sep 17 00:00:00 2001 From: Boris Rybalkin Date: Thu, 9 Jul 2026 08:47:54 +0100 Subject: [PATCH 2/6] stability: always enable zram, drop the memory-total gate Compressed swap is cheap headroom even on high-RAM boxes; the 6 GiB MemTotal gate skipped exactly the pressured 8 GB multi-snap boxes (boris prod) that benefit most. Remove the gate so EnsureConfigured always configures zram; alreadyOn() keeps it idempotent. --- TODO-zram-health.md | 55 ++++++++++++++++++++++++++++++++++ backend/stability/zram.go | 5 ---- backend/stability/zram_test.go | 7 ----- 3 files changed, 55 insertions(+), 12 deletions(-) create mode 100644 TODO-zram-health.md diff --git a/TODO-zram-health.md b/TODO-zram-health.md new file mode 100644 index 00000000..5e768d69 --- /dev/null +++ b/TODO-zram-health.md @@ -0,0 +1,55 @@ +# TODO: zram swap + Health-page bottleneck visibility + +## zram swap on high-RAM-but-overloaded boxes (boris prod) + +### Context +- boris.syncloud.it prod: 8 GB RAM (`MemTotal 8052952 kB`), ~15 snaps running (photoprism, mastodon, matrix, nextcloud, pihole, games, mail…). +- Only swap = legacy 953 MB disk partition `/dev/sda1`, 100% full (cold parked pages). +- Box was imaged from the **legacy amd64 image** (`debian-buster-amd64-8gb.img`); swap partition + fstab entry come from the image, NOT added manually. +- Modern amd64 pipeline (`image/tools/extract-amd64.sh` → debian-12-generic + EFI, `rootfs-amd64.sh`) ships **no swap** at all. arm/arm64 fstabs also have no swap. + +### Why platform zram is skipped here +`backend/stability/zram.go` `EnsureConfigured()`: +```go +memThresholdKB = 6 * 1024 * 1024 // 6 GiB +if snap.TotalKB > memThresholdKB { skip } +``` +Box has 8 GB > 6 GiB gate ⇒ zram never runs. Confirmed on prod: zram module not loaded, no `/dev/zram0`. + +### Decision: gate removed — zram always enabled (DONE) +`EnsureConfigured()` no longer checks memory; `memThresholdKB` is gone. zram is +configured on every box regardless of total/available RAM. Compressed swap is +cheap headroom even on large boxes, and `alreadyOn()` keeps it idempotent. +(Considered but rejected: Option A bump to 8 GiB; Option B gate on available RAM.) + +### Existing swap NOT auto-disabled — no extra code needed +- `disableFileSwaps()` only turns off `Type == "file"` swaps (`fields[1] != "file"` → skip). +- boris `/dev/sda1` is `Type == partition` ⇒ left alone. +- Result after enabling zram: zram prio 10 (used first) + sda1 prio -2 (overflow). Ideal. fstab re-adds sda1 on boot, stability service re-adds zram on boot. Stable, zero further changes. +- Only if we wanted to KILL sda1 too would we need code (extend disableFileSwaps to partitions — risky, affects all devices) or edit this box's fstab + `swapoff`. +- zram would size to 2 GiB cap (RAM/2=4G clamped), zstd comp_algorithm, prio 10. + +## Prove disk/swap is (or isn't) the bottleneck + surface it on Health page + +### Current evidence: NOT disk-bound (correction to earlier swap-thrash hypothesis) +Sampled prod ~12s + swap counters: +- `vmstat` si/so = **0/0** sustained → not paging in/out of swap. +- wa (iowait) = **0–1%**, idle 95–97%, load 0.2–0.6. +- swap 100% full but **static** (cold pages) — benign, normal. +- cumulative pswpin 92 days = 354k pages ≈ 1.4 GB (~15 MB/day) — negligible. +- The 100% swap BAR is misleading; RATE (si/so) + iowait are the real proof, both ~0 at rest. +- k9mail slowness more likely IMAP-over-TLS RTT or one-off Dovecot indexing. TODO: sample **during** a k9mail sync (si/so, wa, `/proc//io`). + +### Constraint +- PSI unavailable: kernel `4.19.0-8-amd64` (legacy buster), needs ≥4.20 + CONFIG_PSI. `/proc/pressure/*` absent. +- Note: `backend/stability/events.go` already has a `PSIavg10` field → newer kernels/boards would populate it, this box can't. + +### Health page work (`backend/health/metrics.go` + `web/platform/src/views/Health.vue`) +Backend already collects: `CPU.IOWait` (unused), `Disk.SectorsRead/Wrt` (shown as ↓/↑ KB/s), `Memory.SwapTotalKB/FreeKB` (swap bar). + +Add, in priority order: +1. **Swap-in/out rate (si/so)** — the definitive thrash signal, NOT collected. Add `swap_in_pages`/`swap_out_pages` (from `/proc/vmstat` pswpin/pswpout) to `Memory` struct; frontend computes delta via existing `prevMetrics` pattern (like diskRates) → show `swap ↓X · ↑Y KB/s`. +2. **IOWait %** — already in struct, just not shown. Compute `(iowait_delta/total_delta)*100`, render as its own bar or a segment of the CPU bar. +3. (optional) **PSI mem/io row** — degrade gracefully: hide when `/proc/pressure` absent (this box), show on newer boards. + +Goal: turn the misleading static "swap 100%" into signals that distinguish cold-parked-pages (harmless, current state) from active thrashing (real bottleneck). diff --git a/backend/stability/zram.go b/backend/stability/zram.go index 4919d2df..cb080aea 100644 --- a/backend/stability/zram.go +++ b/backend/stability/zram.go @@ -21,7 +21,6 @@ const ( zramSysBlockDefault = "/sys/block/zram0" zramHotAddDefault = "/sys/class/zram-control/hot_add" procSwapsDefault = "/proc/swaps" - memThresholdKB = 6 * 1024 * 1024 zramMaxSizeBytes = uint64(2 * 1024 * 1024 * 1024) zramPriority = 10 swapMagicV1 = "SWAPSPACE2" @@ -54,10 +53,6 @@ func (z *Zram) EnsureConfigured() error { if err != nil { return fmt.Errorf("zram: meminfo: %w", err) } - if snap.TotalKB > memThresholdKB { - z.log.Info("zram: skipping; memory above threshold", zap.Uint64("total_kb", snap.TotalKB)) - return nil - } on, err := z.alreadyOn() if err != nil { return fmt.Errorf("zram: check swaps: %w", err) diff --git a/backend/stability/zram_test.go b/backend/stability/zram_test.go index d8d8dcfc..117cddfd 100644 --- a/backend/stability/zram_test.go +++ b/backend/stability/zram_test.go @@ -54,13 +54,6 @@ func strconvUint(v uint64) string { return string(digits) } -func TestEnsureSkipsAboveMemThreshold(t *testing.T) { - z, sysBlock := newTestZram(t, 8*1024*1024, "Filename...\n") - require.NoError(t, z.EnsureConfigured()) - size, _ := os.ReadFile(filepath.Join(sysBlock, "disksize")) - assert.Equal(t, "0", string(size)) -} - func TestEnsureSkipsIfAlreadyOn(t *testing.T) { z, sysBlock := newTestZram(t, 4*1024*1024, "Filename\t\tType\tSize\tUsed\tPriority\n") sc, err := os.ReadFile(z.procSwaps) From abf34dab70948e371f0d4bb12d1801b474f03880 Mon Sep 17 00:00:00 2001 From: Boris Rybalkin Date: Thu, 9 Jul 2026 10:10:43 +0100 Subject: [PATCH 3/6] stability: tolerate zstd-unsupported kernels in zram setup Old buster kernels (4.19) expose zram comp_algorithm without zstd ([lzo] lz4 lz4hc). Writing zstd there fails EINVAL, which aborted EnsureConfigured before disksize/swapon, so zram never came up. zram already has a working default algorithm, so treat the comp_algorithm write as best-effort and only require disksize. Surfaced on borisarm64 (amd64 buster) once the memory gate was removed. --- backend/stability/zram.go | 2 +- backend/stability/zram_test.go | 9 +++++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/backend/stability/zram.go b/backend/stability/zram.go index cb080aea..07f4e3bb 100644 --- a/backend/stability/zram.go +++ b/backend/stability/zram.go @@ -170,7 +170,7 @@ func (z *Zram) sizeBytes(totalBytes uint64) uint64 { func (z *Zram) configureSysfs(sizeBytes uint64) error { if err := os.WriteFile(filepath.Join(z.sysBlock, "comp_algorithm"), []byte("zstd"), 0644); err != nil { - return err + z.log.Warn("zram: zstd unsupported, keeping kernel default algorithm", zap.Error(err)) } return os.WriteFile(filepath.Join(z.sysBlock, "disksize"), []byte(fmt.Sprintf("%d", sizeBytes)), 0644) } diff --git a/backend/stability/zram_test.go b/backend/stability/zram_test.go index 117cddfd..9f365f77 100644 --- a/backend/stability/zram_test.go +++ b/backend/stability/zram_test.go @@ -86,6 +86,15 @@ func TestConfigureSysfsWritesAlgoAndSize(t *testing.T) { assert.Equal(t, "123456", string(size)) } +func TestConfigureSysfsToleratesUnsupportedAlgo(t *testing.T) { + z, sysBlock := newTestZram(t, 4*1024*1024, "") + require.NoError(t, os.Remove(filepath.Join(sysBlock, "comp_algorithm"))) + require.NoError(t, os.Mkdir(filepath.Join(sysBlock, "comp_algorithm"), 0755)) + require.NoError(t, z.configureSysfs(123456)) + size, _ := os.ReadFile(filepath.Join(sysBlock, "disksize")) + assert.Equal(t, "123456", string(size)) +} + func TestSizeBytesCapped(t *testing.T) { z, _ := newTestZram(t, 4*1024*1024, "") assert.Equal(t, uint64(2*1024*1024*1024), z.sizeBytes(8*1024*1024*1024)) From c18625e75b71f748bea0a25fe085818a734dcff5 Mon Sep 17 00:00:00 2001 From: Boris Rybalkin Date: Thu, 9 Jul 2026 13:16:43 +0100 Subject: [PATCH 4/6] health: show swap in/out rate (si/so) on Health page MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The swap fullness bar can't distinguish cold parked pages (benign) from active thrashing. Collect pswpin/pswpout from /proc/vmstat as swap_in_pages/swap_out_pages; the frontend deltas them like disk rates and renders 'in X · out Y KB/s' under the swap bar. Missing /proc/vmstat (or absent counters) degrades to 0. PSI intentionally left out — no kernel support on the target boxes. --- backend/health/metrics.go | 68 ++++++++++++++++++++--------- backend/health/metrics_test.go | 15 +++++++ web/platform/src/locales/ar.json | 2 + web/platform/src/locales/de.json | 2 + web/platform/src/locales/en.json | 2 + web/platform/src/locales/es.json | 2 + web/platform/src/locales/fr.json | 2 + web/platform/src/locales/hi.json | 2 + web/platform/src/locales/ja.json | 2 + web/platform/src/locales/pt.json | 2 + web/platform/src/locales/ru.json | 2 + web/platform/src/locales/zh-CN.json | 2 + web/platform/src/views/Health.vue | 8 ++++ 13 files changed, 90 insertions(+), 21 deletions(-) diff --git a/backend/health/metrics.go b/backend/health/metrics.go index cbe04c19..cf2ac6c6 100644 --- a/backend/health/metrics.go +++ b/backend/health/metrics.go @@ -10,14 +10,14 @@ import ( ) type CPU struct { - User uint64 `json:"user"` - Nice uint64 `json:"nice"` - System uint64 `json:"system"` - Idle uint64 `json:"idle"` - IOWait uint64 `json:"iowait"` - IRQ uint64 `json:"irq"` + User uint64 `json:"user"` + Nice uint64 `json:"nice"` + System uint64 `json:"system"` + Idle uint64 `json:"idle"` + IOWait uint64 `json:"iowait"` + IRQ uint64 `json:"irq"` SoftIRQ uint64 `json:"softirq"` - Steal uint64 `json:"steal"` + Steal uint64 `json:"steal"` } func (c CPU) Total() uint64 { @@ -29,13 +29,15 @@ func (c CPU) Busy() uint64 { } type Memory struct { - TotalKB uint64 `json:"total_kb"` - AvailableKB uint64 `json:"available_kb"` - FreeKB uint64 `json:"free_kb"` - BuffersKB uint64 `json:"buffers_kb"` - CachedKB uint64 `json:"cached_kb"` - SwapTotalKB uint64 `json:"swap_total_kb"` - SwapFreeKB uint64 `json:"swap_free_kb"` + TotalKB uint64 `json:"total_kb"` + AvailableKB uint64 `json:"available_kb"` + FreeKB uint64 `json:"free_kb"` + BuffersKB uint64 `json:"buffers_kb"` + CachedKB uint64 `json:"cached_kb"` + SwapTotalKB uint64 `json:"swap_total_kb"` + SwapFreeKB uint64 `json:"swap_free_kb"` + SwapInPages uint64 `json:"swap_in_pages"` + SwapOutPages uint64 `json:"swap_out_pages"` } type Disk struct { @@ -47,9 +49,9 @@ type Disk struct { } type Mount struct { - Path string `json:"path"` - TotalKB uint64 `json:"total_kb"` - UsedKB uint64 `json:"used_kb"` + Path string `json:"path"` + TotalKB uint64 `json:"total_kb"` + UsedKB uint64 `json:"used_kb"` } type Net struct { @@ -59,11 +61,11 @@ type Net struct { } type Snapshot struct { - CPU CPU `json:"cpu"` - Memory Memory `json:"memory"` - Disks []Disk `json:"disks"` + CPU CPU `json:"cpu"` + Memory Memory `json:"memory"` + Disks []Disk `json:"disks"` Mounts []Mount `json:"mounts"` - Net []Net `json:"net"` + Net []Net `json:"net"` } type Collector struct { @@ -85,6 +87,7 @@ func (c *Collector) Snapshot() (Snapshot, error) { if err != nil { return s, err } + mem.SwapInPages, mem.SwapOutPages = c.readSwapCounters() s.Memory = mem s.Disks, _ = c.readDisks() s.Net, _ = c.readNet() @@ -155,6 +158,29 @@ func (c *Collector) readMemory() (Memory, error) { return m, sc.Err() } +func (c *Collector) readSwapCounters() (in, out uint64) { + f, err := os.Open(filepath.Join(c.procDir, "vmstat")) + if err != nil { + return 0, 0 + } + defer f.Close() + sc := bufio.NewScanner(f) + for sc.Scan() { + fields := strings.Fields(sc.Text()) + if len(fields) != 2 { + continue + } + v, _ := strconv.ParseUint(fields[1], 10, 64) + switch fields[0] { + case "pswpin": + in = v + case "pswpout": + out = v + } + } + return in, out +} + func (c *Collector) readDisks() ([]Disk, error) { f, err := os.Open(filepath.Join(c.procDir, "diskstats")) if err != nil { diff --git a/backend/health/metrics_test.go b/backend/health/metrics_test.go index 5adb2077..93718b8b 100644 --- a/backend/health/metrics_test.go +++ b/backend/health/metrics_test.go @@ -21,6 +21,7 @@ func newTestCollector(t *testing.T) (*Collector, string) { dir := t.TempDir() writeProc(t, dir, "stat", "cpu 1000 50 200 5000 30 0 10 0\ncpu0 ...\n") writeProc(t, dir, "meminfo", "MemTotal: 3700000 kB\nMemAvailable: 1500000 kB\nMemFree: 200000 kB\nBuffers: 50000 kB\nCached: 900000 kB\nSwapTotal: 2000000 kB\nSwapFree: 1500000 kB\n") + writeProc(t, dir, "vmstat", "nr_free_pages 50000\npswpin 1234\npswpout 5678\npgfault 999\n") writeProc(t, dir, "diskstats", " 8 0 sda 100 0 200 0 10 0 20 0 0 0 0 0 0 0\n"+ " 8 1 sda1 50 0 100 0 5 0 10 0 0 0 0 0 0 0\n"+ " 179 0 mmcblk0 1000 0 2000 0 100 0 200 0 0 0 0 0 0 0\n"+ @@ -52,6 +53,20 @@ func TestReadMemory(t *testing.T) { assert.Equal(t, uint64(2000000), mem.SwapTotalKB) } +func TestReadSwapCounters(t *testing.T) { + c, _ := newTestCollector(t) + in, out := c.readSwapCounters() + assert.Equal(t, uint64(1234), in) + assert.Equal(t, uint64(5678), out) +} + +func TestReadSwapCountersMissingVmstat(t *testing.T) { + c := NewCollector(t.TempDir()) + in, out := c.readSwapCounters() + assert.Equal(t, uint64(0), in) + assert.Equal(t, uint64(0), out) +} + func TestReadDisksFiltersPartitionsAndLoops(t *testing.T) { c, _ := newTestCollector(t) disks, err := c.readDisks() diff --git a/web/platform/src/locales/ar.json b/web/platform/src/locales/ar.json index d68b485b..b40ae3ef 100644 --- a/web/platform/src/locales/ar.json +++ b/web/platform/src/locales/ar.json @@ -322,6 +322,8 @@ "ioWrite": "كتابة", "netRx": "استقبال", "netTx": "إرسال", + "swapIn": "داخل", + "swapOut": "خارج", "kindZramEnabled": "تم تمكين zram", "kindSwapoffFile": "تم تعطيل ملف التبديل", "kindPressure": "تم اكتشاف ضغط على الذاكرة", diff --git a/web/platform/src/locales/de.json b/web/platform/src/locales/de.json index 27390a4a..bf3abde5 100644 --- a/web/platform/src/locales/de.json +++ b/web/platform/src/locales/de.json @@ -322,6 +322,8 @@ "ioWrite": "schreiben", "netRx": "Empfang", "netTx": "Senden", + "swapIn": "rein", + "swapOut": "raus", "kindZramEnabled": "zram aktiviert", "kindSwapoffFile": "Auslagerungsdatei deaktiviert", "kindPressure": "Speicherengpass erkannt", diff --git a/web/platform/src/locales/en.json b/web/platform/src/locales/en.json index 2679f10d..bd880648 100644 --- a/web/platform/src/locales/en.json +++ b/web/platform/src/locales/en.json @@ -78,6 +78,8 @@ "ioWrite": "write", "netRx": "down", "netTx": "up", + "swapIn": "in", + "swapOut": "out", "kindZramEnabled": "zram enabled", "kindSwapoffFile": "file swap disabled", "kindPressure": "memory pressure detected", diff --git a/web/platform/src/locales/es.json b/web/platform/src/locales/es.json index 7c421104..7da21719 100644 --- a/web/platform/src/locales/es.json +++ b/web/platform/src/locales/es.json @@ -322,6 +322,8 @@ "ioWrite": "escritura", "netRx": "recibido", "netTx": "enviado", + "swapIn": "entra", + "swapOut": "sale", "kindZramEnabled": "zram activado", "kindSwapoffFile": "archivo de intercambio desactivado", "kindPressure": "presión de memoria detectada", diff --git a/web/platform/src/locales/fr.json b/web/platform/src/locales/fr.json index e57496b3..0e15a7a1 100644 --- a/web/platform/src/locales/fr.json +++ b/web/platform/src/locales/fr.json @@ -322,6 +322,8 @@ "ioWrite": "écriture", "netRx": "reçu", "netTx": "envoyé", + "swapIn": "entrée", + "swapOut": "sortie", "kindZramEnabled": "zram activé", "kindSwapoffFile": "fichier swap désactivé", "kindPressure": "pression mémoire détectée", diff --git a/web/platform/src/locales/hi.json b/web/platform/src/locales/hi.json index 21043eef..4428e453 100644 --- a/web/platform/src/locales/hi.json +++ b/web/platform/src/locales/hi.json @@ -322,6 +322,8 @@ "ioWrite": "लिखना", "netRx": "डाउनलोड", "netTx": "अपलोड", + "swapIn": "अंदर", + "swapOut": "बाहर", "kindZramEnabled": "zram सक्षम", "kindSwapoffFile": "फ़ाइल स्वैप अक्षम", "kindPressure": "मेमोरी दबाव का पता चला", diff --git a/web/platform/src/locales/ja.json b/web/platform/src/locales/ja.json index 61f60980..1a5ce431 100644 --- a/web/platform/src/locales/ja.json +++ b/web/platform/src/locales/ja.json @@ -322,6 +322,8 @@ "ioWrite": "書き込み", "netRx": "受信", "netTx": "送信", + "swapIn": "入", + "swapOut": "出", "kindZramEnabled": "zram を有効化", "kindSwapoffFile": "ファイルスワップを無効化", "kindPressure": "メモリ圧迫を検出", diff --git a/web/platform/src/locales/pt.json b/web/platform/src/locales/pt.json index 6c2bca0f..d44eb891 100644 --- a/web/platform/src/locales/pt.json +++ b/web/platform/src/locales/pt.json @@ -322,6 +322,8 @@ "ioWrite": "escrita", "netRx": "recebido", "netTx": "enviado", + "swapIn": "entra", + "swapOut": "sai", "kindZramEnabled": "zram ativado", "kindSwapoffFile": "arquivo de swap desativado", "kindPressure": "pressão de memória detectada", diff --git a/web/platform/src/locales/ru.json b/web/platform/src/locales/ru.json index 2bae25c6..6053039a 100644 --- a/web/platform/src/locales/ru.json +++ b/web/platform/src/locales/ru.json @@ -322,6 +322,8 @@ "ioWrite": "запись", "netRx": "приём", "netTx": "передача", + "swapIn": "вход", + "swapOut": "выход", "kindZramEnabled": "zram включён", "kindSwapoffFile": "файл подкачки отключён", "kindPressure": "обнаружена нехватка памяти", diff --git a/web/platform/src/locales/zh-CN.json b/web/platform/src/locales/zh-CN.json index fc28db05..e1c5a1fc 100644 --- a/web/platform/src/locales/zh-CN.json +++ b/web/platform/src/locales/zh-CN.json @@ -322,6 +322,8 @@ "ioWrite": "写入", "netRx": "下载", "netTx": "上传", + "swapIn": "换入", + "swapOut": "换出", "kindZramEnabled": "已启用 zram", "kindSwapoffFile": "已禁用文件交换", "kindPressure": "检测到内存压力", diff --git a/web/platform/src/views/Health.vue b/web/platform/src/views/Health.vue index 0127a217..b20404db 100644 --- a/web/platform/src/views/Health.vue +++ b/web/platform/src/views/Health.vue @@ -19,6 +19,7 @@

{{ $t('health.swap') }} — {{ swapUsedMb }} / {{ swapTotalMb }} MB

+
{{ $t('health.swapIn') }} {{ swapRate.inKBs }} · {{ $t('health.swapOut') }} {{ swapRate.outKBs }} KB/s
@@ -86,6 +87,7 @@ export default { prevMetrics: null, events: [], cpuPct: 0, + swapRate: { inKBs: 0, outKBs: 0 }, diskRates: [], netRates: [], metricsTimer: null, @@ -177,6 +179,7 @@ export default { computeDeltas (prev, next) { if (!prev) { this.cpuPct = 0 + this.swapRate = { inKBs: 0, outKBs: 0 } this.diskRates = (next.disks || []).map(d => ({ name: d.name, readKBs: 0, writeKBs: 0 })) this.netRates = (next.net || []).map(n => ({ name: n.name, rxKBs: 0, txKBs: 0 })) return @@ -187,6 +190,11 @@ export default { this.cpuPct = totalDelta > 0 ? Math.max(0, Math.min(100, ((totalDelta - idleDelta) / totalDelta) * 100)) : 0 const secs = METRICS_INTERVAL_MS / 1000 + const pageKb = 4 + this.swapRate = { + inKBs: Math.max(0, Math.round(((next.memory.swap_in_pages || 0) - (prev.memory.swap_in_pages || 0)) * pageKb / secs)), + outKBs: Math.max(0, Math.round(((next.memory.swap_out_pages || 0) - (prev.memory.swap_out_pages || 0)) * pageKb / secs)) + } const prevDisks = {} ;(prev.disks || []).forEach(d => { prevDisks[d.name] = d }) this.diskRates = (next.disks || []).map(d => { From 190a5171336ef27a2093ecf125ef6abc7ba360b0 Mon Sep 17 00:00:00 2001 From: Boris Rybalkin Date: Thu, 9 Jul 2026 13:27:28 +0100 Subject: [PATCH 5/6] docs: remove completed TODO notes (zram done, health si/so done) --- TODO-zram-health.md | 55 --------------------------------------------- TODO.md | 16 ------------- 2 files changed, 71 deletions(-) delete mode 100644 TODO-zram-health.md delete mode 100644 TODO.md diff --git a/TODO-zram-health.md b/TODO-zram-health.md deleted file mode 100644 index 5e768d69..00000000 --- a/TODO-zram-health.md +++ /dev/null @@ -1,55 +0,0 @@ -# TODO: zram swap + Health-page bottleneck visibility - -## zram swap on high-RAM-but-overloaded boxes (boris prod) - -### Context -- boris.syncloud.it prod: 8 GB RAM (`MemTotal 8052952 kB`), ~15 snaps running (photoprism, mastodon, matrix, nextcloud, pihole, games, mail…). -- Only swap = legacy 953 MB disk partition `/dev/sda1`, 100% full (cold parked pages). -- Box was imaged from the **legacy amd64 image** (`debian-buster-amd64-8gb.img`); swap partition + fstab entry come from the image, NOT added manually. -- Modern amd64 pipeline (`image/tools/extract-amd64.sh` → debian-12-generic + EFI, `rootfs-amd64.sh`) ships **no swap** at all. arm/arm64 fstabs also have no swap. - -### Why platform zram is skipped here -`backend/stability/zram.go` `EnsureConfigured()`: -```go -memThresholdKB = 6 * 1024 * 1024 // 6 GiB -if snap.TotalKB > memThresholdKB { skip } -``` -Box has 8 GB > 6 GiB gate ⇒ zram never runs. Confirmed on prod: zram module not loaded, no `/dev/zram0`. - -### Decision: gate removed — zram always enabled (DONE) -`EnsureConfigured()` no longer checks memory; `memThresholdKB` is gone. zram is -configured on every box regardless of total/available RAM. Compressed swap is -cheap headroom even on large boxes, and `alreadyOn()` keeps it idempotent. -(Considered but rejected: Option A bump to 8 GiB; Option B gate on available RAM.) - -### Existing swap NOT auto-disabled — no extra code needed -- `disableFileSwaps()` only turns off `Type == "file"` swaps (`fields[1] != "file"` → skip). -- boris `/dev/sda1` is `Type == partition` ⇒ left alone. -- Result after enabling zram: zram prio 10 (used first) + sda1 prio -2 (overflow). Ideal. fstab re-adds sda1 on boot, stability service re-adds zram on boot. Stable, zero further changes. -- Only if we wanted to KILL sda1 too would we need code (extend disableFileSwaps to partitions — risky, affects all devices) or edit this box's fstab + `swapoff`. -- zram would size to 2 GiB cap (RAM/2=4G clamped), zstd comp_algorithm, prio 10. - -## Prove disk/swap is (or isn't) the bottleneck + surface it on Health page - -### Current evidence: NOT disk-bound (correction to earlier swap-thrash hypothesis) -Sampled prod ~12s + swap counters: -- `vmstat` si/so = **0/0** sustained → not paging in/out of swap. -- wa (iowait) = **0–1%**, idle 95–97%, load 0.2–0.6. -- swap 100% full but **static** (cold pages) — benign, normal. -- cumulative pswpin 92 days = 354k pages ≈ 1.4 GB (~15 MB/day) — negligible. -- The 100% swap BAR is misleading; RATE (si/so) + iowait are the real proof, both ~0 at rest. -- k9mail slowness more likely IMAP-over-TLS RTT or one-off Dovecot indexing. TODO: sample **during** a k9mail sync (si/so, wa, `/proc//io`). - -### Constraint -- PSI unavailable: kernel `4.19.0-8-amd64` (legacy buster), needs ≥4.20 + CONFIG_PSI. `/proc/pressure/*` absent. -- Note: `backend/stability/events.go` already has a `PSIavg10` field → newer kernels/boards would populate it, this box can't. - -### Health page work (`backend/health/metrics.go` + `web/platform/src/views/Health.vue`) -Backend already collects: `CPU.IOWait` (unused), `Disk.SectorsRead/Wrt` (shown as ↓/↑ KB/s), `Memory.SwapTotalKB/FreeKB` (swap bar). - -Add, in priority order: -1. **Swap-in/out rate (si/so)** — the definitive thrash signal, NOT collected. Add `swap_in_pages`/`swap_out_pages` (from `/proc/vmstat` pswpin/pswpout) to `Memory` struct; frontend computes delta via existing `prevMetrics` pattern (like diskRates) → show `swap ↓X · ↑Y KB/s`. -2. **IOWait %** — already in struct, just not shown. Compute `(iowait_delta/total_delta)*100`, render as its own bar or a segment of the CPU bar. -3. (optional) **PSI mem/io row** — degrade gracefully: hide when `/proc/pressure` absent (this box), show on newer boards. - -Goal: turn the misleading static "swap 100%" into signals that distinguish cold-parked-pages (harmless, current state) from active thrashing (real bottleneck). diff --git a/TODO.md b/TODO.md deleted file mode 100644 index 2a84833e..00000000 --- a/TODO.md +++ /dev/null @@ -1,16 +0,0 @@ -# TODO - -## Migrate 3rdparty dependencies to in-tree Docker builds - -Currently `btrfs` and `openldap` are downloaded as prebuilt binaries from -`github.com/syncloud/3rdparty` releases during the package step (`package.sh`). -This makes version upgrades harder because each dependency lives in a separate -repo with its own release workflow. - -**Goal:** Build btrfs and openldap inside Docker containers as part of the -platform CI pipeline (similar to how nginx and authelia are already built -in-tree), so upgrading a dependency is just a version bump in this repo. - -**Remaining downloads in `package.sh`:** -- `https://github.com/syncloud/3rdparty/releases/download/openldap/openldap-${ARCH}.tar.gz` -- `https://github.com/syncloud/3rdparty/releases/download/btrfs/btrfs-${ARCH}.tar.gz` From 53b508726948bee59456242fb8db75c6204aa2cf Mon Sep 17 00:00:00 2001 From: Boris Rybalkin Date: Thu, 9 Jul 2026 14:06:56 +0100 Subject: [PATCH 6/6] docs: keep TODO for btrfs/openldap in-tree migration --- TODO.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 TODO.md diff --git a/TODO.md b/TODO.md new file mode 100644 index 00000000..2a84833e --- /dev/null +++ b/TODO.md @@ -0,0 +1,16 @@ +# TODO + +## Migrate 3rdparty dependencies to in-tree Docker builds + +Currently `btrfs` and `openldap` are downloaded as prebuilt binaries from +`github.com/syncloud/3rdparty` releases during the package step (`package.sh`). +This makes version upgrades harder because each dependency lives in a separate +repo with its own release workflow. + +**Goal:** Build btrfs and openldap inside Docker containers as part of the +platform CI pipeline (similar to how nginx and authelia are already built +in-tree), so upgrading a dependency is just a version bump in this repo. + +**Remaining downloads in `package.sh`:** +- `https://github.com/syncloud/3rdparty/releases/download/openldap/openldap-${ARCH}.tar.gz` +- `https://github.com/syncloud/3rdparty/releases/download/btrfs/btrfs-${ARCH}.tar.gz`