[{"content":"The symptom PBS Sync Pull between two Proxmox Backup Servers. Configured, tested once, running. Three months later, a look at the task list:\nsyncjob ... error trying to connect: error:0A000086:SSL routines: tls_post_process_server_certificate:certificate verify failed 82 out of 82 runs failed. For months. No alert.\nThe cause The remote PBS uses Let\u0026rsquo;s Encrypt (ACME) for its web interface certificate. Let\u0026rsquo;s Encrypt certificates are valid for 90 days and typically renew automatically every 60.\nEvery renewal produces a new fingerprint. PBS Sync, however, verifies the remote host against the fingerprint stored in /etc/proxmox-backup/remote.cfg:\nremote: my-remote-pbs auth-id root@pam fingerprint AF:31:EB:5D:15:38:92:F9:... # \u0026lt;- stale after cert renewal host 192.168.66.50 password base64encodedpassword New certificate → new fingerprint → stored fingerprint no longer matches → SSL verify fails → sync dead.\nWhy nobody notices Because PBS distinguishes between backup tasks and sync tasks. Typical monitoring watches:\nPBS: Failed backup tasks count — counts only worker_type: backup PBS: Last failed backup task — surfaces only backup errors Sync jobs carry worker_type: syncjob. They slip through every backup-shaped monitoring net.\nBonus gotcha: without --all, proxmox-backup-manager task list shows only running tasks. A sync that dies after two seconds never appears:\n# running tasks only — a fast-failing sync is invisible here proxmox-backup-manager task list # ALL tasks, including finished ones proxmox-backup-manager task list --all --limit 20 The fix 1. Update the fingerprint by hand # pull the current fingerprint from the remote PBS openssl s_client -connect \u0026lt;remote-ip\u0026gt;:8007 \u0026lt;/dev/null 2\u0026gt;/dev/null \\ | openssl x509 -noout -fingerprint -sha256 # write it into remote.cfg nano /etc/proxmox-backup/remote.cfg # fingerprint \u0026lt;new-fingerprint\u0026gt; systemctl restart proxmox-backup-proxy 2. Automate it on cert renewal Updating a fingerprint by hand every 60 days? You will forget. Use a systemd path unit on the remote PBS to watch the certificate instead.\nOn the remote PBS (the one holding the Let\u0026rsquo;s Encrypt certificate):\n# /etc/systemd/system/pbs-cert-fingerprint.path [Unit] Description=Watch PBS certificate for renewal [Path] PathModified=/etc/proxmox-backup/proxy.pem Unit=pbs-cert-fingerprint.service [Install] WantedBy=multi-user.target # /etc/systemd/system/pbs-cert-fingerprint.service [Unit] Description=Update PBS fingerprint on pull host after cert renewal [Service] Type=oneshot ExecStartPre=/bin/sleep 10 ExecStart=/usr/local/bin/pbs-cert-notify-fingerprint.sh #!/bin/bash # /usr/local/bin/pbs-cert-notify-fingerprint.sh NEW_FP=$(openssl x509 -in /etc/proxmox-backup/proxy.pem \\ -noout -fingerprint -sha256 2\u0026gt;/dev/null | sed \u0026#39;s/.*=//\u0026#39;) ssh -o ConnectTimeout=10 root@\u0026lt;pull-host\u0026gt; \u0026#34; sed -i \\\u0026#34;s|fingerprint .*|fingerprint $NEW_FP|\\\u0026#34; \\ /etc/proxmox-backup/remote.cfg systemctl restart proxmox-backup-proxy \u0026#34; echo \u0026#34;$(date) Fingerprint updated: $NEW_FP\u0026#34; \u0026gt;\u0026gt; /var/log/pbs-cert-notify.log chmod +x /usr/local/bin/pbs-cert-notify-fingerprint.sh systemctl daemon-reload systemctl enable --now pbs-cert-fingerprint.path The path unit fires on every change to proxy.pem. The ten-second sleep gives PBS time to finish loading the new certificate.\nPrerequisite: SSH key auth from the remote PBS to the pull host has to be in place.\n3. Monitor syncs in Zabbix # /usr/local/bin/pbs-sync-check.sh #!/usr/bin/env python3 import json, subprocess, sys action = sys.argv[1] if len(sys.argv) \u0026gt; 1 else \u0026#34;status\u0026#34; try: out = subprocess.check_output( [\u0026#34;proxmox-backup-manager\u0026#34;, \u0026#34;task\u0026#34;, \u0026#34;list\u0026#34;, \u0026#34;--all\u0026#34;, \u0026#34;--limit\u0026#34;, \u0026#34;20\u0026#34;, \u0026#34;--output-format\u0026#34;, \u0026#34;json-pretty\u0026#34;], stderr=subprocess.DEVNULL ) tasks = json.loads(out) syncs = [t for t in tasks if \u0026#34;syncjob\u0026#34; in t.get(\u0026#34;upid\u0026#34;, \u0026#34;\u0026#34;) and t.get(\u0026#34;endtime\u0026#34;, \u0026#34;\u0026#34;)] status = syncs[0].get(\u0026#34;status\u0026#34;, \u0026#34;unknown\u0026#34;) if syncs else \u0026#34;none\u0026#34; except Exception: status = \u0026#34;unknown\u0026#34; if action == \u0026#34;status\u0026#34;: print(0 if status.startswith(\u0026#34;OK\u0026#34;) else (2 if status == \u0026#34;none\u0026#34; else 1)) elif action == \u0026#34;error\u0026#34;: print(\u0026#34;\u0026#34; if status.startswith(\u0026#34;OK\u0026#34;) or status == \u0026#34;none\u0026#34; else status) # /etc/zabbix/zabbix_agent2.d/pbs-sync-monitor.conf UserParameter=pbs.sync.status,sudo /usr/local/bin/pbs-sync-check.sh status UserParameter=pbs.sync.lasterror,sudo /usr/local/bin/pbs-sync-check.sh error Set the trigger to priority High. A failed sync is a failed backup.\nNamespace collision: one more gotcha If you pull PBS Sync into the root namespace of a datastore that also holds local backups, you get VMID collisions. CT 101 on host A and CT 101 on host B share the same backup group.\nPBS reports:\nsync group ct/101 failed - group lock failed Solution: give the sync pull its own namespace:\n# /etc/proxmox-backup/sync.cfg sync: my-sync-job ns remote-backups # \u0026lt;- dedicated namespace instead of root remote my-remote-pbs remote-ns remote-store my_storage_box store local_datastore TL;DR Let\u0026rsquo;s Encrypt on PBS means fingerprint rotation roughly every 60 days PBS Sync Pull verifies against the stored fingerprint, so it breaks Standard monitoring watches backup tasks, not sync tasks task list without --all shows running tasks only Fix: a systemd path unit for automatic fingerprint updates, plus Zabbix monitoring for syncjob tasks ","permalink":"https://homelab-guide.de/en/posts/proxmox-backup-server-sync-letsencrypt-fingerprint/","summary":"PBS Sync Pull breaks after every Let\u0026rsquo;s Encrypt certificate renewal — quietly, without an alert, for months. A systemd path unit on the remote host fixes it for good.","title":"PBS Sync Pull + Let's Encrypt: Why Your Backup Has Been Dead for 3 Months"},{"content":"The problem \u0026ldquo;Help me, Obi-Wan Kenobi. My UserParameter is not supported. You\u0026rsquo;re my only hope.\u0026rdquo;\nYou wrote a PowerShell script that produces metrics for Zabbix. The config sits in zabbix_agentd.d\\, the agent has been restarted. In Zabbix: ZBX_NOTSUPPORTED. Nothing in the agent log, no explanation.\nAnd then, hours later, once the items finally show up: Value \u0026quot;1,1\u0026quot; not suitable for Numeric float.\nTwo separate problems, both specific to Windows, neither obvious.\nPitfall 1: Zabbix Agent 2 silently ignores config files with LF line endings Symptom ZBX_NOTSUPPORTED: Unknown metric ga4.azcopy.upload_success You deployed the config to C:\\Program Files\\Zabbix Agent 2\\zabbix_agentd.d\\. The agent is running. Other UserParameters in the same directory work fine. Yours does not.\nDiagnosis The agent log looks unremarkable:\nzabbix_agent2.log: started zabbix_agent2.log: accepting connections Compare against a config that does work:\n# check the line endings of your own config $bytes = [System.IO.File]::ReadAllBytes(\u0026#34;C:\\Program Files\\Zabbix Agent 2\\zabbix_agentd.d\\my_config.conf\u0026#34;) # look for CR (0x0D) followed by LF (0x0A) $hasCRLF = $false for ($i = 0; $i -lt $bytes.Length - 1; $i++) { if ($bytes[$i] -eq 0x0D -and $bytes[$i+1] -eq 0x0A) { $hasCRLF = $true; break } } Write-Host \u0026#34;Has CRLF: $hasCRLF\u0026#34; Result: Has CRLF: False — the config uses Unix line endings (LF) instead of Windows ones (CRLF).\nCause Zabbix Agent 2 on Windows does not parse config files with LF line endings. There is no documented bug, no error message, and the agent starts normally. It simply never loads the file.\nWhy would the config have LF endings? Because it was written or edited on Linux, transferred with scp, or pulled from a Git repository without a .gitattributes.\nFix # in PowerShell on the target host $confPath = \u0026#34;C:\\Program Files\\Zabbix Agent 2\\zabbix_agentd.d\\my_config.conf\u0026#34; (Get-Content $confPath) | Set-Content $confPath Set-Content always writes CRLF. Then restart the agent:\nRestart-Service \u0026#34;Zabbix Agent 2\u0026#34; Or convert on the Linux side before deploying:\n# convert before the scp sed -i \u0026#39;s/\\r//\u0026#39; config.conf # make sure no stale CRs remain sed -i \u0026#39;s/$/\\r/\u0026#39; config.conf # LF -\u0026gt; CRLF scp config.conf user@windows-host:\u0026#34;C:/Program Files/Zabbix Agent 2/zabbix_agentd.d/\u0026#34; Verification # test the agent directly (as Administrator) \u0026amp; \u0026#34;C:\\Program Files\\Zabbix Agent 2\\zabbix_agent2.exe\u0026#34; -t \u0026#34;my.custom.key\u0026#34; Expected output once fixed:\nmy.custom.key [s|1] Pitfall 2: non-English Windows, PowerShell and the decimal comma Symptom Three of four items deliver data. The fourth:\nValue of type \u0026#34;string\u0026#34; is not suitable for value type \u0026#34;Numeric (float)\u0026#34;. Value \u0026#34;1,1\u0026#34; Your script emits 1,1 where Zabbix wants 1.1.\nCause PowerShell honours the Windows system locale. On German Windows — and on French, Spanish, Italian, Dutch, Polish, Brazilian Portuguese and most of continental Europe — the decimal separator is a comma, not a period. Zabbix expects a period, always, regardless of the agent\u0026rsquo;s system language.\nThis hits every operation that emits a decimal number:\n# German Windows — prints \u0026#34;1,5\u0026#34; $hours = [math]::Round(1.5, 1) Write-Output $hours # -\u0026gt; \u0026#34;1,5\u0026#34; # an explicit ToString does not necessarily help either Write-Output $hours.ToString() # -\u0026gt; \u0026#34;1,5\u0026#34; If you build your monitoring on an English-locale machine and roll it out to a fleet with mixed locales, this shows up on some hosts and not on others — which makes it look like a script bug rather than a locale problem.\nFix # InvariantCulture forces the period as decimal separator $hours = [math]::Round(((Get-Date) - $logFile.LastWriteTime).TotalHours, 1) Write-Output ($hours.ToString([System.Globalization.CultureInfo]::InvariantCulture)) Apply this to every numeric output of a Zabbix UserParameter script that returns decimal values on a non-English Windows host.\nBonus trap: two Zabbix agent services If restarting the service changes nothing at all, check that you are restarting the right one.\nGet-Service *zabbix* Status Name DisplayName ------ ---- ----------- Stopped ZabbixAgent Zabbix Agent Running ZabbixAgent2 Zabbix Agent 2 Windows hosts that have been around a while sometimes carry both versions. The old \u0026ldquo;Zabbix Agent\u0026rdquo; sits there stopped, and restarts aimed at it accomplish nothing.\nThe active service is named \u0026quot;Zabbix Agent 2\u0026quot; (with spaces), not \u0026quot;ZabbixAgent2\u0026quot;:\nRestart-Service \u0026#34;Zabbix Agent 2\u0026#34; Checklist: debugging UserParameters on Windows When a UserParameter returns ZBX_NOTSUPPORTED:\nDoes the config file use CRLF line endings? (run it through Set-Content) Did you restart the right service? (check Get-Service *zabbix*) Test the agent on the host directly: zabbix_agent2.exe -t \u0026quot;my.key\u0026quot; Run the script by hand: powershell -File \u0026quot;C:\\Scripts\\my_script.ps1\u0026quot; -Metric test When the item delivers data but is flagged as an error:\nDecimal values: ToString([System.Globalization.CultureInfo]::InvariantCulture) Integers: [int] or [math]::Round(..., 0) are safe Item type correct? Numeric (float) for decimals, Numeric (unsigned) for integers TL;DR CRLF: Zabbix Agent 2 on Windows does not load LF configs — no error, just ignored. Fix: (Get-Content config.conf) | Set-Content config.conf Decimal separator: PowerShell on a non-English Windows emits 1,1 where Zabbix wants 1.1. Fix: .ToString([System.Globalization.CultureInfo]::InvariantCulture) Service: Get-Service *zabbix* tells you whether you are restarting the right one. ","permalink":"https://homelab-guide.de/en/posts/zabbix-agent-2-windows-userparameter-not-working/","summary":"You wrote a UserParameter script for Zabbix Agent 2 on Windows, deployed the config, restarted the agent — and nothing happens. No items, no error, just silence. Two causes, neither of them obvious.","title":"Zabbix Agent 2 on Windows: Two Pitfalls That Silence Your UserParameter"},{"content":"Plausible Analytics is a privacy-friendly alternative to Google Analytics — no cookie banner required, GDPR-compliant, and you can run it yourself. Version 3 ships a completely rewritten frontend built on Phoenix LiveView, and that changes a few things nobody wrote down.\nHere are the four problems I ran into during setup.\nTL;DR Problem Symptom Fix ClickHouse 24.12: user-level settings rejected Container refuses to start Remove user-level settings from the XML ClickHouse default user has no network access CLICKHOUSE_DATABASE_URL fails Set the CLICKHOUSE_PASSWORD env var docker compose restart does not re-read env_file Env changes have no effect Use docker compose up -d instead of restart Phoenix LiveView needs WebSocket Login button does nothing Add location /live/websocket to nginx Pitfall 1: ClickHouse 24.12 rejects user-level settings The official Plausible docs suggest running ClickHouse with a low-resources.xml on smaller servers. Older versions of that config carried settings such as max_bytes_before_external_group_by or max_bytes_before_external_sort. As of ClickHouse 24.12, those settings are no longer allowed in user profiles.\nSymptom: the ClickHouse container never comes up. From the logs:\nException: Setting max_bytes_before_external_group_by is not allowed in user profiles. Fix: drop those settings from the XML. A low-resources.xml that works with ClickHouse 24.12 looks like this:\n\u0026lt;clickhouse\u0026gt; \u0026lt;max_concurrent_queries\u0026gt;10\u0026lt;/max_concurrent_queries\u0026gt; \u0026lt;max_server_memory_usage_to_ram_ratio\u0026gt;0.4\u0026lt;/max_server_memory_usage_to_ram_ratio\u0026gt; \u0026lt;mark_cache_size\u0026gt;536870912\u0026lt;/mark_cache_size\u0026gt; \u0026lt;/clickhouse\u0026gt; Pitfall 2: the ClickHouse default user has no network access After fixing pitfall 1, ClickHouse starts — and Plausible still cannot connect. Without an explicit password, the default user in ClickHouse 24.12 gets no network access at all.\nSymptom: Plausible boots, but every event request fails. From the logs:\n[error] Clickhouse.Error: Code: 516. DB::Exception: default: Authentication failed Fix: set the CLICKHOUSE_PASSWORD environment variable in your compose.yml:\nplausible_events_db: image: clickhouse/clickhouse-server:24.12-alpine environment: - CLICKHOUSE_PASSWORD=your-secure-password And match it in the Plausible configuration:\nCLICKHOUSE_DATABASE_URL=http://default:your-secure-password@plausible_events_db:8123/plausible_events Pitfall 3: docker compose restart does not re-read env_file Change something in .env or plausible-conf.env, run docker compose restart, and nothing happens. The container restarts, but the environment variables still come from the existing container state.\nSymptom: edits to DISABLE_REGISTRATION, SECRET_KEY_BASE or database URLs have no effect after docker compose restart.\nThis bites hardest during first-time setup, where it creates a chicken-and-egg problem: DISABLE_REGISTRATION=invite_only blocks you from creating the very first account. You flip it to false — and restart never picks the change up.\nFix: use docker compose up -d instead of restart:\n# WRONG: container restarts, env_file is NOT re-read docker compose restart # RIGHT: container is recreated, env_file is re-read docker compose up -d up -d detects changes to compose.yml and env_file and recreates only the affected containers. restart deliberately does not, and Docker documents it: \u0026ldquo;If you make changes to your compose.yml configuration, these changes are not reflected after running this command. For example, changes to environment variables [\u0026hellip;] are not updated after restarting.\u0026rdquo; By design, not a bug.\nPitfall 4: Phoenix LiveView needs WebSocket in nginx Plausible v3 was rewritten on Elixir/Phoenix and drives its interactive frontend through Phoenix LiveView. LiveView talks over WebSockets rather than plain HTTP requests.\nSymptom: the Plausible dashboard is reachable, the login page renders — and clicking \u0026ldquo;Login\u0026rdquo; or \u0026ldquo;Create Account\u0026rdquo; does nothing. No HTTP error, no redirect, just silence. Browser DevTools shows a failed WebSocket attempt against /live/websocket.\nFix: proxy /live/websocket explicitly, with the WebSocket upgrade headers:\nserver { listen 443 ssl http2; server_name stats.example.com; # WebSocket for Phoenix LiveView location /live/websocket { proxy_pass http://plausible-backend:8000; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection \u0026#34;Upgrade\u0026#34;; proxy_set_header Host $host; } # everything else location / { proxy_pass http://plausible-backend:8000; proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } } Without the location /live/websocket block, nginx never upgrades the connection, and every form submit in Plausible v3 fails silently.\nVerification # 1. all containers up and healthy? docker compose ps # 2. Plausible reachable? curl -s -o /dev/null -w \u0026#34;%{http_code}\u0026#34; https://stats.example.com # expected: 200 # 3. send a pageview by hand (in case an ad blocker eats the script) curl -s -o /dev/null -w \u0026#34;%{http_code}\u0026#34; \\ \u0026#34;https://stats.example.com/api/event\u0026#34; \\ -H \u0026#34;Content-Type: application/json\u0026#34; \\ -d \u0026#39;{\u0026#34;name\u0026#34;:\u0026#34;pageview\u0026#34;,\u0026#34;url\u0026#34;:\u0026#34;https://example.com/\u0026#34;,\u0026#34;domain\u0026#34;:\u0026#34;example.com\u0026#34;}\u0026#39; # expected: 202 Takeaway Self-hosting Plausible v3 is doable. What makes it slow is the combination of ClickHouse 24.12 breaking changes and the new Phoenix LiveView frontend, neither of which the documentation covers yet.\nTwo habits save the most time: reach for docker compose up -d instead of restart whenever environment variables change, and check WebSocket support in nginx whenever you put a LiveView app behind a proxy.\n","permalink":"https://homelab-guide.de/en/posts/plausible-analytics-v3-self-hosting-pitfalls/","summary":"Self-hosting Plausible v3 with ClickHouse 24.12 sounds like an hour of work. Four non-obvious problems are waiting: ClickHouse breaking changes, how Docker Compose actually treats env_file, the WebSocket requirement of Phoenix LiveView, and an nginx trap.","title":"Self-Hosting Plausible Analytics v3: Four Pitfalls That Cost Me Hours"}]