The symptom

PBS Sync Pull between two Proxmox Backup Servers. Configured, tested once, running. Three months later, a look at the task list:

syncjob ... error trying to connect: error:0A000086:SSL routines:
tls_post_process_server_certificate:certificate verify failed

82 out of 82 runs failed. For months. No alert.

The cause

The remote PBS uses Let’s Encrypt (ACME) for its web interface certificate. Let’s Encrypt certificates are valid for 90 days and typically renew automatically every 60.

Every renewal produces a new fingerprint. PBS Sync, however, verifies the remote host against the fingerprint stored in /etc/proxmox-backup/remote.cfg:

remote: my-remote-pbs
    auth-id root@pam
    fingerprint AF:31:EB:5D:15:38:92:F9:...  # <- stale after cert renewal
    host 192.168.66.50
    password base64encodedpassword

New certificate → new fingerprint → stored fingerprint no longer matches → SSL verify fails → sync dead.

Why nobody notices

Because PBS distinguishes between backup tasks and sync tasks. Typical monitoring watches:

  • PBS: Failed backup tasks count — counts only worker_type: backup
  • PBS: Last failed backup task — surfaces only backup errors

Sync jobs carry worker_type: syncjob. They slip through every backup-shaped monitoring net.

Bonus gotcha: without --all, proxmox-backup-manager task list shows only running tasks. A sync that dies after two seconds never appears:

# running tasks only — a fast-failing sync is invisible here
proxmox-backup-manager task list

# ALL tasks, including finished ones
proxmox-backup-manager task list --all --limit 20

The fix

1. Update the fingerprint by hand

# pull the current fingerprint from the remote PBS
openssl s_client -connect <remote-ip>:8007 </dev/null 2>/dev/null \
  | openssl x509 -noout -fingerprint -sha256

# write it into remote.cfg
nano /etc/proxmox-backup/remote.cfg
# fingerprint <new-fingerprint>

systemctl restart proxmox-backup-proxy

2. Automate it on cert renewal

Updating a fingerprint by hand every 60 days? You will forget. Use a systemd path unit on the remote PBS to watch the certificate instead.

On the remote PBS (the one holding the Let’s Encrypt certificate):

# /etc/systemd/system/pbs-cert-fingerprint.path
[Unit]
Description=Watch PBS certificate for renewal

[Path]
PathModified=/etc/proxmox-backup/proxy.pem
Unit=pbs-cert-fingerprint.service

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/pbs-cert-fingerprint.service
[Unit]
Description=Update PBS fingerprint on pull host after cert renewal

[Service]
Type=oneshot
ExecStartPre=/bin/sleep 10
ExecStart=/usr/local/bin/pbs-cert-notify-fingerprint.sh

#!/bin/bash
# /usr/local/bin/pbs-cert-notify-fingerprint.sh

NEW_FP=$(openssl x509 -in /etc/proxmox-backup/proxy.pem \
  -noout -fingerprint -sha256 2>/dev/null | sed 's/.*=//')

ssh -o ConnectTimeout=10 root@<pull-host> "
  sed -i \"s|fingerprint .*|fingerprint $NEW_FP|\" \
    /etc/proxmox-backup/remote.cfg
  systemctl restart proxmox-backup-proxy
"

echo "$(date) Fingerprint updated: $NEW_FP" >> /var/log/pbs-cert-notify.log

chmod +x /usr/local/bin/pbs-cert-notify-fingerprint.sh
systemctl daemon-reload
systemctl enable --now pbs-cert-fingerprint.path

The path unit fires on every change to proxy.pem. The ten-second sleep gives PBS time to finish loading the new certificate.

Prerequisite: SSH key auth from the remote PBS to the pull host has to be in place.

3. Monitor syncs in Zabbix

# /usr/local/bin/pbs-sync-check.sh
#!/usr/bin/env python3
import json, subprocess, sys

action = sys.argv[1] if len(sys.argv) > 1 else "status"
try:
    out = subprocess.check_output(
        ["proxmox-backup-manager", "task", "list", "--all",
         "--limit", "20", "--output-format", "json-pretty"],
        stderr=subprocess.DEVNULL
    )
    tasks = json.loads(out)
    syncs = [t for t in tasks
             if "syncjob" in t.get("upid", "") and t.get("endtime", "")]
    status = syncs[0].get("status", "unknown") if syncs else "none"
except Exception:
    status = "unknown"

if action == "status":
    print(0 if status.startswith("OK") else (2 if status == "none" else 1))
elif action == "error":
    print("" if status.startswith("OK") or status == "none" else status)

# /etc/zabbix/zabbix_agent2.d/pbs-sync-monitor.conf
UserParameter=pbs.sync.status,sudo /usr/local/bin/pbs-sync-check.sh status
UserParameter=pbs.sync.lasterror,sudo /usr/local/bin/pbs-sync-check.sh error

Set the trigger to priority High. A failed sync is a failed backup.

Namespace collision: one more gotcha

If you pull PBS Sync into the root namespace of a datastore that also holds local backups, you get VMID collisions. CT 101 on host A and CT 101 on host B share the same backup group.

PBS reports:

sync group ct/101 failed - group lock failed

Solution: give the sync pull its own namespace:

# /etc/proxmox-backup/sync.cfg
sync: my-sync-job
    ns remote-backups       # <- dedicated namespace instead of root
    remote my-remote-pbs
    remote-ns
    remote-store my_storage_box
    store local_datastore

TL;DR

  1. Let’s Encrypt on PBS means fingerprint rotation roughly every 60 days
  2. PBS Sync Pull verifies against the stored fingerprint, so it breaks
  3. Standard monitoring watches backup tasks, not sync tasks
  4. task list without --all shows running tasks only
  5. Fix: a systemd path unit for automatic fingerprint updates, plus Zabbix monitoring for syncjob tasks