Operate Argus
This page covers the day-to-day running of an Argus site: health checks, logs, retention, backups and troubleshooting.
Run the commands from the release folder on the site node. They use the settings and files listed in your .env, as set up in the install.
Check the health of a site
Argus has no metrics endpoint. Use these checks, or wire them into your own host monitoring.
| Check | How | Healthy result |
|---|---|---|
| Containers | docker compose ps | Every service is running. postgres, redis, seaweedfs, ollama and caddy show healthy. |
| Site API | curl -fsS https://<site-node>/health | {"status":"ok"}. This shows that Caddy and the API process are up. It doesn't check the other services. With the site certificate authority, add --cacert argus-root-ca.crt. |
| Edge nodes | Cameras & Edge Nodes in the dashboard | Every edge node shows as Online. A node checks in every 10 seconds and shows as Offline after 30 seconds without a check-in. A node that runs but can't reach Redis shows as Redis unreachable. |
| Site status | System Health at the bottom of the dashboard sidebar | All systems operational. Otherwise it names the problem, for example an edge node that is Redis unreachable, or event stream ID poisoned. |
| AI models | docker compose exec ollama ollama list | gemma3:4b and nomic-embed-text are listed. |
| Disk | df -h /var/lib/docker | Enough free space. See Disk space. |
Event pipeline
Detection events and clip messages travel through Redis streams. Each site service reads them as a consumer group. To see whether a service is falling behind, check the groups:
REDIS_PASSWORD=$(grep '^REDIS_PASSWORD=' .env | cut -d= -f2-)
docker compose exec redis redis-cli -a "$REDIS_PASSWORD" --no-auth-warning XINFO GROUPS edge-events
docker compose exec redis redis-cli -a "$REDIS_PASSWORD" --no-auth-warning XINFO GROUPS clip-ready
docker compose exec redis redis-cli -a "$REDIS_PASSWORD" --no-auth-warning XINFO GROUPS clip-attached
| Stream | Groups | Read by |
|---|---|---|
edge-events | site-node-consumers, rule-engine-consumers | site-consumer, site-rule-engine |
clip-ready | clip-updater-consumers | site-clip-updater |
clip-attached | vlm-captioner-consumers | site-vlm-captioner |
Edge nodes write to edge-events and clip-ready. clip-attached is written only by the site, after it has checked a clip and attached it to its event.
For each group, lag is the number of messages it still has to read. A lag that keeps growing means that service can't keep up, or has stopped.
To check Redis memory:
docker compose exec redis redis-cli -a "$REDIS_PASSWORD" --no-auth-warning INFO memory | grep used_memory_human
The edge nodes cap each stream at a fixed number of recent messages, so Redis memory stays bounded.
To see how far captioning has got, count the alert clips by caption status:
docker compose exec postgres psql -U site -d site -c \
"SELECT caption_status, count(*) FROM events WHERE clip_uri IS NOT NULL GROUP BY 1;"
Repair event stream IDs
A misbehaving or tampered edge node can give the event messages it sends IDs far in the future. Events keep arriving and are processed in order, so this is not urgent. But the site logs a SECURITY line, and System Health shows "event stream ID poisoned" until you repair the stream. A healthy edge node never does this, so also check your edge hosts.
The repair runs offline, in a short maintenance window. Cameras are not watched while the edge nodes are stopped.
-
Stop every edge node:
docker compose stop edge-nodeon the site node, anddocker compose stopon each edge host. Wait until Cameras & Edge Nodes shows them all offline, which takes about 30 seconds. -
On the site node, from the release folder, see what the repair would do. This is a dry run and changes nothing:
scripts/repair_stream_ids.sh -
Apply it:
scripts/repair_stream_ids.sh --applyIt refuses to run while an edge node still checks in. It rebuilds each affected stream with fresh IDs in one step. It keeps every message, and the position of each site service, so nothing is skipped or processed twice. Messages that a service had read but not finished are handed to it again.
-
Start the edge nodes again:
docker compose start edge-nodeon the site node, anddocker compose up -don each edge host. -
Check that System Health no longer shows the message.
If your docker compose command is different, for example docker-compose, set it in ARGUS_COMPOSE when you run the script.
Logs
Each service logs to Docker. To follow one service:
docker compose logs -f <service>
| Service | Look here for |
|---|---|
edge-node | Cameras starting and stopping, stream errors, clip uploads. |
site-consumer | Detections being written to the database. |
site-rule-engine | Rules loaded and alerts raised. |
site-clip-updater | Clips linked to alerts. |
site-vlm-captioner | Captions and caption failures. |
site-retention | Each retention sweep and what it deleted. |
site-api | Dashboard and API requests. |
caddy | HTTPS, certificates and live video proxying. |
migrate | Schema updates at start. |
Docker's default log driver doesn't rotate logs. To keep logs from filling the disk, set rotation for the Docker daemon in /etc/docker/daemon.json, for example:
{
"log-driver": "json-file",
"log-opts": { "max-size": "50m", "max-file": "5" }
}
Restart Docker, then recreate the containers with docker compose up -d --force-recreate so the setting applies to them.
Some services print their connection settings when they start. Treat the logs as sensitive, and limit who can read them.
Retention
Argus deletes old events and their clips on a schedule. An admin sets the policy in the dashboard under Settings > Retention Policy. Retention explains the policy in full. In short:
| Setting | Default | What it does |
|---|---|---|
| Retention window (days) | 30 | Events older than this are deleted, with their clips, unless an open alert still points to them. |
| Hard backstop (days) | Off (blank) | Events older than this are deleted even if an open alert points to them. It must be longer than the retention window. |
| Sweep interval (hours) | 6 | How often the retention service runs. |
Alerts that are acknowledged, resolved or dismissed don't protect their events. Open alerts do, until the hard backstop if you set one.
The retention service reads the settings at the start of each sweep. It sweeps once when it starts, then once per interval. To apply a change straight away, restart it:
docker compose restart site-retention
To check that sweeps finish, look for sweep complete in its log:
docker compose logs site-retention | grep "sweep complete"
If site-retention keeps restarting, or no sweep completes, the old data is not deleted. Check the log for the error and contact us.
If nobody triages alerts at a site, open alerts keep their clips forever. Set a hard backstop so that storage doesn't grow without limit.
Back up a site
| What | Where it lives | How to back it up |
|---|---|---|
| Site database | Volume argus_pgdata | pg_dump, below. |
| Alert clips | Volume argus_seaweedfs-data | Copy the clip storage data, below. |
| HTTPS certificates | Volume argus_caddy-data, and certs/ if you use your own certificate | Copy the volume, below. It holds the site certificate authority. A new one means installing a new root certificate on every computer. |
| Your settings | .env and any Compose file you added on the site node, plus .env and docker-compose.yml on each edge host | Copy the files. They contain your secrets, so store them safely. |
| AI models | Volume argus_ollama-data | Optional. You can pull them again, or keep your offline export. |
| Redis | Container | Not needed. It only holds messages in transit. |
Back up the database and the clips at the same time, so that they match. Put each backup in a new folder:
BACKUP=backup/$(date +%Y%m%d-%H%M)
mkdir -p "$BACKUP"
Back up the database
docker compose exec -T postgres pg_dump -U site -d site -Fc > "$BACKUP/argus-db.dump"
You can run this while Argus is running.
Back up the clips
Stop the clip storage, copy its data, then start it again:
docker compose stop seaweedfs
docker compose cp seaweedfs:/data "$BACKUP/seaweedfs"
docker compose start seaweedfs
While the clip storage is stopped, edge nodes can't upload clips. Alerts raised during that time have no clip.
Back up the certificates
docker compose cp caddy:/data "$BACKUP/caddy"
Copy each backup folder to storage other than the site node.
Restore a site
Restoring replaces the current database and clips with the backup. Set BACKUP to the folder of the backup you want to restore, for example BACKUP=backup/20261001-0200.
-
Stop Argus:
docker compose down -
Replace the clip storage with the backup:
docker volume rm argus_seaweedfs-data
docker compose create seaweedfs
docker compose cp "$BACKUP/seaweedfs/." seaweedfs:/data/ -
Start the database and restore it:
docker compose up -d --wait postgres
docker compose exec -T postgres pg_restore -U site -d site --clean --if-exists < "$BACKUP/argus-db.dump" -
Start everything else:
docker compose up -d
To restore onto a new host, install Argus first with the same .env, then follow these steps. To keep the site certificate authority, also restore the certificates before you start Caddy: docker compose create caddy, then docker compose cp "$BACKUP/caddy/." caddy:/data/.
Disk space
Watch free disk space on the site node. The site database stores every detection event, and clip storage grows with the number of alerts. Retention deletes both after the retention window.
To see how much space each volume uses:
docker system df -v
When the disk is full, Redis and PostgreSQL stop accepting writes, and the site stops recording detections and alerts. Free some space, then check the services with docker compose ps.
GPU and CPU
| Part | CPU host | GPU host, with docker-compose.gpu.yml |
|---|---|---|
| Detection and tracking (edge node) | Run on CPU. Fewer cameras per edge node. | Run on the GPU. |
| Captions and Copilot (Ollama) | Run on CPU. Answers take longer. | Run on the GPU. |
| Database, dashboard and camera views | CPU. | CPU. |
The edge node uses the GPU when one is visible to it, and falls back to CPU otherwise. To check which it uses, start a camera and look for device=cuda or device=cpu in docker compose logs edge-node.
How the AI models behave:
- The caption and Copilot models are local language models served by Ollama. You choose them with
VISION_MODELandCOPILOT_MODELin.env. Pull any model you set withdocker compose exec ollama ollama pull <model>. - Ollama loads a model when it is first needed and unloads it after it has been idle for a while. The first caption or Copilot answer after that waits for the model to load. To keep models loaded, add a Compose file that sets
OLLAMA_KEEP_ALIVEon theollamaservice, for exampleOLLAMA_KEEP_ALIVE: "24h", and add it toCOMPOSE_FILE. - Each call to the model has a time limit. If the Copilot model doesn't answer in time, the Copilot replies with a summary of recent activity from the database instead of a written answer.
- If a caption request fails or times out, Argus tries once more with a single frame, then marks the caption as failed. The alert and its clip are not affected.
Troubleshooting
| Problem | What to check |
|---|---|
| The dashboard doesn't open | docker compose ps caddy site-api. The dashboard is at https://<site-node>/. The browser can reach the site node on port 443. Check docker compose logs caddy for certificate errors. |
| The browser warns about the certificate | With the default mode, install the site's root certificate on that computer. See Trust the site certificate authority. With SITE_DOMAIN, check that the DNS name points at the site node and that ports 80 and 443 are reachable from the internet. |
| Sign-in fails and the form says the connection isn't secure | The browser reached the dashboard over plain HTTP. Open it at https://. |
| Nothing starts after an upgrade | docker compose logs migrate. The schema update failed, so the other services wait. |
| A service keeps restarting | docker compose logs <service> for the error. site-api refuses to start without a valid SESSION_SECRET or LIVE_STREAM_SECRET, and seaweedfs without S3_ACCESS_KEY and S3_SECRET_KEY. Check .env against .env.example. |
| No captions | Both Ollama models are listed by docker compose exec ollama ollama list. Check docker compose logs site-vlm-captioner. On CPU, captions take longer and can fail on the time limit. |
Copilot answers start with System online: followed by an error message | Copilot hit an error, and the message after System online: describes it. The rest of that reply is fixed text and doesn't reflect the state of your cameras or services. Check docker compose logs site-api, and check that postgres and ollama are healthy and the Copilot model is pulled. |
| An edge node shows as Redis unreachable | The node runs, but can't deliver detections, and drops them. Check that redis is healthy, that the edge host can reach port 6379, and that its EDGE_REDIS_PASSWORD matches the site node's. See If the edge node can't reach Redis. |
| System Health shows "event stream ID poisoned" | An edge node sent event messages with IDs far in the future. Events still arrive. Repair it in a maintenance window. See Repair event stream IDs. |
| Copilot is slow, or gives activity summaries instead of answers | The model is loading, or the host has no GPU. See GPU and CPU. |
For camera and edge node connection problems, see Troubleshoot camera connections.
Alert clips
A clip is cut by the edge node that watches the camera. It covers 5 seconds before and 5 seconds after the trigger, so it is ready a few seconds after the alert appears.
| Problem | What to check |
|---|---|
| An alert has no clip | The edge node keeps only the last 15 seconds of video in memory. If it restarted, or fell behind, around the time of the alert, the clip is skipped and its log says clip skipped. Alerts raised while the clip storage was stopped also have no clip. |
| Older clips don't play after an upgrade | The clips are still in the old MinIO volume. Run the copy script. See Upgrade from a release with MinIO and plain HTTP. |
| Clip playback fails for every alert | docker compose ps seaweedfs shows healthy. S3_ACCESS_KEY and S3_SECRET_KEY in .env haven't changed since the first start. Check docker compose logs site-api. See Clip playback and live video. |