API Documentation
Everything you need to integrate with labwatch. The API uses JSON for request/response bodies and standard HTTP status codes.
Overview
The labwatch API is organized around REST. All endpoints accept and return JSON (except where noted). The base URL for all API requests is your labwatch server instance.
Rate limits: agents can ingest once per minute per node. API queries are limited to 60 requests per minute per admin key.
Authentication
labwatch uses two authentication methods depending on the endpoint:
JWT Token (Account Users)
Account users authenticate via JWT. After logging in at /api/v1/auth/login, include the token as a cookie or header:
Tokens expire after 30 days. Most /api/v1/account/* endpoints use JWT auth.
Agent Bearer Token
Agents authenticate with a Bearer token received during registration. Include it in the Authorization header:
Admin Secret
Admin endpoints require the X-Admin-Secret header. This is set during server setup and gives full access to all data and configuration.
Quick Start
Get monitoring running in under 2 minutes:
1. Sign up for a free account
curl -X POST https://labwatch.zazastation.duckdns.org/api/v1/signup \
-H 'Content-Type: application/json' \
-d '{"email": "you@example.com", "hostname": "my-server"}'You'll receive an admin_secret, lab_id, and token in the response.
2. Install the agent
3. Configure and start
# Edit /etc/labwatch/config.yaml with your server URL and token
labwatch --register
systemctl enable --now labwatchMetrics will start flowing within 60 seconds. Open your dashboard to see them.
Create Account
Create a full account with email and password. Returns a JWT for immediate use. A verification email will be sent.
Request Body
| Field | Type | Required | Description |
|---|---|---|---|
| string | required | Your email address | |
| password | string | required | Password (minimum 8 characters) |
| name | string | optional | Display name |
Example
curl -X POST https://labwatch.zazastation.duckdns.org/api/v1/auth/signup \
-H 'Content-Type: application/json' \
-d '{"email": "you@example.com", "password": "secure-pass-123"}'Response 200
{
"account_id": "acc_...",
"email": "you@example.com",
"plan": "free",
"token": "eyJhbGci..."
}Login
Login with email and password. Returns a JWT valid for 30 days. Rate limited: 5 failed attempts per email, 20 per IP in a 15-minute window.
Request Body
| Field | Type | Required | Description |
|---|---|---|---|
| string | required | Your email address | |
| password | string | required | Your password |
Response 200
{
"account_id": "acc_...",
"email": "you@example.com",
"plan": "homelab",
"token": "eyJhbGci..."
}Error 429
{
"detail": "Too many login attempts. Try again in 15 minutes."
}Forgot Password
Sends a password reset link to the registered email. Always returns 200 regardless of whether the email exists (prevents email enumeration).
Request Body
| Field | Type | Required | Description |
|---|---|---|---|
| string | required | Account email address |
Response 200
{
"message": "If that email is registered, a reset link was sent."
}Reset Password
Reset your password using a valid token from the reset email. Token is single-use and expires after 1 hour.
Request Body
| Field | Type | Required | Description |
|---|---|---|---|
| token | string | required | Reset token from email link |
| new_password | string | required | New password (minimum 8 characters) |
Response 200
{
"message": "Password reset successfully."
}Update Account
Update your account's display name.
Request Body
| Field | Type | Required | Description |
|---|---|---|---|
| name | string | optional | New display name (max 100 characters) |
Response 200
{
"message": "Account updated."
}Change Password
Change your password. Requires your current password for verification.
Request Body
| Field | Type | Required | Description |
|---|---|---|---|
| current_password | string | required | Your current password |
| new_password | string | required | New password (minimum 8 characters) |
Response 200
{
"message": "Password updated."
}Error 401
{
"detail": "Current password is incorrect"
}Delete Account
Permanently deletes your account and all associated data: labs, metrics, alert rules, notification channels, status pages, and incidents. This cannot be undone.
Request Body
| Field | Type | Required | Description |
|---|---|---|---|
| password | string | required | Your password (confirmation) |
Response 200
{
"message": "Account and all associated data deleted."
}Add Lab
Add a new monitored node to your account. Returns the lab ID and agent token. Subject to plan limits on maximum nodes.
Request Body
| Field | Type | Required | Description |
|---|---|---|---|
| hostname | string | optional | Node hostname (default: "my-server") |
Response 200
{
"lab_id": "lab_proxmox-01_x7k2",
"token": "lw_...",
"install_command": "curl -fsSL .../install.sh | sudo bash",
"config_snippet": "api_endpoint: ...\ntoken: ...\nlab_id: ...\ninterval: 60s",
"message": "Lab registered."
}Plan Limits
| Plan | Max Nodes |
|---|---|
| Free | 1 |
| Homelab | 10 |
| Pro | 50 |
| Business | Unlimited |
Rename Lab
Update the hostname of one of your labs. Only alphanumeric characters, dots, hyphens, and underscores are allowed.
Request Body
| Field | Type | Required | Description |
|---|---|---|---|
| hostname | string | required | New hostname (max 64 characters) |
Response 200
{
"message": "Lab updated."
}Regenerate Token
Generates a new agent token for the lab. The old token is invalidated immediately. You'll need to update the agent's config file with the new token.
Response 200
{
"token": "lw_new_token...",
"config_snippet": "api_endpoint: ...\ntoken: ...\nlab_id: ...\ninterval: 60s",
"message": "Token regenerated. Update your agent config with the new token."
}Delete Lab (Account)
Permanently removes a lab and all associated metrics, alerts, and digests. This cannot be undone.
Response 200
{
"message": "Lab and all associated data deleted."
}Signup
Create a new free-tier account. Returns credentials for the admin dashboard and a pre-registered first node.
Request Body
| Field | Type | Required | Description |
|---|---|---|---|
| string | required | Your email address | |
| hostname | string | optional | Name for your first node (default: "my-server") |
Example
curl -X POST https://your-server/api/v1/signup \
-H 'Content-Type: application/json' \
-d '{"email": "admin@lab.local", "hostname": "proxmox-01"}'Response 200
{
"message": "Account created successfully",
"admin_secret": "abc123...",
"lab_id": "lab_proxmox-01_x7k2",
"token": "lw_...",
"dashboard_url": "/dashboard?secret=abc123..."
}Register Agent
Register a new node for monitoring. The agent calls this on first startup to get its credentials.
Headers
| Header | Value |
|---|---|
| X-Admin-Secret | Your admin secret |
Request Body
| Field | Type | Required | Description |
|---|---|---|---|
| hostname | string | required | Hostname of the node |
| os | string | required | Operating system (e.g., "linux") |
| arch | string | required | Architecture (e.g., "amd64", "arm64") |
| agent_version | string | required | Agent version string |
Response 200
{
"lab_id": "lab_proxmox-01_x7k2",
"token": "lw_...",
"message": "Registered successfully"
}Ingest Metrics
Submit a metrics snapshot from a monitoring agent. The agent sends this every 60 seconds (configurable). Metrics are stored and processed for alerts, digests, and queries.
Headers
| Header | Value |
|---|---|
| Authorization | Bearer <token> |
Request Body
| Field | Type | Required | Description |
|---|---|---|---|
| lab_id | string | required | Your lab identifier |
| timestamp | string | optional | ISO 8601 timestamp (defaults to server time) |
| collectors | object | required | Collector data keyed by type |
Collector Types
The collectors object can contain the following keys:
| Key | Description |
|---|---|
| system | CPU, memory, disk, network, load, uptime, temperatures |
| docker | Container list with health, restarts, CPU/memory per container |
| services | HTTP/TCP health check results with response times |
| gpu | GPU utilization, memory, temperature per device |
Example
curl -X POST https://your-server/api/v1/ingest \
-H 'Authorization: Bearer lw_abc123...' \
-H 'Content-Type: application/json' \
-d '{
"lab_id": "lab_proxmox-01_x7k2",
"collectors": {
"system": {
"data": {
"cpu": {"total_percent": 23.5, "count": 8},
"memory": {"used_percent": 62.1, "total_bytes": 34359738368},
"disk": [{"mount": "/", "used_percent": 45.2}],
"load_average": {"1m": 1.2, "5m": 0.8, "15m": 0.6},
"uptime_seconds": 864000
}
},
"docker": {
"data": {
"containers": [
{"name": "caddy", "state": "running", "health": "healthy",
"cpu_percent": 0.5, "memory_mb": 32}
]
}
}
}
}'Response 200
{
"status": "ok",
"alerts_triggered": 0
}Agent Status
Returns the current status of a monitored node, including latest metrics and active alerts.
Path Parameters
| Param | Type | Description |
|---|---|---|
| lab_id | string | The lab identifier |
Response 200
{
"lab_id": "lab_proxmox-01_x7k2",
"hostname": "proxmox-01",
"last_seen": "2026-03-19T22:30:00",
"cpu_percent": 23.5,
"memory_percent": 62.1,
"disk_percent": 45.2,
"container_count": 12,
"alerts": []
}List Labs
Returns a list of all registered labs with their current online status and latest metrics summary.
Response 200
{
"labs": [
{
"lab_id": "lab_proxmox-01_x7k2",
"hostname": "proxmox-01",
"last_seen": "2026-03-19T22:30:00",
"online": true,
"cpu_percent": 23.5,
"memory_percent": 62.1,
"disk_percent": 45.2
}
]
}Lab History
Returns time-series metrics for a specific lab, suitable for charting. Default window is 24 hours.
Query Parameters
| Param | Type | Default | Description |
|---|---|---|---|
| hours | int | 24 | Number of hours of history to return |
| secret | string | — | Admin secret (alternative to header) |
Response 200
{
"lab_id": "lab_proxmox-01_x7k2",
"hours": 24,
"metrics": [
{
"timestamp": "2026-03-19T21:00:00",
"cpu_percent": 23.5,
"memory_percent": 62.1,
"disk_percent": 45.2,
"load_1m": 1.2
}
]
}Export Data
Exports all stored metrics for a lab as a downloadable JSON file. Useful for backups, migrations, or external analysis.
Response 200
Returns a JSON file download with all metrics, alerts, and configuration for the specified lab.
Delete Lab
Permanently removes a lab and all its stored metrics, alerts, and digests. This cannot be undone.
Response 200
{
"message": "Lab deleted",
"lab_id": "lab_proxmox-01_x7k2"
}Natural Language Query
Ask questions about your infrastructure in natural language. The engine analyzes your metrics data and returns a plain-English answer with supporting data.
Request Body
| Field | Type | Required | Description |
|---|---|---|---|
| question | string | required | Your question in plain English |
Example Questions
"How's my lab?"
"Why is pve-storage slow?"
"Is caddy running?"
"Which server uses the most CPU?"
"Am I running out of disk?"
"What happened last night?"Example
curl -X POST https://your-server/api/v1/query \
-H 'X-Admin-Secret: your-secret' \
-H 'Content-Type: application/json' \
-d '{"question": "Why is pve-storage slow?"}'Response 200
{
"answer": "I/O pressure — load average 38.1 with only 2% CPU usage suggests a disk or network bottleneck, not a compute issue. Memory is fine at 62%. Check for heavy NFS traffic or failing drives.",
"confidence": 0.85,
"sources": ["system_metrics", "load_analysis"]
}Lab Digest
Generates a plain-English intelligence digest for a specific lab, analyzing trends, anomalies, and health over the specified period.
Query Parameters
| Param | Type | Default | Description |
|---|---|---|---|
| days | int | 7 | Number of days to analyze |
Response 200
{
"lab_id": "lab_proxmox-01_x7k2",
"period_days": 7,
"grade": "B+",
"summary": "proxmox-01 had moderate activity over the last 7 days...",
"metrics": {
"cpu_avg": 15.2,
"cpu_peak": 72.1,
"memory_avg": 62.0,
"disk_current": 78.3
},
"concerns": ["Disk at 78% — approaching warning threshold"],
"generated_at": "2026-03-19T22:30:00"
}Returns the most recently generated digest for a lab without regenerating it.
Fleet Digest
Generates a comprehensive digest across all monitored nodes. Includes per-node grades, fleet-wide trends, anomalies, and recommendations.
Response 200
{
"fleet_size": 5,
"period_days": 7,
"overall_grade": "B",
"summary": "Your fleet of 5 nodes is generally healthy...",
"nodes": [
{"lab_id": "...", "hostname": "proxmox-01", "grade": "A"},
{"lab_id": "...", "hostname": "pve-storage", "grade": "C"}
],
"recommendations": [
"pve-storage disk usage trending upward — consider cleanup"
]
}Uptime Checks
Monitor HTTP endpoints, TCP ports, and SSL certificates from the cloud. Checks run every 60 seconds by default.
Returns all uptime checks configured for your account.
Response 200
{
"checks": [
{
"id": "uc_...",
"name": "Main API",
"check_type": "http",
"target": "https://api.example.com/health",
"interval_seconds": 60,
"timeout_seconds": 10,
"expected_status": 200,
"enabled": true,
"last_status": "up",
"last_checked_at": "2026-04-05T12:00:00"
}
],
"total": 1
}Create Uptime Check
Create a new HTTP, TCP, or ping check. HTTPS targets automatically get SSL certificate monitoring. Subject to plan limits.
Request Body
| Field | Type | Required | Description |
|---|---|---|---|
| name | string | required | Display name for this check |
| check_type | string | optional | http (default), tcp, or ping |
| target | string | required | URL for HTTP, host:port for TCP |
| interval_seconds | int | optional | Check interval (default: 60, min: 30) |
| timeout_seconds | int | optional | Timeout per check (default: 10) |
| expected_status | int | optional | Expected HTTP status code (default: 200) |
Example
curl -X POST https://your-server/api/v1/account/uptime-checks \
-H 'Authorization: Bearer eyJhbGci...' \
-H 'Content-Type: application/json' \
-d '{"name": "API Health", "target": "https://api.example.com/health"}'Plan Limits
| Plan | Uptime Checks |
|---|---|
| Free | 3 |
| Homelab | 10 |
| Pro | 50 |
| Business | Unlimited |
Check Results
Returns the result history for an uptime check. Includes status, response time, HTTP status code, and SSL certificate info for HTTPS checks.
Query Parameters
| Param | Type | Default | Description |
|---|---|---|---|
| hours | int | 24 | Hours of history to return |
Response 200
{
"results": [
{
"status": "up",
"response_time_ms": 142,
"status_code": 200,
"ssl_cert": {
"valid": true,
"expires": "2026-12-31T23:59:59",
"days_remaining": 270,
"issuer": "R3",
"subject": "api.example.com"
},
"checked_at": "2026-04-05T12:00:00"
}
],
"total": 1440,
"hours": 24
}Delete Uptime Check
Permanently removes an uptime check and all its stored results.
Response 200
{
"status": "deleted"
}Maintenance Windows
Schedule maintenance windows to suppress alerts and skip uptime checks during planned downtime.
Returns all maintenance windows for your account, including past, active, and scheduled windows.
Response 200
{
"windows": [
{
"id": "mw_...",
"title": "Proxmox cluster upgrade",
"start_time": "2026-04-06T02:00:00",
"end_time": "2026-04-06T06:00:00",
"lab_ids": ["lab_proxmox-01_x7k2"],
"suppress_alerts": true,
"created_at": "2026-04-05T12:00:00"
}
],
"total": 1
}Schedule Maintenance
Schedule a maintenance window. During active windows, alerts are suppressed and uptime checks skip affected resources.
Request Body
| Field | Type | Required | Description |
|---|---|---|---|
| title | string | required | Description of the maintenance |
| start_time | string | required | ISO 8601 start time |
| end_time | string | required | ISO 8601 end time |
| lab_ids | string[] | optional | Labs to include (empty = all labs) |
| suppress_alerts | bool | optional | Suppress alerts during window (default: true) |
Response 200
{
"id": "mw_...",
"status": "created"
}Delete Maintenance Window
Cancel or remove a scheduled maintenance window.
Response 200
{
"status": "deleted"
}List Notification Channels
Returns all configured notification channels.
Response 200
{
"channels": [
{
"id": 1,
"type": "ntfy",
"name": "Phone alerts",
"config": {"topic": "labwatch-alerts"},
"enabled": true
}
]
}Create Notification Channel
Create a new notification channel. Supported types: webhook, discord, slack, telegram, ntfy, email.
Request Body
| Field | Type | Required | Description |
|---|---|---|---|
| channel_type | string | required | webhook, discord, slack, telegram, ntfy, or email |
| name | string | required | Display name for the channel |
| config | object | required | Channel-specific configuration (see below) |
| min_severity | string | optional | Minimum severity: info, warning (default), or critical |
Discord Config
{"webhook_url": "https://discord.com/api/webhooks/..."}Slack Config
{"webhook_url": "https://hooks.slack.com/services/..."}Telegram Config
{"bot_token": "123456:ABC-DEF...", "chat_id": "-1001234567890"}ntfy Config
{"topic": "labwatch-alerts", "server": "https://ntfy.sh"}Email Config
{"to_address": "alerts@example.com", "smtp_host": "smtp.gmail.com", "smtp_port": 587, "smtp_user": "...", "smtp_pass": "..."}Generic Webhook Config
{"url": "https://your-endpoint.com/hook"}Response 200
{
"id": 1,
"message": "Channel created"
}Update Notification Channel
Update an existing notification channel's name, config, or enabled status.
Delete Notification Channel
Permanently removes a notification channel.
Test Notification Channel
Sends a test message through the specified channel to verify it's configured correctly.
Response 200
{
"message": "Test notification sent",
"channel_id": 1
}Alert Rules
Custom alert rules let you override the default thresholds for specific metrics, either account-wide or per-lab.
Returns all custom alert rules for your account.
Response 200
{
"rules": [
{
"id": 1,
"metric": "cpu",
"warning_threshold": 80.0,
"critical_threshold": 95.0,
"lab_id": null,
"enabled": true
}
]
}Create Alert Rule
Create a custom alert rule. If a rule already exists for the same metric and lab, it will be updated instead. Requires a paid plan (Homelab or higher).
Request Body
| Field | Type | Required | Description |
|---|---|---|---|
| metric | string | required | One of: cpu, memory, disk, load, gpu_util, gpu_mem, gpu_temp |
| warning_threshold | number | optional | Warning threshold percentage |
| critical_threshold | number | optional | Critical threshold percentage |
| lab_id | string | optional | Apply to specific lab (omit for account-wide) |
Priority Layering
Thresholds are merged with this priority: lab-specific > account-wide > system defaults. You only need to set the thresholds you want to override.
Effective Thresholds
Returns the effective thresholds after merging system defaults, account-wide rules, and lab-specific rules.
Query Parameters
| Param | Type | Description |
|---|---|---|
| lab_id | string | Lab to check (optional, omit for account-wide) |
Response 200
{
"thresholds": {
"cpu": {"warning": 80.0, "critical": 95.0},
"memory": {"warning": 85.0, "critical": 95.0},
"disk": {"warning": 80.0, "critical": 90.0},
"load": {"warning": null, "critical": null},
"gpu_util": {"warning": 90.0, "critical": null},
"gpu_mem": {"warning": 90.0, "critical": null},
"gpu_temp": {"warning": 85.0, "critical": 95.0}
}
}Status Pages
Create public status pages to share your infrastructure uptime with users or teammates.
Returns all status pages for your account.
Create Status Page
Create a public status page with a custom slug. The page will be accessible at /status/your-slug.
Request Body
| Field | Type | Required | Description |
|---|---|---|---|
| slug | string | required | URL slug (3-50 chars, lowercase alphanumeric + hyphens) |
| title | string | optional | Page title (default: "System Status") |
| lab_ids | string[] | optional | Labs to include (default: all your labs) |
Plan Limits
| Plan | Status Pages |
|---|---|
| Free | 1 |
| Homelab | 3 |
| Pro | 10 |
| Business | Unlimited |
Incidents
Returns all incidents for your account. By default only shows active (non-resolved) incidents.
Query Parameters
| Param | Type | Default | Description |
|---|---|---|---|
| include_resolved | bool | false | Include resolved incidents |
Response 200
{
"incidents": [
{
"id": "abc-123",
"title": "Database connectivity issues",
"status": "investigating",
"severity": "major",
"status_page_id": "def-456",
"created_at": "2026-04-05T09:00:00",
"updated_at": "2026-04-05T09:15:00",
"resolved_at": null
}
],
"total": 1
}Create a new incident. Optionally attach it to a status page to make it visible publicly. An initial timeline update is created automatically.
Body Parameters
| Field | Type | Required | Description |
|---|---|---|---|
| title | string | Yes | Short description (max 200 chars) |
| severity | string | No | minor (default), major, or critical |
| status_page_id | string | No | Attach to a status page |
| message | string | No | Initial timeline message |
Response 200
{
"id": "abc-123",
"status": "created"
}Add a timeline update to an incident and change its status. Each update is appended to the incident's timeline, visible on the public status page.
Body Parameters
| Field | Type | Required | Description |
|---|---|---|---|
| status | string | Yes | investigating, identified, monitoring, or resolved |
| message | string | Yes | Description of the update |
Response 200
{
"status": "updated",
"incident_status": "identified"
}Status Flow
investigating → identified → monitoring → resolved
Account Digest
Generate digests using JWT auth (for account users instead of admin secret).
Generates a fleet-wide intelligence digest across all labs in your account. Requires Homelab plan or higher.
Query Parameters
| Param | Type | Default | Description |
|---|---|---|---|
| hours | int | 168 | Hours of data to analyze (default: 7 days) |
Response 200
{
"summary": "**3 nodes** monitored. 2 healthy, 0 fair, 1 need attention.\n...",
"nodes": [...],
"node_count": 3,
"concerns_count": 1
}Returns the most recently generated digest for a specific lab. Only returns data for labs you own.
Response 200
Returns the full digest object including grade, summary, metrics, and concerns.
Error 404
{
"detail": "No digest found. Generate one first."
}Chart Data
Returns downsampled CPU, memory, and disk time-series data for all labs in your account. Max ~200 points per lab.
Query Parameters
| Param | Type | Default | Description |
|---|---|---|---|
| hours | int | 24 | Time window (1-720 hours) |
| lab_id | string | — | Filter to a specific lab |
Response 200
{
"series": {
"proxmox-01": [
{"t": "2026-03-19T20:00:00", "cpu": 23.5, "mem": 62.1, "disk": 45.2},
{"t": "2026-03-19T20:05:00", "cpu": 18.2, "mem": 61.8, "disk": 45.2}
]
},
"hours": 24,
"lab_count": 3
}