GitHub Sign in Start free trial
Ingestion Monitoring

Ingestion Monitoring

Watch ingestion queues and database writes to distinguish accepted SDK traffic from data that Tindra has confirmed as stored.

Delivery and storage

Tindra queues errors, transactions, logs, and profiles in bounded memory buffers, then writes batches to Postgres. An HTTP success response confirms handling of the request, not that every item has been persisted. Use Project Setup to confirm a tagged test event and inspect stored-data milestones.

Errors and transactions refused by a full or closing buffer return HTTP 429. Logs and profiles are best effort: capacity pressure can drop those items while the envelope still succeeds. An envelope can also be partially accepted before a later error or transaction is refused. Retrying the whole envelope may therefore resend already accepted items.

Malformed envelopes, including null headers or truncated items, are rejected before item processing. SDK authentication failures, quota/rate limits, payload-size limits, and invalid individual payloads have separate outcomes. Check the SDK response and setup diagnostics alongside queue metrics; the queue counters do not count every request rejected before it reaches a buffer.

Retryable database failures receive bounded retries with backoff, up to five write attempts within a batch deadline. Permanent record failures are isolated where possible so other records can proceed. Shutdown stops admission and gives writers a finite drain period. These buffers are not disk-backed: a crash, forced stop, exhausted retries, or an expired drain deadline can lose pending data. See Upgrades for graceful shutdown guidance.

Enable Prometheus metrics

Set STATS_API_KEY to an operator secret on the Tindra server and restart it. This key is separate from project API tokens and the SDK's DSN. With the same key available in your shell environment, request:

curl --fail-with-body \
  'https://your-hostname.tindra.sh/metrics' \
  --header "Authorization: Bearer $STATS_API_KEY"

An unset server key disables this endpoint with HTTP 404; an incorrect credential returns HTTP 401. Authentication does not use a database connection or a user session, so database trouble does not itself hide ingestion metrics.

For Prometheus, mount a file containing only the operator key into the Prometheus process and add this scrape job, replacing the host and secret-file path:

scrape_configs:
  - job_name: tindra
    scheme: https
    metrics_path: /metrics
    scrape_interval: 15s
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/secrets/tindra-stats-key
    static_configs:
      - targets: ['your-hostname.tindra.sh']

The file is read by Prometheus, not Tindra. See the Prometheus HTTP configuration for credential-file configuration.

Read the metrics

All names below begin with tindra_ingest_ and have a type label: events, transactions, logs, or profiles. Counters and runtime state are process-local and reset on restart, so use counter rates or increases when comparing periods.

Metric suffix Meaning
accepted_total Items admitted to a buffer. This is not a storage count.
persisted_total Items confirmed by a successful write, including duplicates already present in the database. This is not a count of new rows.
rejected_total Error or transaction items refused at queue admission.
dropped_total Items abandoned or best-effort admission drops, split by the reason label.
unknown_commit_items_total Items whose commit acknowledgement could not be confirmed. They may already be stored.
failed_writes_total Failed batch write attempts, rather than individual item failures.
retries_total Batch retry attempts.
queued / capacity Items waiting in the queue and its item capacity.
pending All unfinished items, including batches currently writing or retrying.
pending_bytes / capacity_bytes Tracked pending bytes and the configured internal byte ceiling. A zero ceiling means no separate byte budget for that queue.
oldest_pending_seconds Age of the oldest unfinished item.
last_success_timestamp_seconds Unix timestamp of the last confirmed write, or zero before the first success.
degraded 1 when the writer reports failure or pressure, including old pending work; otherwise 0.

Dropped reasons include buffer_full, buffer_bytes, shutdown, write_failed, deadline, invalid_record, and encode_failed. Some apply only to particular queues or failure paths. Admission drops and post-admission failures should be interpreted separately from rejected errors/transactions.

An unknown commit is not confirmed loss. Tindra avoids blindly retrying an ambiguously committed batch because that could duplicate records. Inspect stored data and database logs before deciding whether to resend it.

JSON status and the Overview page

GET /api/instance/ingestion exposes the same queue state as JSON to a signed-in user with manage_projects. Neither a project token nor the operator metrics key substitutes for that session permission.

The Health Overview page shows storage and usage figures. It does not display an ingestion queue dashboard. Queue sizes, retry limits, and byte budgets are internal implementation settings; there are no additional queue-tuning environment variables to configure.

Investigate a backlog

  1. Compare pending count, oldest pending age, failed-write/retry rates, and last successful writes for the affected data type. An old success timestamp is normal when no traffic is arriving, so interpret it alongside pending work.
  2. Inspect Tindra writer logs and Postgres connectivity, availability, disk space, and load. A healthy HTTP endpoint does not prove writes are succeeding.
  3. Check SDK responses, instance usage, request limits, and whether logs or profiles were dropped independently of the error event. Consult Configuration before changing instance limits.
  4. After resolving the cause, confirm that pending age and counts fall, successful writes resume, and new rejections or drops stop increasing. Historical counters will not return to zero until restart.
  5. Send a fresh setup check and verify stored data. Recovery of the queue does not restore items already dropped or lost during a crash.

Pending in-memory data is not included in database backups. See Backup for the persisted data and source-map files to retain.