brain
Network · Status REAL

Up, on a spare lane.

What the network measures about itself: a timed read on the database, whether every hourly settlement ran and how late, how fast native nodes answered, how much browser work verified, and every incident with its cause. Nothing here comes from an outside monitor and nothing is estimated when its source is down; a section that cannot be read says so.
measured 2026-10-10 03:28:26Z
Database
answering
read in 21 ms
Pooler lanes open
2 / 3
alternate (cooling 126s) · primary · session
Hourly settlements on time
41 / 48
latest window of 48 · slowest settled 297 min after close
Latest epoch
2026-10-10T02:00Z
311 wallets · settled 0 min after close
Native node inferencelast 7 jobs · since 2026-10-07 22:35Z
ModelDoneFailedp50p95tok/s
qwen/qwen2.5-1.5b-instruct702,428 ms19.2 s73.6
Wall time is request to completion on the coordinator's clock. Tokens per second uses node-reported token counts over coordinator-timed compute.
Browser networklast 50 distributed jobs
Units verified
80.5%
2,263 verified · 549 failed
Job latency p50
239.6 s
p95 2956.2 s
Units reassigned
2,183
median 140.1 s from loss to new node
A unit on a node that stops answering is handed to another node; the time between the two assignments is what "reassigned" measures. Detail per job on /explorer.
Incidents5 recorded · UTC
  1. 2026-10-09
    Native nodes dropped offline

    Impact. From roughly 04:00 to 12:30 UTC the GPU nodes running the Brain Node agent could not stay registered. Register, heartbeat and work-poll calls timed out from the agent's side, nodes were marked offline, canary jobs failed on deadline, and operators who restarted saw no benchmark arrive. Browser nodes were unaffected. The explorer shows the window as failed cj- canaries with no node attached.

  2. 2026-10-09
    Database disk and write load

    Impact. No outage. /api/health/db flapped to database_unavailable for a few minutes around 10:00 UTC while the rest of the site answered; some instances tripped their breaker on slow connections.

  3. 2026-10-08
    Pooler pool poisoned again; deployments disabled

    Impact. Database-backed routes returned database_unavailable for most of the evening (UTC), then every route returned HTTP 402 for about an hour. Browser nodes retried and rejoined; the two native nodes went offline and re-registered after service returned. The 17:00 to 20:00 epochs settled late, none were skipped.

  4. 2026-10-07
    Settlement contention, then pooler failure

    A long day. Three distinct problems, in sequence.

  5. 2026-10-06
    Database exhausted

    Impact. The site and every database-backed route were down for several hours. Browser nodes kept heartbeating and were not dropped; no epochs were lost.

Written by hand after each incident, with cause, cost and what changed. Full text in the docs. Machine-readable version of this page: /api/status.