At 12:00, a service operator, the provider running the job, has a healthy graphics processing unit (GPU), and a client receives a five-minute payment quote. At 12:04, the execution process crashes. At 12:05, the quote expires. At 12:15, the operator’s last on-chain heartbeat is still visible, even though no new job can finish.
Which signal should remove the operator from new work? Which signal tells the operator how to recover? Why can a service look online on-chain while its paid HTTP (Hypertext Transfer Protocol) endpoint is already failing?
Operator health monitoring on Tangle is not one green light. It combines an on-chain liveness record, local runtime checks, payment-quote state, and a recovery process for settled payments without terminal Job results.
Heartbeats, health checks, quotes, and traces
Use on-chain heartbeats to publish service status where the network can inspect it. Use local and remote health checks to catch process, deployment, queue, and dependency failures sooner. Use payment-quote lifetimes to prevent old prices from becoming new work. Use a trace, the record joining request and execution state, to reconcile payment, job, runtime, heartbeat, and result state.
The current public OperatorStatusRegistry, Tangle’s on-chain operator-status contract, uses a five-minute default heartbeat interval and a default maximum of three missed heartbeats, while service configuration can change those values.
It also rejects signed heartbeat submissions older than five minutes, which is a replay bound rather than the service’s liveness interval.
The current public remote-provider monitor checks every 60 seconds by default; it reports a stopped or terminated instance as unhealthy immediately and attempts automatic recovery after three consecutive unhealthy results when that option is enabled.
Heartbeats describe chain-visible status, remote checks describe deployment health, quotes describe payment freshness, and traces describe actual Job outcomes.
Signals used to admit and recover work
A Blueprint is a public Tangle service definition with named jobs and operator rules.
An operator runs a Blueprint service instance on machines it supplies or controls.
A runtime is the process and environment that executes a job. A job router maps a job type to its handler.
An x402 payment is an HTTP (Hypertext Transfer Protocol) flow in which a service returns payment requirements and the client retries with payment authorization. A quote lifetime is the period in which that payment requirement may be used.
An attestation is signed evidence about a protected runtime, such as a trusted execution environment (TEE) measurement. It can be another health or security signal, but a valid attestation does not prove that the process is available right now.
An agent profile is the client’s policy for endpoint, token, budget, timeout, retry, and result checks. A trace is the timeline joining one request, payment, job, runtime attempt, and result.
An evaluation is a check against an expected condition, such as “a sample known job returns the approved schema within 30 seconds.” It is stronger than a process ping and narrower than a claim that every future job will succeed.
Network, deployment, gateway, and Job health
The same operator can be healthy in one layer and broken in another.
| Layer | Signal | What it answers | Typical owner |
|---|---|---|---|
| Network | On-chain heartbeat | What status did the operator last publish? | Operator and network caller |
| Deployment | Remote health monitor | Is the machine or deployment responding? | Operator |
| Gateway | Quote registry and payment counters | Are paid requests being priced, verified, and consumed? | Gateway operator |
| Job | Runtime trace and result evaluation | Did this specific job run and pass its checks? | Service developer and operator |
An on-chain heartbeat is durable and externally inspectable. It is also slower and more expensive to update than a local process metric.
A local health check can react in seconds. It can also be wrong if it checks only the process and not the chain, model, queue, storage, or result path.
A quote registry can say that a quote is active. It cannot say that a GPU has capacity or that the answer will be correct.
A job evaluation can show that one known request passed. It cannot replace liveness monitoring for the next request.
On-chain heartbeat behavior
The public Tangle OperatorStatusRegistry records heartbeats for an operator and service.
A heartbeat includes the service and Blueprint identity, a status code, optional metrics, a timestamp, and a signature in the signed submission path.
The current public defaults are:
heartbeat interval: 5 minutes
maximum missed heartbeats: 3
signed heartbeat max age: 5 minutes (replay bound)Treat those as current contract defaults, not an immutable network promise. The contract allows service-specific heartbeat configuration within its allowed bounds.
The registry distinguishes several concepts that are easy to confuse:
isHeartbeatCurrentanswers whether the latest heartbeat is recent enough.isOnlinereports whether the recorded status is considered usable by the registry’s online rule.checkOperatorStatuslets an external caller evaluate missed time and update stored status.
The third point matters. A view function can report stale state without changing it. An external keeper, operator, or authorized caller may need to call the status-checking function for the registry to record the transition. Do not describe the contract as if a background process automatically wakes up and marks every operator offline.
Status codes need a policy
The registry stores a small status enum, while the heartbeat carries a numeric report code.
Code 0 stores Healthy; a nonzero code below 100 stores Degraded; and codes 100 or higher also enter the contract’s critical-status handling.
Codes 200 or higher can emit a SlashingTriggered event subject to the contract’s cooldown, but that event is not an executed slash.
The current contract comments reserve 255 for an operator exit request, while the explicit goOffline function uses the Exiting status, so 255 is not a universal offline or exit code.
The service should publish what its status codes mean. For example:
| Example code range | Meaning to document | Suggested operator action |
|---|---|---|
0 | Healthy | Continue normal service |
1–99 | Degraded but potentially usable | Investigate and reduce new load if needed |
100–199 | Critical condition requiring policy review | Stop risky work and preserve evidence |
200–254 | Protocol-violation alert candidate | Follow the service’s slashing process; do not treat the event as a slash |
255 | Exit-request code mentioned in current contract comments | Check the service policy; do not confuse it with goOffline |
The exact meaning is contract and service configuration, not a universal HTTP status code. Read the current registry before writing an alert that assumes “anything above zero means offline.”
Metrics are evidence, not decoration
The heartbeat path can carry custom metric key-value pairs. The current registry caps processing at 50 metric pairs and ignores payloads over 50,000 bytes; decoding and validation failures are caught so a malformed metric payload does not revert the heartbeat path. Those limits make a heartbeat suitable for small operational summaries, not a log transport.
Useful metrics are compact and actionable:
queue_depth=4
gpu_utilization=71
model_loaded=1
last_success_age_seconds=18
runtime_error_rate_bps=25Do not put user prompts, model outputs, secrets, or high-cardinality request IDs in an on-chain heartbeat. Publish a digest or aggregate and keep detailed evidence in an access-controlled monitoring system.
The metric value does not become true merely because it was signed. The operator is attesting to what it measured, while the service or network decides whether that metric matters for eligibility or slashing.
Remote deployment health
The public Blueprint remote-provider health monitor operates on a different clock. Its current default check interval is 60 seconds, and its default recovery threshold is three consecutive unhealthy results with automatic recovery enabled.
That monitor can report deployment-level states such as healthy, degraded, unhealthy, or unknown. It can catch a dead process or unreachable instance before an on-chain heartbeat has aged out.
The monitor should check more than a basic TCP (Transmission Control Protocol) connection. Use a layered probe:
- Process responds to a local liveness check.
- Runner can reach the configured chain or remote procedure call (RPC) endpoint.
- Model or worker dependencies are loaded.
- Queue accepts and persists a test job.
- A known safe job returns the expected result shape.
- Payment discovery and status reconciliation work without executing paid work.
The fifth check is an evaluation. It should use synthetic input and a bounded output, not a customer’s private data.
Quote lifetimes are a separate clock
An x402 quote says what payment terms the gateway will accept for a limited period. The public Tangle gateway exposes a configurable quote time-to-live (TTL), and the quote registry tracks creation, expiration, and consumption. The quote registry source also supports counting active entries and garbage collection.
The quote clock protects price and capacity. It does not tell you whether the operator is healthy. The current quote registry is process-local memory, so it is not a substitute for durable records of settled payments and admitted jobs.
A five-minute example works like this:
12:00:00 quote created, expires at 12:05:00
12:02:00 client signs and submits payment
12:02:01 gateway verifies and settles
12:02:02 job admitted
12:05:01 same quote must be rejected if unusedIf the runtime fails at 12:02:03, the quote’s expiration does not resolve the already-settled job. The payment reference and job record must survive independently.
If the client has not paid by 12:05:00, rejecting the quote is safer than honoring an old price. The client can request a fresh quote.
Do not describe a quote TTL as a protocol-wide maximum unless the current gateway and network documentation says so. The gateway’s configuration, token, network, price source, and service policy all matter.
A useful operator dashboard
Map each dashboard panel to an alert, owner, and recovery action.
| Panel | Alert condition | First question |
|---|---|---|
| Heartbeat freshness | Time since last heartbeat approaches interval | Can the signer reach the chain? |
| Missed-heartbeat count | Count rises or status degrades | Is the process, RPC, wallet, or gas path broken? |
| Deployment health | Three consecutive checks fail under current config | Is the remote instance reachable and ready? |
| Quote expiry | Expired quote rate rises | Is the client slow, gateway overloaded, or price stale? |
| Payment state | Unknown settlement count is nonzero | Which payments need reconciliation before work? |
| Queue | Depth or age exceeds policy | Is capacity exhausted or a worker stuck? |
| Job evaluation | Known-job failures rise | Did a release or model change break output? |
| Recovery | Jobs settled without final outcome | What does the client see and who owns repair? |
The following is illustrative alert logic. Metric names are application-defined and should match the instrumentation you publish.
alerts:
- name: HeartbeatNearExpiry
when: heartbeat_age_seconds > heartbeat_interval_seconds * 0.8
action: page_operator
- name: PaymentStateUnknown
when: payment_settlement_unknown_total > 0
action: stop_execution_and_reconcile
- name: PaidJobWithoutOutcome
when: settled_jobs_without_terminal_result > 0
action: page_operator_and_support
- name: QuoteExpirySpike
when: expired_quotes / issued_quotes > 0.05 for 10m
action: inspect_client_latency_and_gateway_capacityThe values are examples. Calibrate them against the service’s real job duration and network latency rather than copying a threshold because it looks tidy.
The incident timeline
Return to the opening incident.
At 12:00, the operator submits a healthy heartbeat and the runtime reports a loaded model. At 12:00:20, the gateway issues a quote with a five-minute lifetime. At 12:02, the client pays and the gateway admits one job. At 12:04, the GPU worker crashes. At 12:05, the quote expires, but that only affects unused quotes. At 12:10, the heartbeat is older than the configured interval. At 12:15, an external status check records enough misses to mark the operator offline under the default policy.
The recovery order should be:
- Stop issuing new paid jobs from the unhealthy instance.
- Reconcile settled payment references and identify jobs without a terminal result.
- Preserve the trace, artifact digest, logs, and last heartbeat evidence.
- Restore the worker or move the durable jobs to an eligible replacement according to service policy.
- Run a synthetic evaluation and verify the chain path.
- Submit a healthy heartbeat and confirm the recorded status.
- Re-enable paid ingress only after a known job succeeds.
Do not restart the process and call the incident closed before checking the paid-job gap. The user-visible harm is a settled payment without a result, rather than the pod outage by itself.
What monitoring does not prove
A current heartbeat does not prove job correctness. It is a recent status statement.
An online status does not prove capacity. An operator can be online while its GPU queue is full.
A healthy remote probe does not prove on-chain liveness. The process can be reachable while its signer, RPC, gas, or contract path is broken.
An active quote does not reserve a worker forever. The service still needs queue limits, admission policy, and expiration handling.
A slashing-related event is not an automatic slash. The configured authority, evidence, and dispute process determine what happens next.
A passing synthetic evaluation does not prove every customer job will pass. It is a controlled sample that catches some release and dependency failures.
Checks required before reopening paid traffic
Monitor heartbeats for externally visible liveness, remote checks for deployment health, quote state for payment freshness, and job traces for actual outcomes. Keep their clocks and meanings separate.
The operator is ready for paid traffic only when the service can answer all four questions:
- Is the operator’s recorded status current?
- Can the runtime accept and execute a job now?
- Will a new payment quote expire safely if the client is slow?
- Can the team recover a settled job that has not returned a result?
The service should retain status, execution readiness, quote expiry, and Job outcome records for each admission and recovery decision.
The paid Blueprint job guide explains the payment-to-job transition. The deployment architecture guide explains how the runner and remote deployment boundary affect recovery.
How often should a Tangle operator send heartbeats?
The current registry default is five minutes, with three missed heartbeats as the default maximum, but service configuration can change the cadence. Read the active service configuration and alert before the contract threshold, not after it.
Does isHeartbeatCurrent mark an operator offline?
No. It reports whether the latest heartbeat is recent. An external status-check call may be needed to update stored status after enough time has passed.
Is the x402 quote lifetime the same as heartbeat freshness?
No. A heartbeat describes operator status. A quote lifetime describes how long a payment requirement may be used.
What should happen to paid jobs when the operator goes offline?
Stop admitting new work, reconcile already-settled payments, and follow the service’s retry, replacement, refund, or dispute policy for jobs without a result.
Do heartbeat metrics automatically trigger slashing?
A metric outside an expected range alone does not establish a slashable violation. The service and contract configuration must define the evidence and enforcement path.
Can I use a health endpoint as proof of Tangle availability?
No. Combine the endpoint with a chain/RPC check, heartbeat observation, a bounded known-job evaluation, and payment-state reconciliation.
For the final pre-release payment and execution checks, continue with the x402 Blueprint production deployment checklist.
Public sources
The public OperatorStatusRegistry defines current heartbeat submission and status behavior.
The Blueprint remote-provider health monitor documents deployment-level checks.
The Tangle x402 gateway and quote registry document quote and payment integration points.
The Blueprint QoS guide explains custom metrics, status signals, and why metric observations do not automatically execute a slash.
The Tangle operator incentives guide explains why service participation and availability affect operator economics.