Blogx402 Production Runwaypart 8 of 13

Operator Monitoring: What a Green Health Check Misses

Choose the runtime, heartbeat, quote, payment, and result signals an operator should expose before a service quietly stops accepting work.

This post is about the Tangle Network protocol, a separate project from Tangle's infrastructure for AI agents. Introduction to Tangle covers the agent infrastructure.

The failure is easy to miss. The service process is running, the port is open, and the health check is green, yet the work has stopped.

A green health check cannot see the payment, queue, and execution boundaries, because “alive” describes only one part of a paid service. The machine may be reachable while its operator identity is stale on-chain, its price endpoint returns old offers, or its runner accepts requests that never reach a job handler.

Tangle is a protocol for registering and coordinating services run by independent operators. Blueprint operator monitoring watches those boundaries separately. A Blueprint is a packaged service definition with typed jobs that declare their inputs and outputs. An operator is the person or service that runs an instance of that Blueprint and commits infrastructure to it. A runtime is the process and execution environment that receives a job and performs the work. A Blueprint Runner is the runtime component that receives and dispatches typed job calls. A service instance is one registered deployment of a Blueprint, identified by a service ID. A Blueprint ID identifies the service definition, and a job index identifies one callable operation inside it. A quote is a time-limited price offer for a job.

Tangle’s QoS integration guide covers heartbeats, Prometheus-compatible metrics, optional logs, and custom on-chain metrics as one operating surface. Prometheus is the metrics format and scrape endpoint; Grafana displays those metrics.

Operator health signals: heartbeat, quote freshness, and capacity utilization

A green process is not a selectable operator

Start with the path a paid request takes.

client
  -> price or payment challenge
  -> payment verification
  -> job enqueue
  -> Blueprint runner
  -> job handler
  -> result and receipt

The chain heartbeat observes one part of this path. The local metrics endpoint observes another. The quote and payment records add a commercial clock.

Signal What it answers What it cannot answer
Runtime health Is the process responding? Can the protocol see a live operator?
Heartbeat Has this operator recently reported liveness for this service? Did the last job produce a correct result?
Price endpoint Can a caller discover a current price? Has the caller paid or received work?
Queue depth Is work waiting inside the runner? Did the operator’s transaction settle?
Job result Did this execution finish? Was the price fresh when the request began?
Payment record Was the commercial condition satisfied? Is the returned output useful or correct?

An alert without service_id, job_index, operator, and payment-to-call identifiers cannot identify the failing path.

Heartbeats are a protocol signal, not a process probe

The Blueprint QoS service submits periodic liveness signals to Tangle’s OperatorStatusRegistry, the on-chain record of whether an operator is responding for a service. Each heartbeat is signed with an operator key and associated with a service and Blueprint.

The cadence is configuration, not a universal promise. The public SDK exposes fields such as interval_secs and max_missed_heartbeats, so the service owner and operator can reason about the actual tolerance instead of borrowing a number from another deployment.

The current documentation shows the shape of an explicit configuration:

let qos = QoSServiceBuilder::new()
    .with_heartbeat_config(HeartbeatConfig {
        service_id,
        blueprint_id,
        interval_secs: 60,
        jitter_percent: 10,
        max_missed_heartbeats: 3,
        status_registry_address,
    })
    .with_heartbeat_consumer(Arc::new(consumer))
    .with_http_rpc_endpoint(rpc_url)
    .with_keystore_uri(keystore_uri)
    .with_status_registry_address(status_registry_address)
    .build()
    .await?;

The values above are an illustrative configuration from the public guide, not a network-wide default. The heartbeat needs a signing key, an RPC path, a registry address, and a retry policy that fits the service’s tolerance for absence.

A heartbeat can fail while the job handler is healthy. Common causes include an unavailable RPC endpoint, an unreadable keystore, a wrong registry address, a rejected transaction, and a clock or nonce problem. An RPC (remote procedure call) endpoint is the node interface that submits or reads chain requests. The monitor should therefore track heartbeat submission attempts and confirmed on-chain state separately.

For an operator, the first useful query is the current state for the specific service and address. The registry reference includes isHeartbeatCurrent, getOperatorState, getMetricValue, and getSlashableOperators. Use the operator status and metrics documentation to match the call names to the contract version you operate.

Keep local metrics and on-chain metrics in different buckets

Tangle exposes two metric paths with different consequences. Local metrics go to Prometheus, dashboards, and incident response. On-chain metrics are numeric values included in heartbeats and stored in the status registry, so a service owner or monitor can inspect them later.

A local job_duration_seconds series can use high-cardinality labels, meaning labels with many distinct values, because it exists for operations. An on-chain response_time_ms value should be small, stable, and defined by a service owner, because every submitted value becomes part of a public protocol record.

The public QoS guide distinguishes add_custom_metric() for Prometheus and Grafana from add_on_chain_metric() for values used in service-level policy. Do not send every debug counter on-chain. Choose metrics a caller can interpret without access to your private dashboard.

A service owner defines a MetricDefinition with a name, inclusive minimum and maximum values, and whether the metric is required. An out-of-range value or a missing required value emits a MetricViolation event. The current documentation is explicit that a violation does not auto-slash an operator. An off-chain keeper can monitor the event and call reportForSlashing when policy warrants a slashing proposal. One transient spike therefore does not punish an operator, and the event is a durable signal to combine with job failures, customer reports, and heartbeat history.

Monitor the paid HTTP path as a separate service

The current Blueprint x402 gateway provides health, stats, price, payment/auth responses, and synthetic call identifiers for monitoring. x402 is an HTTP payment protocol in which a server returns 402 Payment Required and a client signs a payment payload. In Tangle’s gateway, the payment is verified and settled through the configured facilitator before the gateway enqueues work. A JobCall is the runner’s typed record that carries an accepted job from the gateway or producer to the Blueprint handler.

The Blueprint integration exposes:

GET /x402/health
200 OK

ok

GET /x402/jobs/42/0/price
200 OK

{
  "price_wei": "1000000000000000"
}

POST /x402/jobs/42/0
202 Accepted

{
  "service_id": 42,
  "call_id": 17,
  "status": "queued"
}

The response body above is a simplified shape for the monitoring conversation. Use the current Blueprint x402 gateway documentation for the exact fields the version you run exposes.

The most important line is 202 Accepted. In this integration, it means the payment was accepted and the job was enqueued. It does not mean the job has completed.

A monitor that counts every 202 as a successful job reports excellent reliability while work is stuck behind a full queue. Carry the synthetic service and call identifiers into the result path so you can measure accepted, started, completed, failed, and abandoned jobs as distinct states.

The price endpoint deserves its own alert. It answers whether a job is priced and enabled for x402, but it does not prove the runner is ready. The gateway’s default invocation mode is disabled, and jobs are enabled one by one with job_policies. The current docs say an unpriced job is not callable through the gateway.

Because that default is fail-closed, a new job can be fully implemented and still return an error until its price and invocation policy are added on purpose.

Alerts separate stale liveness, queue saturation, and quote expiry

Metric names differ by deployment. Define them around states rather than copying a vendor dashboard. These names are illustrative:

blueprint_heartbeat_submission_failures_total
blueprint_heartbeat_current{service_id,operator}
blueprint_job_queue_depth{service_id,job_index}
blueprint_job_execution_seconds
blueprint_job_failures_total{reason}
blueprint_quote_generation_failures_total
blueprint_quote_age_seconds
blueprint_paid_requests_total{state}
Alert First question Safe first action
Heartbeat submission failures Is the key, RPC, nonce, or registry address wrong? Preserve the last good state and repair submission before changing job code.
isHeartbeatCurrent becomes false Did the chain miss the configured tolerance or did the reader use the wrong service? Compare local submission logs with the on-chain service and operator identifiers.
Quote generation failures Is the pricing input unavailable or invalid? Stop advertising new paid work until a fresh, inspectable price is available.
Quote age rises while demand continues Are callers waiting longer than the configured quote lifetime? Reduce downstream latency or return a fresh quote before retrying.
Queue depth rises after 202 responses Is the runner saturated or is a consumer disconnected? Cap admission and preserve the payment-to-call correlation.
Metric violations appear Is the bound too tight, the metric missing, or the workload degraded? Investigate the evidence; do not treat the event as automatic slashing.
Payment accepted but completion rate falls Is the failure in execution, result delivery, or reconciliation? Separate refunds, retries, and operator recovery instead of retrying blindly.

The alert should include service_id, blueprint_id, job_index, operator address, and the last observed call identifier. Those fields turn “the service is unhealthy” into a query another person can run.

Failure paths from health check to job result

The process is healthy but the heartbeat is stale. The runtime can answer /x402/health while the RPC provider rejects transactions. The fix belongs in the signing and chain path, not in the job handler.

The heartbeat is current but jobs fail. Liveness proves that an operator is reporting, not that the Blueprint returns valid outputs. Inspect runner logs, input validation, dependency errors, and result callbacks.

The price is current but payment succeeds and work never appears. The gateway may have accepted the payment and queued a JobCall while the producer or runner was disconnected. Treat the payment as a reconciliation record and the job as a separate state machine.

The metric violates its bounds while customers remain satisfied. The metric definition may measure the wrong thing, or its threshold may not match the job’s normal distribution. Change the service policy only after reviewing the actual values and the false-positive cost.

A second process probe cannot distinguish stale heartbeats, queued calls, or failed results. The monitor has to keep the path from request to payment to queued call to result.

Check gateway, price, heartbeat, queue, and result state

When a paid Blueprint appears offline, check in this order:

  1. Read the gateway health and stats endpoints without sending a paid request.
  2. Read the price endpoint for the exact service and job pair that is failing.
  3. Compare the latest local heartbeat attempt with the on-chain liveness result.
  4. Check queue depth and the age of the oldest queued call.
  5. Trace one paid request from its quote or payment reference to its synthetic Tangle call identifier.
  6. Decide whether to disable admission, repair the runtime, or wait for an in-flight result.
  7. Reconcile any payment that succeeded before the job failed or the response was lost.

Step 5 needs a trace: a record that connects related events across those stages, using stable identifiers and timestamps. A request ID printed in one log line does not count.

Publish an operator contract that another person can inspect

Monitoring is useful when the operator can explain what each signal means to a caller. Write down the heartbeat interval and missed-heartbeat tolerance, the quote lifetime, the queue admission limit, the expected completion state, and the response to a stale or missing result.

Give every signal a stable identity. At minimum, include the service ID, Blueprint ID, job index, operator address, and environment. Avoid putting an unbounded request ID or customer identifier into every metric label, because high-cardinality labels can make the monitoring system itself expensive and slow. Keep those per-call identifiers in traces and structured logs instead.

The thresholds should come from the service’s observed behavior. For example, an alert at two missed heartbeats may be reasonable for a five-minute interval and dangerously slow for a job that expires in thirty seconds. A sample alert at 80 percent queue utilization may be useful for a short batch job and too late for a latency-sensitive call. Record the chosen threshold beside the reason and review it after a real incident.

Treat missing data as a state. Zero quote failures means no quote failure was recorded. No quote metric at all may mean the pricing worker stopped exporting, the label changed, or the scrape path broke. The alerting policy should distinguish those cases so that a silent exporter cannot look healthier than a busy but functioning service.

The Tangle operator services guide explains the service and operator relationship, and the operator health guide covers heartbeat and quote-lifetime decisions. Use those concepts to write the contract for the actual Blueprint rather than copying a generic uptime target.

Turn the signals into a customer-facing view

An operator dashboard is not a customer promise. Customers care whether a paid request was accepted, started, finished within the stated time, and what remedy exists if it did not. Expose those states in a small status view that uses the same identifiers as the internal trace.

For a simple service, a weekly reliability statement could look like this. P50 is the median, and p95 is the value that 95 percent of completed jobs do not exceed.

Measure Count or distribution Denominator
Paid requests accepted 1,000 Requests with valid payment
Jobs started 992 Accepted jobs
Jobs completed 978 Accepted jobs
Completion time p50 18s, p95 74s Completed jobs
Results requiring retry 21 Completed jobs
Payments needing reconciliation 4 Settled payments

Those numbers are a reporting shape, not a claim about any Tangle service. The denominator keeps “978 completed” from reading as 97.8 percent reliability when the service never counted accepted jobs. It also shows where the loss occurs: before the worker starts, during execution, or after payment when the result is delivered.

Keep the heartbeat visible for operators and the job outcome visible for callers. A current heartbeat can support an admission decision, but it is not evidence that a particular paid job will complete. The customer-facing status should link to the job, payment, and result records without exposing private inputs or operator secrets.

How often should a Blueprint operator send a heartbeat?

Use the interval and missed-heartbeat tolerance configured for the service. The public QoS API exposes both values, and different services can make different availability tradeoffs.

Do on-chain metric violations slash operators automatically?

No. The current QoS documentation says violations emit an event; an off-chain keeper can decide whether to call reportForSlashing.

Does a healthy x402 endpoint prove that a job is working?

No. The health endpoint checks the gateway surface. Use queue, execution, result, and payment-to-call metrics to verify the complete path.

What does 202 Accepted mean in the Blueprint x402 gateway?

It means the paid request was accepted and enqueued. It is not a completion receipt, so the caller needs a result contract or a way to observe the job later.

What should an operator monitor first?

Start with heartbeat submission and current on-chain state, quote generation and age, queue depth, job completion, and payment reconciliation. Those signals cover the main boundaries in a “the service is up but no work arrives” incident without pretending one number represents the whole system.

With those signals visible, an operator can decide whether to keep serving, stop accepting new paid work, or repair the chain and runtime path.

Read the Tangle QoS guide for the current configuration surface and the Blueprint x402 gateway guide for the payment-to-job boundary. For the economics that follow those operational signals, see Blueprint x402 operator economics.

Next in x402 Production Runway: Operator Outage Recovery: When to Reopen Paid Traffic