Skip to content

0011. Adopt VictoriaTraces for distributed tracing

Date: 2026-07-28 Status: Accepted

Context

Conflux already generates and propagates OpenTelemetry trace context end-to-end: incoming HTTP requests get a span via SetSpanContext middleware, outgoing integration calls (WooCommerce, PrestaShop) get a traceparent header injected via a global HTTP client middleware, queue jobs carry trace context across RabbitMQ via a _traceparent payload key, and CLI commands get their own root span. Log entries are enriched with trace_id/span_id via a Monolog processor.

However, only open-telemetry/api and open-telemetry/sdk are installed. No exporter package is required, and TelemetryServiceProvider binds the TracerProvider with no span processor configured, so the SDK defaults to a no-op processor: spans are created and ended in-process, but never exported anywhere. There is no distributed tracing UI. This gap is tracked in CFX-129 and was already anticipated in Architecture: adding an exporter is a configuration change, not a code change to the existing instrumentation.

Conflux has no pre-existing observability backend to build on — no Grafana, Loki, Prometheus, Sentry, or APM tool is in use anywhere in the stack. Logging is plain Monolog to file/stderr/Slack, with no central aggregation. This is a greenfield decision, and the local environment is plain Docker Compose (no Kubernetes).

Three backends were evaluated:

  • Jaeger (all-in-one image) — the simplest possible local setup: a single container, no config, built-in UI, in-memory storage, native OTLP ingestion. Not viable beyond local development, though, since the all-in-one image has no durable storage or clustering story for staging/prod.
  • Grafana Tempo — mature, CNCF-adjacent, first-party Grafana integration with TraceQL. Requires object storage (S3-compatible) even for a monolithic deployment, and has no UI of its own — trace viewing depends on running Grafana alongside it, which is more moving parts than the local environment needs today.
  • VictoriaTraces — a newer, resource-efficient tracing database from the VictoriaMetrics ecosystem. Ingests OTLP natively, needs no external storage dependency (single binary/container), and ships a built-in UI (VMUI, under /select/vmui) for browsing traces without requiring Grafana or a separate Jaeger frontend. It also exposes a Jaeger-compatible Query API, so a Jaeger frontend or Grafana (via a Jaeger datasource) can be layered on top later without switching backends. Its main drawback is relative immaturity compared to Tempo/Jaeger — smaller community, less battle-tested at scale, and no TraceQL-equivalent query language.

Decision

Conflux adopts VictoriaTraces as its OpenTelemetry trace backend, starting with local development. The same tool is the current intention for staging and production, but that rollout is not decided in detail here — it depends on the separate conflux-infrastructure repository and may warrant its own follow-up decision if constraints emerge there.

For local development, VictoriaTraces runs as an additional docker-compose.yml service, with Conflux's OTel SDK configured (via an added OTLP exporter package and a SpanProcessor wired into TelemetryServiceProvider) to export spans to it over OTLP. Export is opt-in: config/telemetry.php's otlp_endpoint is null by default (falling back to the pre-existing no-op behaviour described above), and must be set to VictoriaTraces' OTLP endpoint via OTEL_EXPORTER_OTLP_TRACES_ENDPOINT in .env for spans to actually be exported. The exporter talks to it via the internal service name (victoria-traces:10428), never leaving the Docker network. The container's port 10428 is published directly to the host (ports: ["10428:10428"]), so the VMUI is reached at http://localhost:10428/select/vmui — no docker-hoster alias and no nginx TLS terminator involved, unlike mq.${APP_HOST}/docs.${APP_HOST} (proxied through nginx) or sql.${APP_HOST} (a direct network alias). A plain published port is the simplest option here since VictoriaTraces carries no session cookies or app-domain concerns that would call for TLS or a conflux.local subdomain. Traces are viewed via the built-in VMUI during development; a Grafana or Jaeger frontend is not introduced at this stage, since none exists in the stack today.

Consequences

Implementing this (composer package, TelemetryServiceProvider wiring, docker-compose.yml service) is tracked in CFX-129, not this record.

Choosing a backend with no external storage dependency and OTLP-native ingestion keeps the local environment simple: no object storage, no Grafana, no additional config file, consistent with Conflux's existing plain-Docker-Compose local setup.

Relying on a comparatively young project is an accepted trade-off: if VictoriaTraces turns out to be insufficiently mature for staging/prod (missing clustering guarantees, retention/query limitations at scale, or unresolved bugs), that will need a dedicated follow-up decision rather than a silent reversal of this one. Because ingestion is standard OTLP and querying is Jaeger-API-compatible, switching to Tempo or Jaeger later would not require re-instrumenting the application — only reconfiguring the exporter endpoint and the viewing frontend.