Ctrlk
Edit

odd-tracing-gateway

Optional standalone push-adapter service that ingests OpenTelemetry traces from operator microservices and exposes the inferred services as ODD entities for the Platform to pull.

odd-tracing-gateway is an optional standalone Java service that bridges OpenTelemetry distributed tracing into the ODD catalog. Operators deploy the gateway alongside their existing OpenTelemetry collection pipeline; their microservices emit traces to it, and the gateway extracts service identities and dependencies from those traces and exposes them as ODD Data Entities.

In the two-axis adapter taxonomy, the gateway is a push adapter with the standalone gateway deployment shape — operator microservices push (over OpenTelemetry/OTLP), the gateway processes the spans and caches the inferred entities, and the ODD Platform (or a collector configured to pull from the gateway) reads those entities through the standard adapter-contract entities API. Operator-mental-model is "my services push to the gateway"; the Platform-side leg is a pull hidden behind the gateway's standalone deployment.

The gateway is optional. Deployments that already populate the catalog through pull collectors (databases, BI, MLOps) and in-process push adapters (dbt, Airflow, Spark, Great Expectations) do not need it. Reach for the gateway when the catalog should also include the microservices in your stack — their identities, the HTTP endpoints they expose, the databases / Kafka topics / AWS resources they call — automatically inferred from the same distributed traces your observability stack already collects.

What it does

Per the gateway's source code (Spring Boot WebFlux + gRPC; entry point at Application.java):

  1. Receives OpenTelemetry traces over OTLP/gRPC at port 9090. Sources can either point their OTel SDK directly at the gateway (OTEL_EXPORTER_OTLP_TRACES_ENDPOINT), or — more commonly — emit through an existing OpenTelemetry collector that fans traces to the gateway as one of its exporters.

  2. Resolves the service that emitted each trace through a chain of ServiceNameResolver implementations, sorted by priority:

    • Default (DefaultServiceNameResolver, priority app.defaultNamePriority, default 0) — uses the OpenTelemetry service.name resource attribute. Always active.

    • Docker (DockerServiceNameResolver, priority app.docker.namePriority, default 10) — gated by app.docker.enabled=true. Resolves the source's container.id resource attribute against a Docker daemon to find the container's image, and uses that image as the canonical service name. Useful when source-side OpenTelemetry instrumentation reports container IDs but not service names (e.g., generic JVM auto-instrumentation in Docker Compose).

    • Kubernetes (K8sServiceNameResolver, priority app.k8s.namePriority, default 10) — gated by app.k8s.enabled=true. Resolves host.name (the pod) plus container.id against the Kubernetes API and reads the container's image from the pod's spec.containers. Useful when you want the canonical image to be the service identity, not the OpenTelemetry-supplied service name.

  3. Processes spans through type-specific SpanProcessor implementations, each accepting a specific OpenTelemetry instrumentation library:

    • HttpSpanProcessor — accepts io.opentelemetry.netty, io.opentelemetry.jetty, io.opentelemetry.spring-webflux libraries. Server spans become API_SERVICE (the host) + API_CALL (per host+method+path) entities; client spans add inputs to the calling service. IPs and the configurable app.http.staticPrefix (default /static/) are excluded.

    • JdbcSpanProcessor — emits dataset entities for tables observed in JDBC client spans (uses jsqlparser to extract table names from SQL).

    • KafkaSpanProcessor — emits Kafka topic entities for producer / consumer spans, scoped by the configured app.kafkaServers.

    • GrpcSpanProcessor — emits service / method entities for inter-service gRPC.

    • AwsSpanProcessor — emits AWS-resource entities for AWS SDK client spans (DynamoDB tables observed via OTel auto-instrumentation, etc.).

  4. Caches every observed (service, version) and its inputs / outputs in Redis, keyed by ODDRN. The cache is what the Platform-side leg of the integration reads.

  5. Exposes the inferred entities through GET /entities on port 8080, with an optional changedSince ISO-8601 query parameter. The endpoint implements org.opendatadiscovery.adapter.contract.api.EntitiesApi — the same contract every pull adapter / collector exposes — so an ODD pull collector can subscribe to the gateway as if it were any other source.

What it does NOT do

  • Does not store, ship, or replay OpenTelemetry traces. The gateway extracts service-identity and dependency metadata from each trace and discards the trace itself. Use a separate observability backend (Tempo, Jaeger, an OTel-compatible APM) if you need the full trace stream.

  • Does not push to the Platform's Ingestion API. Despite the historical "transfers metadata to the Platform" framing, the gateway does not call /ingestion/entities. Entities flow Platform-ward only when an ODD pull collector is configured to read the gateway's GET /entities endpoint.

  • Does not authenticate inbound OTLP traffic. The gRPC trace receiver is open to whoever can reach :9090. Run the gateway inside a network perimeter that already restricts who can push (a private subnet, a service mesh, a network policy).

  • Does not authenticate GET /entities. The REST endpoint is unauthenticated. Treat the gateway as an internal service that only the ODD Platform / a trusted collector talks to.

  • Does not perform retention or cleanup of cached entities. Redis holds the inferred catalog as long as the gateway is running against it; restarting the gateway with a fresh Redis empties the cache. There is no built-in TTL or operator-managed retention window today.

  • Does not deduplicate across replicas. A single Redis backs a single gateway instance; running multiple gateway replicas without a shared Redis produces divergent catalogs.

How the data flows

A typical operator setup wires four components together:

The OpenTelemetry collector is optional but recommended — it lets you fan trace traffic to the gateway and to your observability backend in parallel without instrumenting every microservice with two exporters.

Operator setup

Prerequisites

  • A running ODD Platform (any deployment shape from Deployment Options).

  • A reachable Redis instance for the gateway to use as its cache. The default Helm chart bundles a Redis sub-chart for evaluation; production deployments should point at a managed Redis (spring.redis.host, spring.redis.port, spring.redis.database).

  • An OpenTelemetry trace pipeline already collecting traces from your microservices, or microservices instrumented with the OpenTelemetry SDK and configured to export OTLP/gRPC.

Install via Helm (recommended)

The gateway ships as a Helm chart in the same chart repository as the Platform — charts/odd-tracing-gateway. Add the chart repo and install:

The chart exposes a config field for the gateway's runtime environment variables — set the resolver flags (APP_DOCKER_ENABLED, APP_K8S_ENABLED, APP_K8S_NAMESPACES), the Redis connection (SPRING_REDIS_HOST, SPRING_REDIS_PORT, SPRING_REDIS_DATABASE), and any HTTP-processor tuning (APP_HTTP_STATIC_PREFIX, APP_HTTP_EXCLUDE_IPS) through this map. See the chart's values.yaml for the full surface.

Install via Docker Compose (local evaluation)

For a local or demo deployment, run the gateway directly from its published Docker image. The image is published to two registries:

  • Docker Hub: opendatadiscovery/odd-traces-gateway:latest (note the plural "traces" — this is the Jib build target). New images are pushed on every main build.

  • GitHub Container Registry: ghcr.io/opendatadiscovery/odd-tracing-gateway:<version> (singular "tracing"). Released versions are retagged here on release events; no latest tag.

A minimal Docker Compose setup wiring the gateway, Redis, and an OpenTelemetry collector:

A reproducible end-to-end example — the gateway plus a sample Spring Boot app emitting traces through an OpenTelemetry collector — lives in the gateway repo's test/test-app/ directory.

Wire the Platform to pull from the gateway

The gateway exposes the standard adapter-contract entities endpoint (GET /entities). Configure an ODD pull collector to point at the gateway as one of its plugin sources, or have the Platform fetch entities directly. The exact wiring depends on which collector you operate; see your collector's configuration reference under Integrations.

Configuration reference

The gateway reads its configuration from environment variables (or a Spring Boot application.yml mounted into the container). Sources cited inline are paths inside the gateway's source tree.

Application

Key
Env var
Default
What it does
Source

app.oddrn

APP_ODDRN

(none)

The gateway's own ODDRN, set as dataSourceOddrn on every DataEntityList it returns. Used by the Platform to attribute entities back to this gateway instance.

config/AppProperties.java

app.defaultNamePriority

APP_DEFAULT_NAME_PRIORITY

0

Priority of DefaultServiceNameResolver in the resolver chain. Lower-priority resolvers run last; the first resolver returning a name wins.

config/AppProperties.java

app.kafkaServers

APP_KAFKA_SERVERS

unknown

Bootstrap-servers identifier used in Kafka-topic ODDRN composition. Set to your Kafka cluster's bootstrap address (host:port[,host:port...]) for the gateway to produce topic ODDRNs that match what your collectors and pull adapters generate.

config/AppProperties.java

app.exposeLatestVersion

APP_EXPOSE_LATEST_VERSION

true

When true (default), GET /entities returns only the latest version of each service ODDRN. When false, every observed version is exposed.

config/AppProperties.java

Docker resolver

Key
Env var
Default
What it does

app.docker.enabled

APP_DOCKER_ENABLED

false

Enables DockerServiceNameResolver. When false, the gateway never queries Docker.

app.docker.host

APP_DOCKER_HOST

(none)

Docker daemon endpoint (unix:///var/run/docker.sock for Docker Compose deployments).

app.docker.tlsVerify

APP_DOCKER_TLS_VERIFY

false

Enables TLS verification for the Docker client.

app.docker.namePriority

APP_DOCKER_NAME_PRIORITY

10

Priority of DockerServiceNameResolver in the resolver chain. With the default value, the Docker resolver runs after the default resolver but its result wins (the resolver chain stops at the first match in priority order).

Kubernetes resolver

Key
Env var
Default
What it does

app.k8s.enabled

APP_K8S_ENABLED

false

Enables K8sServiceNameResolver. When false, the gateway never queries the Kubernetes API.

app.k8s.host

APP_K8S_HOST

(none — uses in-cluster config)

Kubernetes API endpoint. Leave unset when the gateway runs as a pod inside the cluster (the fabric8 client picks up in-cluster credentials).

app.k8s.namespaces

APP_K8S_NAMESPACES

(all namespaces)

Comma-separated list of namespaces the gateway is allowed to inspect. When unset, the gateway lists every namespace at startup — make sure the service account has cluster-wide read on pods and namespaces if you leave this empty.

app.k8s.namePriority

APP_K8S_NAME_PRIORITY

10

Priority of K8sServiceNameResolver in the resolver chain.

HTTP-span processor

Key
Env var
Default
What it does

app.http.staticPrefix

APP_HTTP_STATIC_PREFIX

/static/

HTTP paths starting with this prefix are skipped — useful for excluding asset bundles, health checks, or admin endpoints from the catalog. Set to an empty string to ingest every observed path.

app.http.excludeIps

APP_HTTP_EXCLUDE_IPS

true

When true, server hosts and client URLs whose host portion is a literal IP address are skipped (the inferred service identity tends to be unstable when the host is an ephemeral pod IP). Set to false to include them.

Redis cache

Key
Env var
Default
What it does

spring.redis.host

SPRING_REDIS_HOST

localhost (Spring Boot default)

Redis hostname.

spring.redis.port

SPRING_REDIS_PORT

6379 (Spring Boot default)

Redis port.

spring.redis.database

SPRING_REDIS_DATABASE

0 (Spring Boot default)

Redis database index.

The full Spring Boot Redis configuration surface (auth, sentinel, cluster mode, SSL) is available — see the Spring Boot reference for the complete property list. The gateway uses spring-boot-starter-data-redis-reactive.

API surface

The gateway exposes two surfaces. Neither is part of the ODD Platform's HTTP API — both surfaces live on the gateway process itself.

Inbound — OpenTelemetry traces (sources push)

Protocol
Port (default)
Endpoint
Spec

OTLP/gRPC

9090

opentelemetry.proto.collector.trace.v1.TraceService/Export

OpenTelemetry Protocol

Source applications (or an OpenTelemetry collector in front of them) push traces over OTLP/gRPC. The gateway uses grpc-server-spring-boot-starter to host the receiver; instrumentation libraries the gateway extracts metadata from are listed under "How the data flows" above.

Outbound — Adapter-contract entities (Platform pulls)

Protocol
Port (default)
Endpoint
Spec

HTTP

8080

GET /entities[?changedSince=<ISO-8601>]

org.opendatadiscovery.adapter.contract — the contract every adapter exposes

The endpoint returns a DataEntityList JSON shape — dataSourceOddrn (set from app.oddrn) plus an items array of inferred services / API calls / API services / data inputs. The changedSince query parameter scopes the response to entities whose cached updatedAt is strictly greater than the supplied timestamp.

Where it sits architecturally

In the two-axis adapter taxonomy, the gateway is a Push adapter (standalone gateway) — distinct from in-process plugins (which live inside a source tool's runtime, like dbt or Airflow) and from collectors (which host pull adapters in a single Python container). The standalone-gateway shape is the right answer when:

  • The "source" is a population of microservices, not a single application — instrumenting each microservice with an in-process plugin is impractical.

  • The metadata you want catalogued comes from observability infrastructure (distributed traces) rather than from the source tool's own configuration / runtime.

  • The platform-side integration can be a single read-only fetch rather than continuous ingestion.

For the architectural overview that places the gateway among the Platform, collectors, and in-process push adapters, see Architecture → Deployment topology.

Known limitations

The OTLP receiver is unauthenticated. Anyone who can reach the gateway's port 9090 can push arbitrary traces — and therefore arbitrary "service" entities — into the cache. Run the gateway only inside a network perimeter that restricts inbound traffic (private subnet, service mesh policy, Kubernetes NetworkPolicy). The same applies to GET /entities on port 8080: the endpoint is unauthenticated; treat it as an internal service exposed only to the ODD Platform / a trusted pull collector.

Cached entities have no built-in retention. Redis holds every observed (service, version) indefinitely. A noisy or compromised source can grow the cache without bound. If you need eviction, configure it at the Redis layer (LRU max-memory policy, scheduled FLUSHDB); the gateway has no operator-tunable TTL today.

Multiple gateway replicas need a shared Redis. The gateway stores state in Redis, but each gateway replica writes independently. Running two replicas pointing at separate Redis instances produces two divergent catalogs; pointing both at the same Redis works, but at-most-once semantics on writes are not guaranteed. The simplest production posture is a single replica behind a Kubernetes Deployment with replicas: 1 and a managed Redis.

  • The default latest tag on opendatadiscovery/odd-traces-gateway floats. Pin the image tag to a specific gateway version (or pin the GHCR ghcr.io/opendatadiscovery/odd-tracing-gateway:<version> released image) for reproducible deployments.

  • The Docker resolver requires Docker socket access. Mounting /var/run/docker.sock into the gateway container grants the gateway read access to every container on the host. Use the Kubernetes resolver in production K8s deployments; the Docker resolver is for Docker Compose / single-host setups.

  • The Kubernetes resolver needs cluster-wide pod read when app.k8s.namespaces is unset. Restrict to a known namespace list to scope the service-account permissions tightly.

  • app.kafkaServers defaults to the literal string unknown. Kafka topic ODDRNs the gateway emits will carry that placeholder unless you set the key — they will not match the topic ODDRNs your Kafka pull adapter generates, and lineage edges between the two surfaces will not converge. Set app.kafkaServers to your Kafka cluster's bootstrap address before deploying.

  • Two image-name spellings are in circulation — the Docker Hub image is opendatadiscovery/odd-traces-gateway (plural "traces"; this is the Jib build target) and the GHCR retag is ghcr.io/opendatadiscovery/odd-tracing-gateway (singular "tracing"). Both publish from the same source repo. Prefer the GHCR pinned-version tag for production; the Docker Hub latest is for evaluation.

Where to next

Previousodd-cliNextIngestion filters

Last updated