Build and run ODD Collectors
Developer guide on how to build and run ODD Collectors
For instructions on how to run the ODD Platform and ODD Collectors locally in a Docker environment, please follow Try locally article.
ODD Collectors tech stack
There are 4 main collectors at the moment, all bundled in the odd-collectors monorepo:
ODD Collector — covering databases, BI tools, data warehouses, etc. (per-adapter reference: odd-collector)
ODD Collector AWS — covering AWS services (per-adapter reference: odd-collector-aws)
ODD Collector GCP — covering GCP services (per-adapter reference: odd-collector-gcp)
ODD Collector Azure — covering Azure services (Azure SQL, Data Factory, Blob Storage, Power BI; per-adapter reference: odd-collector-azure)
A specialist profiler collector — ODD Collector Profiler — lives in its own repo and produces statistical profiles for Postgres / Azure SQL sources; its per-adapter reference is at odd-collector-profiler. For the broader pull / push picture (how the four monorepo collectors compare to push adapters like dbt, Spark, Airflow, Great Expectations), see the Integrations hub.
ODD Collector AWS uses boto3, ODD Collector GCP uses Google's Cloud SDKs, and ODD Collector Azure uses the Azure SDK family; ODD Collector itself relies on per-data-source libraries.
The previously standalone
odd-collector,odd-collector-aws,odd-collector-gcp, andodd-collector-azurerepositories were archived on 2023-11-06 and consolidated into theodd-collectorsmonorepo above. New work happens only in the monorepo.
General tech stack is:
Python
Poetry
asyncio
Prerequisites
Python 3.9 or higher (the monorepo's
pyproject.tomlfiles all pinpython = "^3.9", so any 3.9.x or later 3.x interpreter works)Poetry 1.2.0
preferably the latest docker-compose
Build ODD Collector into Docker container
Fork and clone the odd-collectors monorepo if you haven't done it already.
Go into the directory of the sub-collector you want to build (one of odd-collector, odd-collector-aws, odd-collector-gcp, odd-collector-azure):
Run the following command, replacing <tag> with any tag name you'd like
Run ODD Collector locally
Run ODD Platform locally as a target for ODD Collector
In order to run ODD Platform locally please follow this guide.
Activate environment
Go into the sub-collector's directory inside the monorepo (substituting the sub-collector you want to run):
Run following commands to create local python environment and install dependencies
Change your python context to created one.
Configure ODD Collector to send request to target catalog
Create collector in the ODD Platform and copy created token using this guide.
Regenerating a collector's token has no grace window — it cuts off the running collector immediately. The Regenerate action on the same Management → Collectors surface is an in-place overwrite of the single stored token: there is no overlap period where the old and new tokens both work. The instant you regenerate, a collector still configured with the old token starts getting 401s on its next ingestion call, and ingestion stops until you update token: in collector-config.yaml and restart the collector. Treat regeneration as a coordinated, destructive operation — not a routine refresh. The token is also stored in plaintext and is not the audited operation you might expect; the full platform-side contract and mitigations are documented at Management → Collectors known caveats.
Configure collector-config.yaml for the adapters you intend to run. The canonical per-adapter reference — field-by-field plugin shape, defaults, supported features — lives on the per-collector pages of the Integrations hub: odd-collector, odd-collector-aws, odd-collector-azure, odd-collector-gcp, odd-collector-profiler. The raw upstream YAML examples are also browsable at each collector's config_examples/ directory — odd-collector, odd-collector-aws, odd-collector-gcp, odd-collector-azure. Replace <COLLECTOR_TOKEN> with the token obtained in the previous step.
Full configuration reference
The example above is the minimum that gets a collector running. The collector SDK's CollectorConfig accepts the following top-level fields:
platform_host_url
string
required
URL of the ODD Platform that the collector pushes metadata to.
plugins
list
required
Adapter configurations — each entry is one configured connection.
default_pulling_interval
integer (minutes)
unset
Polling cadence applied to every plugin. When unset, each plugin runs once and the collector exits — useful for one-shot ingestion in CI / cron.
connection_timeout_seconds
integer
300
HTTP timeout for requests from the collector to the ODD Platform's Ingestion API. Does not affect adapter-to-source connections — those are governed by each adapter's own client. Raise it when the platform is slow to acknowledge large pushes.
chunk_size
integer
250
Maximum number of data entities batched into a single Ingestion API request. Lower values reduce memory pressure when ingesting very large catalogs; higher values reduce request count.
misfire_grace_time
integer (seconds)
unset (falls back to default_pulling_interval × 60)
APScheduler grace window — if a scheduled run is delayed by more than this many seconds (e.g. previous run still executing, host pause), the missed run is dropped instead of firing late. When this field is unset, the SDK substitutes default_pulling_interval × 60 seconds — i.e. one full polling interval expressed in seconds — so missed runs are tolerated for up to one interval before being dropped. Set explicitly only when you want a tighter or looser tolerance than one polling interval.
max_instances
integer
1
Maximum concurrent runs of the same plugin. The default prevents a slow source from queuing overlapping pulls; raise it only when a plugin is explicitly safe to run in parallel.
verify_ssl
boolean
true
Whether the collector verifies the ODD Platform's TLS certificate on every Ingestion API call. Set to false only when the platform is served behind a self-signed certificate (development clusters, air-gapped deployments) — disabling certificate verification in production is a security risk.
Run ODD Collector
Run ODD Collector locally using following command:
How to implement new integration
For authoring a new pull adapter — defining a Plugin subclass, implementing the AbstractAdapter / BaseAdapter / AsyncAbstractAdapter contract, wiring PLUGIN_FACTORY, generating ODDRNs, packaging, and contributing back — see the dedicated Build a custom collector developer guide.
For push-strategy integrations (Airflow, dbt, Spark, Great Expectations, custom CI/CD), see the per-tool pages under the Integrations hub — those follow each host system's plugin or listener API rather than the pull-collector SDK.
Troubleshooting
Running ODD Collector on M1
libraries pyodbc , confluent-kafka and grpcio have problem during installing and building project on M1 Macbooks.
Possible solution:
The easiest way is to add all export statements to your .bashrc/.zshrc file
Last updated