Collector secrets backend

By default, every value in a collector's collector_config.yaml — the Platform token, per-plugin database passwords, cloud-provider credentials — lives in plaintext on disk. That is acceptable for local development, but for production deployments the collector SDK can load sensitive values from an external secrets backend (also referred to as the alternative secrets backend) instead, leaving only the backend pointer in the YAML file.

Supported providers

Provider	secrets_backend.provider value
AWS Systems Manager Parameter Store	AWSSystemsManagerParameterStore

Only one provider is available today; additional providers can be plugged in via the BaseSecretsBackend abstract class in odd-collector-sdk.

How the backend is consumed

When a secrets_backend block is present in collector_config.yaml, the loader fetches two things from the backend:

1. Collector settings — a single parameter whose YAML value is merged into the top-level collector settings (for example token, platform_host_url, default_pulling_interval).

2. Plugin settings — one parameter per plugin under a shared prefix; each parameter's YAML value is an individual plugin block.

The local YAML file is still parsed, but values from the secrets backend win. The local plugins list can contribute plugins whose names do not collide with plugins loaded from the backend, and the local top-level settings can contribute keys that are not set in the backend — but they can never override a key that is set in the backend. Treat the local file as a fallback, not an override.

Configuration reference

All keys below live under secrets_backend: in collector_config.yaml.

Key	Type	Default	Description
provider	string	(required)	Backend provider code. Must be AWSSystemsManagerParameterStore.
region_name	string	(resolved from env / IMDS — see below)	AWS region the SSM parameters live in.
collector_settings_parameter_name	string	/odd/collector_config/collector_settings	Full SSM parameter name holding the top-level collector settings.
collector_plugins_prefix	string	/odd/collector_config/plugins	SSM parameter prefix under which one parameter per plugin is stored.

The region_name is resolved in this order: AWS_REGION environment variable → region_name in collector_config.yaml → the EC2 IMDS (for collectors running on EC2 or EKS). If none of these resolves to a region, the collector process still starts, but the first SSM call raises botocore.exceptions.NoRegionError and the collector exits before ingestion begins. Set the region explicitly for non-EC2 deployments.

The SDK's settings loader uses extra="allow", so any additional keys under secrets_backend: are forwarded to the provider constructor. Future providers may introduce their own keys without needing a schema change here.

Parameter naming convention

For the AWS SSM provider, each plugin in the collector is stored as a separate SSM parameter under collector_plugins_prefix. The parameter's name is not significant to the loader — the SDK fetches every parameter under the prefix with a recursive GetParametersByPath call and parses each value as a plugin YAML block. The convention shown in the SSM example below uses the plugin name as the final path segment purely for operator readability; what matters is that each plugin's YAML is stored as its own parameter.

The value of each parameter is a YAML document with the same schema as a single entry in the plugins: list in collector_config.yaml.

Worked example

Step 1 — Store the collector settings in SSM

Create one SecureString parameter named /odd/collector_config/collector_settings with this value:

default_pulling_interval: 60
platform_host_url: https://odd.internal.example.com
token: <platform_token>

Step 2 — Store each plugin in SSM

For every data source the collector should pull, create one parameter under /odd/collector_config/plugins/. For example, a PostgreSQL plugin at /odd/collector_config/plugins/postgresql_adapter:

type: postgresql
name: postgresql_adapter
description: ""
database: warehouse
host: db.internal.example.com
port: 5432
user: odd_ro
password: <db_password>

Step 3 — Point the collector at the backend

Keep only the backend pointer (and anything you intentionally want the local file to contribute) in collector_config.yaml:

secrets_backend:
  provider: AWSSystemsManagerParameterStore
  region_name: eu-central-1
  collector_settings_parameter_name: /odd/collector_config/collector_settings
  collector_plugins_prefix: /odd/collector_config/plugins

Step 4 — Run the collector

On startup the collector will:

1. Parse collector_config.yaml.

2. Connect to SSM using the configured region.

3. Fetch /odd/collector_config/collector_settings and merge its values into the top-level settings (SSM wins on conflicts).

4. Fetch every parameter under /odd/collector_config/plugins/ and merge its plugins into the plugin list (SSM wins on name conflicts).

5. Validate the resulting config and start pulling.

Required IAM permissions

The IAM identity the collector runs under — an EKS service account role, an EC2 instance role, or a user with static credentials — needs at least these permissions on the parameters you created:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameter",
        "ssm:GetParametersByPath"
      ],
      "Resource": [
        "arn:aws:ssm:{region}:{account_id}:parameter/odd/collector_config/collector_settings",
        "arn:aws:ssm:{region}:{account_id}:parameter/odd/collector_config/plugins/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt"
      ],
      "Resource": "arn:aws:kms:{region}:{account_id}:key/{kms_key_id}"
    }
  ]
}

The kms:Decrypt statement is only needed when the parameters were created as SecureString with a customer-managed KMS key (or with the AWS-managed alias/aws/ssm key). Plain String parameters do not require KMS permissions.

Known limitations

The AWS SSM backend is a thin wrapper around boto3. The following behaviors come from the SDK defaults and cannot currently be overridden from collector_config.yaml. Review each against your deployment before relying on the backend in production.

The backend loads at most 10 plugin parameters from SSM. The SDK calls ssm.get_parameters_by_path(..., Recursive=True) once and reads response["Parameters"] without paginating. The AWS default MaxResults for this call is 10. A collector whose collector_plugins_prefix contains 11 or more plugin parameters will silently start with only the first 10; everything past that point is dropped with no error and no log warning.

Mitigations until the SDK adds pagination:

• Keep the plugin count under 10 per collector. Split larger fleets into multiple collectors, each with its own prefix.

• After deploying, compare the plugin count in SSM against the plugin count logged at collector startup — a mismatch means you have hit the cap.

No custom SSM endpoint can be configured. The SDK constructs the SSM client with boto3.client("ssm", region_name=...) and does not expose endpoint_url. Deployments that need to reach SSM via a VPC interface endpoint, a private DNS override, LocalStack, or any non-default endpoint must rely on system-level DNS / network configuration — there is no secrets_backend.endpoint_url key to set.

No timeout or retry overrides. The backend does not pass a botocore.config.Config, so the SDK's default connect timeout, read timeout, and retry mode apply. On a partitioned or slow SSM endpoint the collector can block on startup for the full default timeout before failing. There is no secrets_backend.connect_timeout / read_timeout / retries key.

SSM parameter values are parsed as YAML without schema validation at fetch time. A malformed YAML document in any parameter under collector_plugins_prefix will raise during safe_load and abort collector startup with a parsing error, not a message pointing at the offending parameter. Treat SSM parameter edits the same way you would treat edits to collector_config.yaml — validate the YAML locally before putting the parameter.