> For the complete documentation index, see [llms.txt](https://docs.opendatadiscovery.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.opendatadiscovery.org/configuration-and-deployment/collectors-secrets-backend.md).

# Collector secrets backend

By default, every value in a collector's `collector_config.yaml` — the Platform token, per-plugin database passwords, cloud-provider credentials — lives in plaintext on disk. That is acceptable for local development, but for production deployments the collector SDK can load sensitive values from an external **secrets backend** (also referred to as the **alternative secrets backend**) instead, leaving only the backend pointer in the YAML file.

## Supported providers

| Provider                            | `secrets_backend.provider` value  |
| ----------------------------------- | --------------------------------- |
| AWS Systems Manager Parameter Store | `AWSSystemsManagerParameterStore` |

Only one provider is available today; additional providers can be plugged in via the `BaseSecretsBackend` abstract class in `odd-collector-sdk`.

## How the backend is consumed

When a `secrets_backend` block is present in `collector_config.yaml`, the loader fetches two things from the backend:

1. **Collector settings** — a single parameter whose YAML value is merged into the top-level collector settings (for example `token`, `platform_host_url`, `default_pulling_interval`).
2. **Plugin settings** — one parameter per plugin under a shared prefix; each parameter's YAML value is an individual plugin block.

The local YAML file is still parsed, but **values from the secrets backend win**. The local `plugins` list can contribute plugins whose names do not collide with plugins loaded from the backend, and the local top-level settings can contribute keys that are not set in the backend — but they can never override a key that is set in the backend. Treat the local file as a fallback, not an override.

## Configuration reference

All keys below live under `secrets_backend:` in `collector_config.yaml`.

| Key                                 | Type   | Default                                    | Description                                                          |
| ----------------------------------- | ------ | ------------------------------------------ | -------------------------------------------------------------------- |
| `provider`                          | string | *(required)*                               | Backend provider code. Must be `AWSSystemsManagerParameterStore`.    |
| `region_name`                       | string | *(resolved from env / IMDS — see below)*   | AWS region the SSM parameters live in.                               |
| `collector_settings_parameter_name` | string | `/odd/collector_config/collector_settings` | Full SSM parameter name holding the top-level collector settings.    |
| `collector_plugins_prefix`          | string | `/odd/collector_config/plugins`            | SSM parameter prefix under which one parameter per plugin is stored. |

The `region_name` is resolved in this order: `AWS_REGION` environment variable → `region_name` in `collector_config.yaml` → the EC2 IMDS (for collectors running on EC2 or EKS). If none of these resolves to a region, the collector process still starts, but the first SSM call raises `botocore.exceptions.NoRegionError` and the collector exits before ingestion begins. Set the region explicitly for non-EC2 deployments.

{% hint style="info" %}
The SDK's settings loader uses `extra="allow"`, so any additional keys under `secrets_backend:` are forwarded to the provider constructor. Future providers may introduce their own keys without needing a schema change here.
{% endhint %}

## Parameter naming convention

For the AWS SSM provider, each plugin in the collector is stored as a separate SSM parameter under `collector_plugins_prefix`. The parameter's **name** is not significant to the loader — the SDK fetches every parameter under the prefix with a recursive `GetParametersByPath` call and parses each value as a plugin YAML block. The convention shown in the SSM example below uses the plugin `name` as the final path segment purely for operator readability; what matters is that each plugin's YAML is stored as its own parameter.

The **value** of each parameter is a YAML document with the same schema as a single entry in the `plugins:` list in `collector_config.yaml`.

## Worked example

### Step 1 — Store the collector settings in SSM

Create one `SecureString` parameter named `/odd/collector_config/collector_settings` with this value:

```yaml
default_pulling_interval: 60
platform_host_url: https://odd.internal.example.com
token: <platform_token>
```

### Step 2 — Store each plugin in SSM

For every data source the collector should pull, create one parameter under `/odd/collector_config/plugins/`. For example, a PostgreSQL plugin at `/odd/collector_config/plugins/postgresql_adapter`:

```yaml
type: postgresql
name: postgresql_adapter
description: ""
database: warehouse
host: db.internal.example.com
port: 5432
user: odd_ro
password: <db_password>
```

### Step 3 — Point the collector at the backend

Keep only the backend pointer (and anything you intentionally want the local file to contribute) in `collector_config.yaml`:

```yaml
secrets_backend:
  provider: AWSSystemsManagerParameterStore
  region_name: eu-central-1
  collector_settings_parameter_name: /odd/collector_config/collector_settings
  collector_plugins_prefix: /odd/collector_config/plugins
```

### Step 4 — Run the collector

On startup the collector will:

1. Parse `collector_config.yaml`.
2. Connect to SSM using the configured region.
3. Fetch `/odd/collector_config/collector_settings` and merge its values into the top-level settings (SSM wins on conflicts).
4. Fetch every parameter under `/odd/collector_config/plugins/` and merge its plugins into the plugin list (SSM wins on name conflicts).
5. Validate the resulting config and start pulling.

## Required IAM permissions

The IAM identity the collector runs under — an EKS service account role, an EC2 instance role, or a user with static credentials — needs at least these permissions on the parameters you created:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameter",
        "ssm:GetParametersByPath"
      ],
      "Resource": [
        "arn:aws:ssm:{region}:{account_id}:parameter/odd/collector_config/collector_settings",
        "arn:aws:ssm:{region}:{account_id}:parameter/odd/collector_config/plugins/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt"
      ],
      "Resource": "arn:aws:kms:{region}:{account_id}:key/{kms_key_id}"
    }
  ]
}
```

The `kms:Decrypt` statement is only needed when the parameters were created as `SecureString` with a customer-managed KMS key (or with the AWS-managed `alias/aws/ssm` key). Plain `String` parameters do not require KMS permissions.

## Known limitations

The AWS SSM backend is a thin wrapper around `boto3`. The following behaviors come from the SDK defaults and cannot currently be overridden from `collector_config.yaml`. Review each against your deployment before relying on the backend in production.

{% hint style="danger" %}
**The backend loads at most 10 plugin parameters from SSM.** The SDK calls `ssm.get_parameters_by_path(..., Recursive=True)` once and reads `response["Parameters"]` without paginating. The AWS default `MaxResults` for this call is 10. A collector whose `collector_plugins_prefix` contains 11 or more plugin parameters will silently start with only the first 10; everything past that point is dropped with no error and no log warning.

Mitigations until the SDK adds pagination:

* Keep the plugin count under 10 per collector. Split larger fleets into multiple collectors, each with its own prefix.
* After deploying, compare the plugin count in SSM against the plugin count logged at collector startup — a mismatch means you have hit the cap.
  {% endhint %}

{% hint style="warning" %}
**No custom SSM endpoint can be configured.** The SDK constructs the SSM client with `boto3.client("ssm", region_name=...)` and does not expose `endpoint_url`. Deployments that need to reach SSM via a VPC interface endpoint, a private DNS override, LocalStack, or any non-default endpoint must rely on system-level DNS / network configuration — there is no `secrets_backend.endpoint_url` key to set.
{% endhint %}

{% hint style="warning" %}
**No timeout or retry overrides.** The backend does not pass a `botocore.config.Config`, so the SDK's default connect timeout, read timeout, and retry mode apply. On a partitioned or slow SSM endpoint the collector can block on startup for the full default timeout before failing. There is no `secrets_backend.connect_timeout` / `read_timeout` / `retries` key.
{% endhint %}

{% hint style="warning" %}
**SSM parameter values are parsed as YAML without schema validation at fetch time.** A malformed YAML document in any parameter under `collector_plugins_prefix` will raise during `safe_load` and abort collector startup with a parsing error, not a message pointing at the offending parameter. Treat SSM parameter edits the same way you would treat edits to `collector_config.yaml` — validate the YAML locally before putting the parameter.
{% endhint %}