Configure ODD Platform
This section defines how to configure ODD Platform in order to leverage all of its functionality and features.
Last updated
This section defines how to configure ODD Platform in order to leverage all of its functionality and features.
Last updated
There are two ways to configure the Platform:
Environment variables are used for simple entries
Configuring via YAML can come in handy when it is necessary to define a complex configuration block (e.g OAuth2 authentication or logging levels).
For all of its features ODD Platform uses PostgreSQL database and PostgreSQL database only. These variables are needed to be defined to connect ODD Platform to database:
spring.datasource.url: of your PostgreSQL database. Default value is jdbc:postgresql://127.0.0.1:5432/odd-platform
spring.datasource.username: your PostgreSQL user's name. Default value is odd-platform
spring.datasource.password: your PostgreSQL user's password. Default value is odd-platform-password
These variables are optional (by default, they have the same value as spring.datasource) and will be used to connect to PostgreSQL and store Lookup Tables :
spring.custom-datasource.url: of your PostgreSQL database where we store Lookup Tables. Default value is jdbc:postgresql://127.0.0.1:5432/odd-platform. Note: you can specify any {database_host}, {database_port} or {database_name} but schema, where Lookup Tables are stored always is lookup_tables_schema.
spring.custom-datasource.username: your PostgreSQL user's name for custom-datasource. Default value is odd-platform
spring.custom-datasource.password: your PostgreSQL user's password for custom-datasource. Default value is odd-platform-password
So that your database connection defining block would look like this:
Please follow Enable security section for enabling security in ODD Platform.
ODD Platform is able to keep users' sessions in several places such as in memory, PostgreSQL database or Redis. A session provider can be set via session.provider variable with following expected values:
IN_MEMORY: Local in-memory storage. ODD Platform defaults to this value
INTERNAL_POSTGRESQL: Underlying PostgreSQL database
In memory (default)
Internal PostgreSQL
Redis
In order to connect to Redis following variables are needed to be defined:
spring.redis.host: Redis host
spring.redis.port: Redis port
spring.redis.username: Redis user's name
spring.redis.password: Redis user's password
spring.redis.database: Redis database index
YAML for Redis session provider
metrics.export.enabled: Must be set to true
metrics.export.otlp-endpoint: OTLP Collector endpoint
Name of the entity upon which alert has been created
Data source and namespace of an entity
Owners of an entity
Possibly affected entities
ODD Platform uses the PostgreSQL replication mechanism to be able to send a notification even if there's a network lag occurred or the Platform crushes. In order to enable this functionality, an underlying PostgreSQL database needs to be configured as well.
To configure the database, add the following entries to the postgresql.conf file:
Or if the replication mechanism is already configured, just increment the max_wal_senders and max_replication_slots numbers.
ODD Platform database user must be granted with replication permissions:
Following variables need to be defined:
notifications.enabled: must be set to true. Defaults to false
notifications.message.downstream-entities-depth: limits the amount of fetching of affected data entities in terms of lineage graph level. Defaults to 1
notifications.wal.replication-slot-name: PostgreSQL replication slot name will be created if it doesn't exist yet. Defaults to odd_platform_replication_slot
notifications.wal.publication-name: PostgreSQL publication name will be created if it doesn't exist yet. Defaults to odd_platform_publication_alert
notifications.receivers.webhook.url: Generic webhook URL
notifications.receivers.email.host: the SMTP server.
notifications.receivers.email.port: the port used for the email protocol (SMTP, IMAP, or POP3)
notifications.receivers.email.protocol: the email protocol (e.g., SMTP, SMTPS, IMAP, IMAPS, POP3, POP3S)
notifications.receivers.email.smtp.auth: a boolean value (true or false) indicating whether the SMTP server requires authentication
notifications.receivers.email.smtp.starttls: a boolean indicating whether to use STARTTLS, a security protocol that upgrades an unencrypted connection to an encrypted one
notifications.receivers.email.password: the password used for email authentication
notifications.receivers.email.sender: the email address sending the notifications
notifications.receivers.email.notification.emails: the list of recipients for the email notifications
odd.platform-base-url: ODD Platform URL to be used in alert messages' hyperlinks.
ODD Platform configuration would look like this:
ODD Platform doesn't clean up replication slot it has created. If you need to disable Alert Notification functionality, please perform the following steps along with disabling a feature on a ODD Platform side
In order to remove replication slot and publication, these SQL queries must be run against the database:
where <> is a name of replication slot defined in the ODD Platform. Default is odd_platform_replication_slot
where <> is a name of publication defined in the ODD Platform. Default is odd_platform_publication_alert
Data collaboration feature allows users to initiate discussion about specific data entity in messengers directly from the ODD Platform. Thread replies are tracked by ODD Platform and saved in it, allowing users to retrieve conversation's context and decisions from one place.
Select a workspace you want to add an application to and click Next
Enter the following manifest into the YAML section, replace the <ODD_PLATFORM_BASE_URL> with URL of your ODD Platform deployment and click Next
Review your application's scopes and permissions and click Create
Proceed with Slack instructions on how to install application into workspace and you should be good to go.
Following variables need to be defined:
datacollaboration.enabled: must be set to true. Defaults to false
datacollaboration.receive-event-advisory-lock-id: PostgreSQL advisory lock id for a job, which translates events from messengers to messages. Defaults to 110
datacollaboration.sender-message-advisory-lock-id: PostgreSQL advisory lock id for a job, which sends messages created in the platform to messengers. Defaults to 120
datacollaboration.message-partition-period: time interval in days for a message table partition in PostgreSQL. Defaults to 30
datacollaboration.sending-messages-retry-count: how many times the Platform will attempt to send a message to provider. Cannot be less than zero. Defaults to 3
datacollaboration.slack-oauth-token: Slack application OAuth token used for communicating with Slack. Can be retrieved in the OAuth & Permissions section of a Slack application.\
The Housekeeping module is enabled (enabled: true) allowing for automated maintenance tasks. The Time-To-Live (TTL) settings define the retention period for following data categories:
Resolved Alerts: data related to resolved alerts will be retained for 30 days.
Search Facets: historical search facets data will be maintained for 30 days.
Data Entity Deletion: information about deleted data entities will be preserved for 30 days.
Stale metadata refers to the situation where the metadata becomes outdated within ODD Platform. This indicates a lack of information regarding updates from the source, which could occur due to issues such as collector not functioning as planned, the collector being deactivated or other issues in the source data system that result in the unavailability of that metadata.
By default, the refresh period is set to 7 days in the configuration file.
This indicates that if the platform received information from the source over 7 days ago, the item would be labeled as "Stale" within the platform. ODD users have the flexibility to adjust this period to better suit their needs - whether opting for a shorter or longer timeframe.
Logs provide detailed information about errors in the application helping its users quickly identify and fix problems. Setting up logging is recommended for ensuring operational excellence, system reliability, effective monitoring and troubleshooting. Here is a code snippet for setting up logs in ODD Platform:
Setting the logging level to info allows you to see useful messages about the platform’s functioning without being overwhelmed by too much detail as with trace or debug or missing important issues as with warn or higher level.
However, feel free to adjust the logging level as needed to get more or less information based on your specific requirements.
For M2M communication with the API, a secret is provided to the ODD platform before deployment. This allows the platform to bypass identity providers. When a user request is sent to the API with the correct secret, the API will respond without any issues.
This functionality is not the preferred method and is disabled by default, but it can be enabled and configured when needed.
REDIS: .
Some of metadata ODD Platform ingests can be conveniently represented in a shape of time-series chart, for example, an amount of data in a MySQL table or a physical size of a Redshift database. ODD Platform pushes metadata to the as a telemetry in order to be able to create charts in , or any other backend that supports . These variables are needed to be set in order to leverage this functionality:
Any alert that is created inside the platform can be sent via webhook and/or and/or email notifications (via , , etc). Such notifications contain information such as:
PostgreSQL database must be in order to leverage the replication mechanism of the Platform along with the granting the database user replication permissions.
notifications.wal.advisory-lock-id: ODD Platform uses in order to make sure that in a case of horizontal scaling only one instance of the Platform processes alert messages. This setting defines advisory lock id. Defaults to 100
notifications.receivers.slack.url: URL
At the moment ODD Platform supports only Slack as a target messenger. It uses Slack APIs to send messages and to receive message's thread replies.
Go to the website and click on Create New App -> From an app manifest
These settings ensure that unnecessary or stale data is automatically cleaned up after the specified time periods. allows for customization based on specific business needs and data retention policies.
