# Build and run ODD Collectors

For instructions on how to run the ODD Platform and ODD Collectors locally in a Docker environment, please follow [Try locally](/~/changes/pcY8r1Ougw5Sv8zDaz8G/configuration-and-deployment/trylocally.md) article.

## ODD Collectors tech stack

There are 3 main collectors at the moment:

* \*\*\*\*[**ODD Collector**](https://github.com/opendatadiscovery/odd-collector) \*\*\*\* — covering databases, BI tools, data warehouses, etc
* \*\*\*\*[**ODD Collector AWS**](https://github.com/opendatadiscovery/odd-collector-aws) — covering AWS services
* \*\*\*\*[**ODD Collector GCP**](https://github.com/opendatadiscovery/odd-collector-gcp) \*\*\*\* — covering GCP services

While ODD Collector AWS and ODD Collector GCP use [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) and Google SDKs respectively, ODD Collector has a bunch dependencies for each data source.

General tech stack is:

* Python
* Poetry
* asyncio

## Prerequisites

* Python 3.9.1
* [Poetry](https://python-poetry.org/) 1.2.0
* [Docker Engine 19.03.0+](https://docs.docker.com/engine/install/)
* preferably the latest [docker-compose](https://docs.docker.com/compose/install/)

## Build ODD Collector into Docker container

Fork and clone a repository if you haven't done it already.

```shell
git clone https://github.com/{username}/odd-collector.git
```

Go into the repository's root directory

```shell
cd odd-collector
```

Run the following command, replacing `<tag>` with any tag name you'd like

```shell
docker build . -t odd-collector:<tag>
```

## Run ODD Collector locally

### Run ODD Platform locally as a target for ODD Collector

In order to run ODD Platform locally please follow [this guide](/~/changes/pcY8r1Ougw5Sv8zDaz8G/configuration-and-deployment/trylocally.md).

### Activate environment

Go into the repository's root directory

```shell
cd odd-collector
```

Run following commands to create local python environment and install dependencies

```shell
poetry install
```

Change your python context to created one.

```shell
poetry shell
```

### Configure ODD Collector to send request to target catalog

Create collector in the ODD Platform and copy created token using [this guide](/~/changes/pcY8r1Ougw5Sv8zDaz8G/configuration-and-deployment/trylocally.md#create-collector-entity).

Configure `collector-config.yaml` using [this](https://github.com/opendatadiscovery/odd-collector/tree/main/config_examples) as an example. Replace `<COLLECTOR_TOKEN>` with token obtained in the previous step.

```yaml
default_pulling_interval: 10
token: <COLLECTOR_TOKEN>
platform_host_url: http://localhost:8080
plugins:
  - type: my_adapter
    some_field_one: str
    some_field_two: int
```

### Run ODD Collector

Run ODD Collector locally using following command:

```shell
sh ./start.sh
```

## How to implement new integration

### Create configuration entry for new integration

Add new integration plugin derived from `BasePlugin` and register it in `PluginFactory`

{% code title="domain/plugin.py" %}

```python
...
class MyAdapterPlugin(BasePlugin):
    type: Literal["my_adapter"]
    some_field_one: str
    some_field_two: int

...

PLUGIN_FACTORY: PluginFactory = {
    ...
    "my_adapter": MyAdapterPlugin,
}
```

{% endcode %}

### Implement new integration

Each adapter module (i.e `odd_collector.adapters.my_adapter`) must have `adapter.py` python file. That file must have class derived from `AbstractAdapter.`

{% code title="adapters/new\_adapter/adapter.py" %}

```python
from odd_collector_sdk.domain.adapter import AbstractAdapter

class Adapter(AbstractAdapter):
    def __init__(
        self,
        config: MyAdapterPlugin
    ):
        ...

    def get_data_source_oddrn(self) -> str:
        ...

    async def get_data_entity_list(self) -> DataEntityList:
        ...
```

{% endcode %}

### Make pull request to the origin repository

Please use [this guide](/~/changes/pcY8r1Ougw5Sv8zDaz8G/developer-guides/how-to-contribute.md#forking-a-repository) for making forks and pull requests

## Troubleshooting

### Running ODD Collector on M1

libraries `pyodbc` , `confluent-kafka` and `grpcio` have problem during installing and building project on M1 Macbooks.

* [mkleehammer/pyodbc#846](https://github.com/mkleehammer/pyodbc/issues/846)
* [confluentinc/confluent-kafka-python#1190](https://github.com/confluentinc/confluent-kafka-python/issues/1190)
* [grpc/grpc#25082](https://github.com/grpc/grpc/issues/25082)

Possible solution:

{% hint style="info" %}
The easiest way is to add all export statements to your .bashrc/.zshrc file
{% endhint %}

```shell
# pyodbc dependencies
brew install unixodbc freetds openssl

# cunfluent-kafka
export LDFLAGS="-L/opt/homebrew/lib  -L/opt/homebrew/Cellar/unixodbc/2.3.11/include -L/opt/homebrew/opt/freetds/lib -L/opt/homebrew/opt/openssl@3/lib"export CFLAGS="-I/opt/homebrew/Cellar/unixodbc/2.3.11/include -I/opt/homebrew/opt/freetds/include"export CPPFLAGS="-I/opt/homebrew/include -I/opt/homebrew/Cellar/unixodbc/2.3.11/include -I/opt/homebrew/opt/openssl@3/include"
brew install librdkafka
export C_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/1.9.0/include
export LIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/1.9.0/lib
export PATH="/opt/homebrew/opt/openssl@3/bin:$PATH"# grpcio
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1
```


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.opendatadiscovery.org/~/changes/pcY8r1Ougw5Sv8zDaz8G/developer-guides/build-and-run/build-and-run-odd-collectors.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.