Setting Up Back-End#

Pre-requisites#

Before setting up the back-end for development, you will need to have the following installed:

See Setting Up Development Tools for more information on setting up your development environment.

Setting Up MongoDB#

Assuming you have MongoDB set up, you will need to start it up.

Using Docker#

If you are using the Docker Compose setup, you can start up MongoDB by running:

docker compose up -d mongodb

Non-docker Setup#

To install MongoDB locally without Docker, follow the instructions here

Follow the instructions to install and start up MongoDB. You should be able to connect to the MongoDB instance at mongodb://localhost:27017.

Setting Up MinIO#

Using Docker#

If you are using the Docker Compose setup, you can start up MinIO by running:

docker compose up -d minio

Non-docker Setup#

To install Minio locally without Docker, follow the instructions here

Once started, you should be able to connect to the MinIO API instance at http://localhost:9000. The console can be accessed at http://localhost:9090.

Installing Dependencies#

To install, change to the back-end directory and run:

python -m venv venv
source venv/bin/activate
poetry install

The core dependencies are listed in back-end/pyproject.toml. The key dependencies are:

  • FastAPI: a modern, fast (high-performance), web framework for building APIs with Python 3.6+ based on standard Python type hints.

  • Pydantic: data validation and settings management using Python type hinting.

  • Motor: an asynchronous Python driver for MongoDB.

  • Minio: a Python client for S3-compatible object storage.

  • ClearML: a Python client for ClearML.

  • Jinja2: a modern and designer-friendly templating language for Python.

  • BeautifulSoup: a Python library for parsing HTML and XML documents.

  • Kubernetes: a Python client for Kubernetes.

  • The core development dependencies are:

  • Pytest: a testing framework for Python.

  • Hypothesis: a library for property-based testing.

  • Black: a Python code formatter.

  • isort: a Python utility / library to sort imports.

Setting up Environment Variables#

Configuration for the app is done by dotenv files. The dotenv file is located in back-end/src/config/.env. If you do not have this file, you will need to create it as a copy of back-end/src/config/.env.public and fill in the values (as we do not want to commit the dotenv file to the repository to avoid leaking sensitive secrets, and thus we gitignored the .env file).

Encryption and Decryption#

To supply environment variables for CI, we encrypt the dotenv file using gpg (with a passphrase).

To encrypt the dotenv file, run the following in the config directory:

sh encrypt-env.sh <passphrase>

Substitute <passphrase> with the passphrase you want to use. But note that the Github repository stores the passphrase as a secret, so you will need to use the same passphrase as the one in the Github repository (or update the Github repository with the new passphrase).

To decrypt the dotenv file, run the following in the config directory:

sh decrypt-env.sh <passphrase>

ClearML Credentials#

The backend needs to be able to connect to ClearML for integration. To do this, you will need to set the following environment variables:

  • CLEARML_API_KEY: the API key for your ClearML account

  • CLEARML_API_SECRET: the API secret for your ClearML account

  • CLEARML_API_HOST: the API host for your ClearML account

Running the App#

Local (Non-docker) Setup#

To run the app in development mode, run:

poe boot

This will start up the app on port 7070. You can access the app at http://localhost:7070.

Docker Setup#

Run the following command to start up the app:

docker compose up -d back-end

This starts up the app on port 8080. Currently hot-reloading is not supported, so you will need to restart the docker compose (which will rebuild the image) to see changes.