Collection of tasks for running containerized models. Designed to be used both in Prefect workflows and as modular, useful pieces of code.

Installation

The collection can be installed using:

pip install container-collection

We recommend using Poetry to manage and install dependencies. To install into your Poetry project, use:

poetry add container-collection

Usage

Prefect workflows

All tasks in this collection are wrapped in a Prefect @task decorator, and can be used directly in a Prefect @flow. Running tasks within a Prefect flow enables you to take advantage of features such as automatically retrying failed tasks, monitoring workflow states, running tasks concurrently, deploying and scheduling flows, and more.

from prefect import flow
from container_collection.<module_name> import <task_name>

@flow
def run_flow():
    <task_name>()

if __name__ == "__main__":
    run_flow()

See cell-abm-pipeline for examples of using tasks from different collections to build a pipeline for simulating and analyzing agent-based model data.

Individual tasks

Not all use cases require a full workflow. Tasks in this collection can be used without the Prefect @task decorator by simply importing directly from the module:

from container_collection.<module_name>.<task_name> import <task_name>

def main():
    <task_name>()

if __name__ == "__main__":
    main()

or using the .fn() method:

from container_collection.<module_name> import <task_name>

def main():
    <task_name>.fn()

if __name__ == "__main__":
    main()

Module reference

container_collection.batch	Tasks for working with AWS Batch.
container_collection.docker	Tasks for working with Docker.
container_collection.fargate	Tasks for working with AWS ECS Fargate.
container_collection.manifest	Tasks for working with manifest files.
container_collection.template	Tasks for working with template files.