Skip to main content

Autoloading existing Dagster definitions

info

Dagster Components is currently in Release Candidate status. APIs are stable, with broader integration coverage and full feature parity in active development and coming soon. Check it out and let us know what you think in the #dg-components channel in the Dagster Community Slack!

note

This guide covers using existing Dagster definitions with a dg-compatible project. To convert an existing project to use dg, see "Converting an existing project to use dg".

In projects created with create-dagster, all definitions are typically kept in the defs/ directory. However, if you've converted an existing project to use dg, you may have definitions located in various other modules. This guide will show you how to move these existing definitions into the defs directory in a way that will allow them to be automatically loaded.

Example project structure​

Let's walk through an example of migrating your existing definitions, with a project that has the following structure:

tree
.
├── my_existing_project
│   ├── __init__.py
│   ├── analytics
│   │   ├── __init__.py
│   │   ├── assets.py
│   │   └── jobs.py
│   ├── definitions.py
│   ├── defs
│   │   └── __init__.py
│   └── elt
│       ├── __init__.py
│       ├── assets.py
│       └── jobs.py
├── pyproject.toml
└── README.md

5 directories, 11 files

At the top level, we load definitions from various modules:

my_existing_project/definitions.py


from pathlib import Path
import dagster as dg
import my_existing_project.defs
from my_existing_project.analytics import assets as analytics_assets
from my_existing_project.analytics.jobs import (
    regenerate_analytics_hourly_schedule,
    regenerate_analytics_job,
)
from my_existing_project.elt import assets as elt_assets
from my_existing_project.elt.jobs import sync_tables_daily_schedule, sync_tables_job

defs = dg.Definitions.merge(
    dg.Definitions(
        assets=dg.load_assets_from_modules([elt_assets, analytics_assets]),
        jobs=[sync_tables_job, regenerate_analytics_job],
        schedules=[sync_tables_daily_schedule, regenerate_analytics_hourly_schedule],
    ),
    dg.components.load_from_defs_folder(project_root=Path(__file__).parent.parent),
)

Each of these modules contains a variety of Dagster definitions, including assets, jobs, and schedules.

Let's migrate the elt module to a component.

Move definitions to defs​

We'll start by moving the top-level elt module into defs/elt:

mv my_existing_project/elt/* my_existing_project/defs/elt

Now that our definitions are in the defs directory, we can update the root definitions.py file to no longer explicitly load the elt module's Definitions:

my_existing_project/definitions.py
from pathlib import Path

from my_existing_project.analytics import assets as analytics_assets
from my_existing_project.analytics.jobs import (
    regenerate_analytics_hourly_schedule,
    regenerate_analytics_job,
)

import dagster as dg

defs = dg.Definitions.merge(
    dg.Definitions(
        assets=dg.load_assets_from_modules([analytics_assets]),
        jobs=[regenerate_analytics_job],
        schedules=[regenerate_analytics_hourly_schedule],
    ),
    dg.load_from_defs_folder(project_root=Path(__file__).parent.parent),
)

Our project structure now looks like this:

tree
.
├── my_existing_project
│   ├── __init__.py
│   ├── analytics
│   │   ├── __init__.py
│   │   ├── assets.py
│   │   └── jobs.py
│   ├── definitions.py
│   └── defs
│       ├── __init__.py
│       └── elt
│           ├── __init__.py
│           ├── assets.py
│           └── jobs.py
├── pyproject.toml
├── README.md
└── uv.lock

5 directories, 12 files

The load_defs command in our definitions.py file will automatically load any definitions found within the defs module. This means that all of our definitions are now automatically loaded, with no need to import them up into any top-level organization scheme.

We can repeat the same process for our other modules.

Fully migrated project structure​

Once each of our definitions modules are migrated, our project is left with a standardized structure:

tree
.
├── my_existing_project
│   ├── __init__.py
│   ├── definitions.py
│   └── defs
│       ├── __init__.py
│       ├── analytics
│       │   ├── __init__.py
│       │   ├── assets.py
│       │   └── jobs.py
│       └── elt
│           ├── __init__.py
│           ├── assets.py
│           └── jobs.py
├── pyproject.toml
├── README.md
└── uv.lock

5 directories, 12 files

Our project root now only constructs definitions from the defs module:

my_existing_project/definitions.py
from pathlib import Path
import dagster as dg

defs = dg.load_from_defs_folder(project_root=Path(__file__).parent.parent)

We can run dg list defs to confirm that all of our definitions are being loaded correctly:

my_existing_project/definitions.py
dg list defs

┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section   ┃ Definitions                                                   ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets    │ ┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┓ │
│           │ ┃ Key                ┃ Group   ┃ Deps ┃ Kinds ┃ Description ┃ │
│           │ ┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━┩ │
│           │ │ customers_table    │ default │      │       │             │ │
│           │ ├────────────────────┼─────────┼──────┼───────┼─────────────┤ │
│           │ │ my_analytics_asset │ default │      │       │             │ │
│           │ ├────────────────────┼─────────┼──────┼───────┼─────────────┤ │
│           │ │ orders_table       │ default │      │       │             │ │
│           │ ├────────────────────┼─────────┼──────┼───────┼─────────────┤ │
│           │ │ products_table     │ default │      │       │             │ │
│           │ └────────────────────┴─────────┴──────┴───────┴─────────────┘ │
│ Jobs      │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓                    │
│           │ ┃ Key                      ┃ Description ┃                    │
│           │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩                    │
│           │ │ regenerate_analytics_job │             │                    │
│           │ ├──────────────────────────┼─────────────┤                    │
│           │ │ sync_tables_job          │             │                    │
│           │ └──────────────────────────┴─────────────┘                    │
│ Schedules │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓             │
│           │ ┃ Key                               ┃ Cron      ┃             │
│           │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩             │
│           │ │ regenerate_analytics_job_schedule │ 0 * * * * │             │
│           │ ├───────────────────────────────────┼───────────┤             │
│           │ │ sync_tables_job_schedule          │ 0 0 * * * │             │
│           │ └───────────────────────────────────┴───────────┘             │
└───────────┴───────────────────────────────────────────────────────────────┘

Next steps​