Skip to main content

Adding attributes to assets in a subdirectory

Attaching and modifying meta information on definitions such as assets is important in Dagster projects. You want to have groups, teams, owners, kinds, tags, and metadata set correctly to organize definitions and ensure that other tools and processes that rely on them correctly function.

Within a dg-driven defs project layout, you can apply attribute transformations at any point in the directory structure. This supports uses cases ranging from ensuring that all definitions in a particular folder have an owner set to a particular team to more complex workflows involving applying a group conditionally based on other properties of the definition.

Example​

First, we'll look at a project with assets defined in a subdirectory. You can quickly replicate this example by running the following commands:

uvx -U create-dagster project my-project \
    && cd my-project/src \
    && dg scaffold defs dagster.asset team_a/subproject/a.py \
    && dg scaffold defs dagster.asset team_a/b.py \
    && dg scaffold defs dagster.asset team_b/c.py

There are three assets defined in various subdirectories:

dg list defs
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions                                    ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets  │ ┏━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┓ │
│         │ ┃ Key ┃ Group   ┃ Deps ┃ Kinds ┃ Description ┃ │
│         │ ┡━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━┩ │
│         │ │ a   │ default │      │       │             │ │
│         │ ├─────┼─────────┼──────┼───────┼─────────────┤ │
│         │ │ b   │ default │      │       │             │ │
│         │ ├─────┼─────────┼──────┼───────┼─────────────┤ │
│         │ │ c   │ default │      │       │             │ │
│         │ └─────┴─────────┴──────┴───────┴─────────────┘ │
└─────────┴────────────────────────────────────────────────┘
tree my_project/defs
my_project/defs
├── __init__.py
├── team_a
│   ├── b.py
│   └── subproject
│       └── a.py
└── team_b
    └── c.py

4 directories, 4 files

Let's say we want to add a team_a asset group to all assets in the team_a folder. We can add a defs.yaml file to the my_project/defs/team_a directory with the following contents:

type: dagster.DefsFolderComponent

post_processing:
  assets:
    - target: "*"
      attributes:
        group_name: "team_a"
tree my_project/defs
my_project/defs
├── __init__.py
├── team_a
│   ├── b.py
│   ├── defs.yaml
│   └── subproject
│       └── a.py
└── team_b
    └── c.py

4 directories, 5 files

Finally, we can run dg list defs again to see the new group applied to the assets in the team_a subdirectory:

dg list defs
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions                                    ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets  │ ┏━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┓ │
│         │ ┃ Key ┃ Group   ┃ Deps ┃ Kinds ┃ Description ┃ │
│         │ ┡━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━┩ │
│         │ │ a   │ team_a  │      │       │             │ │
│         │ ├─────┼─────────┼──────┼───────┼─────────────┤ │
│         │ │ b   │ team_a  │      │       │             │ │
│         │ ├─────┼─────────┼──────┼───────┼─────────────┤ │
│         │ │ c   │ default │      │       │             │ │
│         │ └─────┴─────────┴──────┴───────┴─────────────┘ │
└─────────┴────────────────────────────────────────────────┘

Advanced usage​

The configuration for the dagster.components.DefsFolderComponent can be more complex than the example above. You can apply multiple transforms to the assets in the subdirectory, each of which will be processed in order.

You can also target the configuration to a specific selection of assets using the target field. This field uses Dagster's Selection Syntax. All selections are evaluated against the assets defined within the subdirectory.