YAML Projects

DJ entities can be managed through YAML definitions. This is a versatile feature that enables change review and more holistic testing before deploying to production. Using source-controlled YAML definitions provide a more structured approach to development, allowing you to review and audit changes.

Setup Guide #

From Existing #

If you’ve already started developing DJ entities through the UI or a different client, you can export the existing entities to a YAML project to get started quickly. Note that this process only supports exporting a single namespace at a time.

1. Export: Use the DJ CLI (installed when you install the DJ Python client) to export your DJ entities to a YAML project. This snippet will export the default namespace to a YAML project in the ./example_project directory:

dj pull default ./example_project

2. Validation: Once you’ve made changes to the YAML files, you can validate those changes with:

dj deploy ./example_project --dryrun

3. Deployment: When satisfied, you can deploy the changes like this:

dj deploy ./example_project

From Scratch #

You can also create a YAML project from scratch.

1. Create a dj.yaml file with project metadata. See additional details on project metadata fields below. Example:

name: Roads Project
description: Roads example project
prefix: projects.roads
mode: published
tags:
  - name: deprecated
    description: This node is deprecated
    tag_type: Maintenance
build:
  priority:
    - roads.date
    - roads.repair_order_details
    - roads.contractors
    - roads.hard_hats

2. Create YAML files that represent each node. Use file infixes to define node types, like foo.dimension.yaml or foo.transform.yaml). Here is an example for the roads.date_dim node, in file `./roads/

display_name: Date
description: Date dimension
query: |
  SELECT
    dateint,
    month,
    year,
    day
  FROM ${prefix}roads.date
primary_key:
  - dateint

Project Metadata Fields #

Node YAML Fields Overview #

Source Node YAML #

Transform / Dimension Node YAML #

Metric Node YAML #

Cube Node YAML #

Dimension Link YAML #

There are two types of dimension links, join links and reference links. The fields available for each of the two are slightly different. Here’s an example of a node with both:

description: Hard hat dimension
display_name: Local Hard Hats
query: ...
primary_key: ...
dimension_links:
  - type: join
    node_column: state_id
    dimension_node: ${prefix}roads.us_state
  - type: reference
    node_column: birth_date
    dimension: ${prefix}roads.date_dim.dateint
    role: birth_date

Columns YAML #

The columns section can be included if additional column-level settings are needed on the node.

Attributes

Column-level attributes like dimension can be configured like this:

columns:
- name: is_clicked
  attributes:
  - dimension

Display Name

A column can be given a custom display name like this:

columns:
- name: is_clicked
  display_name: Clicked?

Partitions

Partition columns can be configured like this:

columns:
- name: utc_date
  partition:
    format: yyyyMMdd
    granularity: day
    type_: temporal