Docs Development

The DJ project uses the Hugo framework for building the docs site and all of the pages are defined as markdown files. The deployed docs site is actually a combination of multiple sites for different versions of DJ. This page will help to understand how to modify, locally test, and deploy DJ docs.

Running the Docs Site Locally #

Clone the DJ repo locally.

git clone https://github.com/datajunction/dj

Clone the hugo-doks theme submodule.

git submodule update --init docs/themes/doks

Change into the docs directory.

cd dj/docs

Using hugo, start a local server for the specific DJ docs version you want to view.

hugo serve --contentDir content/0.1.0/

In addition to different versions of the docs site, there is a Versions page that is deployed as a small sub-site to the /versions/ route. This is a connecting page that all versions of the site link to and lists release notes for each DJ release. The following command will serve the versions page.

hugo serve --contentDir content/versions/

Updating Content #

Content for all pages are stored in reStructuredText files located at docs/content/. A sub-directory can be found for each version of DJ. In order to update content for an existing docs page, open a PR against that page. In most cases, it’s enough to add updates for the latest version, however when necessary, you can additionally make changes to previous versions of an existing page.

Here are some tips on deciding on whether you should backport changes to previous versions.

• Open a PR against the latest version of a page to…

  • clarify or extend an existing section

  • add a page or section for a new feature

  • add a tutorial

• Open a PR against all versions of a page to…

  • fix broken links

  • annotate feature descriptions with version-specific peculiarities

Updating the DataJunction API Specification Page #

The DataJunction API Specification page is generated using the openapi.json spec file and the widdershins CLI tool.

Install dj from source.

pip install .

Use the generate-openapi.py script to update the openapi.json file.

python ./scripts/generate-openapi.py -o openapi.json

Use widdershins to generate a markdown file from the openapi.json file.

widdershins openapi.json -o docs/content/0.1.0/docs/developers/the-datajunction-api-specification.md --code=true --omitBody=true --summary=true

Launch the hugo server locally to visually inspect the datajunction specification page.

cd docs
hugo serve --contentDir=content/0.1.0

The following are a few manual cleanups that need to be performed on the generated file.

Replace all of the front-matter with a single value weight: 1.

---
- title: DJ server v0.0.post1.dev763+gdf7a15b.d20230411
- language_tabs:
-   - shell: Shell
-   - http: HTTP
-   - javascript: JavaScript
-   - ruby: Ruby
-   - python: Python
-   - php: PHP
-   - java: Java
-   - go: Go
- toc_footers: []
- includes: []
- search: true
- highlight_theme: darkula
- headingLevel: 2
+ weight: 11
---

Replace the header with an H1 page title.

- <!-- Generator: Widdershins v4.0.1 -->
- 
- <h1 id="dj-server">DJ server v0.0.post1.dev763+gdf7a15b.d20230411</h1>
- 
- > Scroll down for code samples, example requests and responses. Select a language for code samples from the tabs above or the mobile navigation menu.
- 
- A DataJunction metrics layer
- 
- License: <a href="https://mit-license.org/">MIT License</a>
- 
- <h1 id="dj-server-default">Default</h1>
+ # The DataJunction API Specification
+
+ License: <a href="https://mit-license.org/">MIT License</a>

Adding New Pages #

A new page can be added by creating an .md file anywhere within the docs/content directory. By default, the page will appear in the side navigation menu and the title will be the file name converted to a display format. For example, the content for this page is defined in a file named docs-development.md and the title automatically appears in the menu as Docs Development. The title can also be overriden by setting it explicitly in the pages front-matter.

example.md

---
title: My Docs Page
---

A weight can also be added to position the page in the side menu. Items with higher weights are displayed below items with lower weights.

---
title: My Docs Page
weight: 10
---

Deployment #

In order to make it easier to build the entire docs-site (including multiple versions) a build script can be found at docs/build-docs.sh. The script takes two positional arguments. You can run the script without any arguments to see the help output.

./build-docs.sh

output

Build a specified version of the DJ docs

Syntax: ./build-docs.sh <VERSION> <SET_AS_LATEST>

VERSION: The version of the docs site to build, i.e. 0.1.0
SET_AS_LATEST: Additionally build the specified version as the default/latest docs site, i.e. true

For example, by running the following you can build the docs for versions 0.1.0, 0.1.1, and 0.1.2, building 0.1.2 as the latest version that’s shown when navigating to the root site.

./build-docs.sh 0.1.2 true

Start building sites …
Total in 2496 ms

./build-docs.sh 0.1.1 false

Start building sites …
Total in 2472 ms

./build-docs.sh 0.1.0 false

Start building sites …
Total in 2549 ms

After the builds are completed, the entire site will be deployed to the docs/public directory. Any local server can be used to render the entire site from the public directory locally such as the popular Live Server extension for VSCode.

Adding Excalidraw Diagrams #

The DataJunction docs site uses excalidraw to create diagrams and renders them within the docs pages using an excalidraw shortcode. You can create new excalidraw diagrams by developing them either on excalidraw.com or using the Excalidraw VSCode extension.

To add diagrams to the docs site, export the diagram as a .excalidraw file and place it in the static/excalidraw-drawings/ directory. You can then add the diagram using the excalidraw shortcode.

As an example, if you’ve added an excalidraw file named excalidraw_is_awesome.excalidraw, you can add it to any markdown page with the following shortcode.

{{< excalidraw excalidraw_is_awesome >}}

Here’s an example of how the excalidraw diagram will appear.