Angular Custom Builders: Markdown + Angular

Since Angular version 8, the Builders API has become stable. In this post, we'll explore how to use it to extend, or add, new commands to the Angular CLI.

Let's build an example project. We'll create a builder that will allow us to use markdown that can be transformed into an html template file for our components. We will also add a requirement: remove all the generated files after building the project.

We'll start by cloning a starter project for angular builders:

git clone git@github.com:flakolefluk/angular-builder-starter.git md-builder // rename the folder to a reasonable name for your project
cd md-builder
npm install

Let's take a look at our folder structure.

src/builders.json

{
  "$schema": "@angular-devkit/architect/src/builders-schema.json",
  "builders": {
    "build": {
      "implementation": "./build",
      "schema": "./build/schema.json",
      "description": "Custom Builder"
    }
  }
}

builders.json contains the required information for the builder that contains our package. The builder will contain a name- in this case build- the location of the builder /build/index.ts or build, a description, and the location of the schema. The schema will provide some information about the builder, and information about the parameters that can be passed to the CLI when running the builder. It's important that package.json points to the builders.json location. Also, remember to rename the package to our desired name for the builder. We'll use this name later to link the package.

{
  "name": "@flakolefluk/md-builder",
  "version": "0.0.1",
  "description": "Starter project for Angular CLI's custom builders.",
  "main": "src/index.js",
  "scripts": {
    "build": "tsc"
  },
  "builders": "src/builders.json",
  "repository": {
    "type": "git",
    "url": "git+https://github.com/flakolefluk/angular-builder-starter.git"
  },
  "keywords": ["angular", "cli", "builder"],
  "author": {
    "name": "Ignacio Falk",
    "email": "flakolefluk@gmail.com"
  },
  "license": "MIT",
  "bugs": {
    "url": "https://github.com/flakolefluk/angular-builder-starter/issues"
  },
  "homepage": "https://github.com/flakolefluk/angular-builder-starter/#readme",
  "devDependencies": {
    "@angular-devkit/architect": "^0.803.0",
    "@angular-devkit/core": "^8.3.0",
    "@types/node": "^12.6.9",
    "prettier": "1.18.2",
    "typescript": "^3.5.3"
  }
}

build/schema.json

{
  "$schema": "http://json-schema.org/schema",
  "title": "Custom builder schema",
  "description": "Custom builder description",
  "type": "object",
  "properties": {
    "log": {
      "type": "boolean",
      "description": "If true, log messages",
      "default": true
    }
  },
  "additionalProperties": false
}

In this starter project, there's a boolean log option. This json file can be used with an interface to have the right typings.

build/schema.ts

export interface Schema {
  log: boolean;
}

Finally, the builder implementation. build/index.ts

import {
  BuilderOutput,
  createBuilder,
  BuilderContext
} from "@angular-devkit/architect";
import { JsonObject } from "@angular-devkit/core";
import { Schema } from "./schema";

async function _build(
  options: JsonObject & Schema,
  context: BuilderContext
): Promise<BuilderOutput> {
  if (options.log) {
    context.logger.info("Building...");
  }

  return { success: true };
}

export default createBuilder(_build);

A builder is a handler function with two arguments:

• options: a JSON object provided by the user
• context: A BuilderContext object that provides access to the scheduling method scheduleTarget and the logger among other things.

The builder can return either a Promise or an Observable.

Let's modify our project to fit our needs. We will start with a simple builder, and will start improving it step by step.

When we build our project, we do not need to watch for file changes. It's a one-time process. It has a start and an end. Our build chain will look something like this.

• Convert markdown into html
• Execute the regular build process
• Clear all generated html files

Also, we want the custom builder to work along other builders (the default Angular builders, or other custom builders).

I will use a couple of packages for traversing/watching the project directory, and converting the markdown files into html.

npm i --save marked chokidar @types/marked

Let's take a look at our implementation.

import {
  BuilderOutput,
  createBuilder,
  BuilderContext
} from "@angular-devkit/architect";
import { JsonObject } from "@angular-devkit/core";
import { Schema } from "./schema";
import * as chokidar from "chokidar";
import * as marked from "marked";
import * as path from "path";
import * as fs from "fs";

function readFiles(watcher: chokidar.FSWatcher) {
  return new Promise((resolve, reject) => {
    watcher.on("ready", () => resolve(null));
    watcher.on("error", error => reject(error));
  }).then(_ => watcher.getWatched());
}

function clearFiles(filesToDelete: string[]) {
  filesToDelete.forEach(file => {
    try {
      fs.unlinkSync(file);
    } catch (e) {
      // do nothing
    }
    return null;
  });
}

function convertFile(path: string): string {
  const content = fs.readFileSync(path, { encoding: "utf-8" });
  const html = marked(content).replace(/^\t{3}/gm, "");
  const index = path.lastIndexOf(".");
  const htmlFileName = path.substring(0, index) + ".html";
  fs.writeFileSync(htmlFileName, html);
  return htmlFileName;
}

async function _build(
  options: JsonObject & Schema,
  context: BuilderContext
): Promise<BuilderOutput> {
  if (options.log) {
    context.logger.info("Building...");
  }
  const root = context.workspaceRoot;

  // setup marked
  marked.setOptions({ headerIds: false });

  // start "watching" files.
  const watcher = chokidar.watch(path.join(root, "src", "**", "*.md"));

  // get all markdown files
  const filesMap = await readFiles(watcher);

  // stop watching files
  await watcher.close();

  // convert to array of paths
  const paths = Object.keys(filesMap).reduce((arr, key) => {
    filesMap[key].forEach(file => { if(file.toLowerCase().endsWith('.md')) {
  arr.push(path.join(key, file));
}});
    return arr;
  }, [] as string[]);

  // convert files and return html paths
  let pathsToDelete: string[] = [];
  paths.forEach(path => {
    const toDelete = convertFile(path);
    pathsToDelete.push(toDelete);
  });

  // schedule new target
  const target = await context.scheduleTarget({
    target: "build",
    project: context.target !== undefined ? context.target.project : ""
  });

  // return result (Promise) and clear files if it fails or succeeds
  return target.result.finally(() => clearFiles(pathsToDelete));
}

export default createBuilder(_build);

Let's go step by step. We will start by setting up marked. Then, we start watching our project source directory and subdirectories for markdown files. When the ready event emits, we will return all the watched files. Then, we will proceed to convert all the files, and will keep track of the html files paths. Then, we schedule a target. Targets are set on the angular.json file. In this initial example, we will schedule the build target, and will return its result. After this, the target fails or succeds, and the files will be cleared.

Let's build our custom builder, and link it to test it locally:

npm run build
npm link

It's time to create a project, and test our builder!

ng new builders-example
cd builders-example
npm link @flakolefluk/md-builder // the name of the builder package

Now that our project is set up, and our dependencies are installed, we should:

• remove app.component.html
• create app.component.md

My markdown file looks like this:

# MD BUILDER

## this is a test

{{title}} works!

Before we run our builder, we must set it in the project's angular.json file.

{
  "$schema": "./node_modules/@angular/cli/lib/config/schema.json",
  "version": 1,
  "newProjectRoot": "projects",
  "projects": {
    "builders-example": {
      "projectType": "application",
      "schematics": {},
      "root": "",
      "sourceRoot": "src",
      "prefix": "app",
      "architect": {
        "md-build": {
          "builder": "@flakolefluk/md-builder:build"
        },
        "build": {
          // ...
        }
      }
    }
  }
}

I created the md-build target. The builder key sets the target: the build builder in the @flakolefluk/md-builder package. Next to it, we have the build target (remember that our builder will schedule it).

To run a target different than the regular ones (build test, e2e, etc), you must call ng run <project>:<target>. In this example, it would be ng run builders-example:md-build.

Let's try it.

Our builder runs the way we expect it to run. Converts the markdown files, builds the project, and removes the generated files.

What if we wanted to schedule another target other than build? What if we wanted to run our command simply as ng build?

Let's add some configuration options to our builder.

build/schema.json

{
  "$schema": "http://json-schema.org/schema",
  "title": "Custom builder schema",
  "description": "Custom builder description",
  "type": "object",
  "properties": {
    "log": {
      "type": "boolean",
      "description": "If true, log messages",
      "default": true
    },
    "target": {
      "type": "string",
      "description": "target to be scheduled after converting markdown"
    }
  },
  "required": ["target"],
  "additionalProperties": false
}

build/schema.ts

export interface Schema {
  log: boolean;
  target: string;
}

build.index.ts

// ...
const target = await context.scheduleTarget({
  target: options.target,
  project: context.target !== undefined ? context.target.project : ""
});
// ...

Don't forget to run npm run build before testing again.

If we try to run our app project with the same command, we will get an error. We need to provide the required option target. We will set this in our angular.json file.

{
  "$schema": "./node_modules/@angular/cli/lib/config/schema.json",
  "version": 1,
  "newProjectRoot": "projects",
  "projects": {
    "builders-example": {
      "projectType": "application",
      "schematics": {},
      "root": "",
      "sourceRoot": "src",
      "prefix": "app",
      "architect": {
        "md-build": {
          "builder": "@flakolefluk/md-builder:build",
          "options": {
            "target": "build"
          }
        },
        "build": {}
      }
    }
  }
}

Now we can run our application using the ng run builders-example:md-build command. Let's make one more change to make the builder easier to use.

{
  "$schema": "./node_modules/@angular/cli/lib/config/schema.json",
  "version": 1,
  "newProjectRoot": "projects",
  "projects": {
    "builders-example": {
      "projectType": "application",
      "schematics": {},
      "root": "",
      "sourceRoot": "src",
      "prefix": "app",
      "architect": {
        "build": {
          "builder": "@flakolefluk/md-builder:build",
          "options": {
            "target": "ng-build"
          }
        },
        "ng-build": {}
      }
    }
  }
}

We changed the target names (remember we can pass any target name to our builder) and now we are able to run this process just by calling ng build.

Our build is working as expected. But our current setup will not work if we want to serve our application during development. We could start a different builder to serve our app, but I'll try to modify this one in a way that can handle both cases (watch mode and a single run)

We'll start by changing how we handle the scheduled target. Initially, we were returning the result property. This property returns the next output from a builder, and it works for single run tasks. If we want to track every output of a builder, then we'll use the output property, which will return an Observable of BuilderOutput.

build/index.ts

// ...
async function setup(
  options: JsonObject & Schema,
  context: BuilderContext
): Promise<{ target: BuilderRun; pathsToDelete: string[] }> {
  const root = context.workspaceRoot;
  marked.setOptions({ headerIds: false });

  const watcher = chokidar.watch(path.join(root, "src", "**", "*.md"));

  const filesMap = await readFiles(watcher);

  await watcher.close();
  const paths = Object.keys(filesMap).reduce((arr, key) => {
    filesMap[key].forEach(file => { if(file.toLowerCase().endsWith('.md')) {
  arr.push(path.join(key, file));
}});
    return arr;
  }, [] as string[]);

  let pathsToDelete: string[] = [];

  paths.forEach(path => {
    const toDelete = convertFile(path);
    pathsToDelete.push(toDelete);
  });
  context.logger.info("files converted");

  const target = await context.scheduleTarget({
    target: options.target,
    project: context.target !== undefined ? context.target.project : ""
  });

  return { target, pathsToDelete };
}

function _build(
  options: JsonObject & Schema,
  context: BuilderContext
): Observable<BuilderOutput> {
  if (options.log) {
    context.logger.info("Building...");
  }

  return from(setup(options, context)).pipe(
    mergeMap(({ target, pathsToDelete }) =>
      target.output.pipe(
        finalize(() => {
          clearFiles(pathsToDelete);
        })
      )
    )
  );
}

export default createBuilder(_build);

We refactor the setup part of our _build method into its own method that returns a Promise. Then, we create an Observable stream from that promise, and return a new Observable that will clear the genreated files once it completes.

Let's build our custom builder, and run the build process in our demo-app. Everything should work the same as before. Let's configure our app to do the same when serving it.

angular.json

{
  "$schema": "./node_modules/@angular/cli/lib/config/schema.json",
  "version": 1,
  "newProjectRoot": "projects",
  "projects": {
    "builders-example": {
      "architect": {
        "build": {},
        "ng-build": {},
        "serve": {
          "builder": "@flakolefluk/md-builder:build",
          "options": {
            "target": "ng-serve"
          }
        },
        "ng-serve": {
          "builder": "@angular-devkit/build-angular:dev-server",
          "options": {
            "browserTarget": "builders-example:ng-build"
          },
          "configurations": {
            "production": {
              "browserTarget": "builders-example:ng-build:production"
            }
          }
        }
      }
    }
  }
}

I renamed the serve target to ng-serve, and added it to the custom builder.

ng serve

Our project works as expected. If we modify any file, it will refresh. However, there are two major issues. If we modify a markdown file, it won't regenerate the html file, and when we kill our process (Ctrl+C), the generated files are not removed.

We need to reconsider how to structure our build/serve process. After a first read of the .md files, we must keep watching for changes (added, changed or removed), and schedule our target. To address the issue when the task is killed, we must listen to the SIGNINT event in our process, then proceed to stop watching the markdown files, and remove the generated files. Finally, exit the process without errors.

import {
  BuilderOutput,
  createBuilder,
  BuilderContext,
  BuilderRun
} from "@angular-devkit/architect";
import { JsonObject } from "@angular-devkit/core";
import { Schema } from "./schema";
import * as chokidar from "chokidar";
import * as marked from "marked";
import * as path from "path";
import * as fs from "fs";
import { Observable, from, fromEvent } from "rxjs";
import { finalize, mergeMap, first, tap } from "rxjs/operators";

function clearFiles(filesToDelete: string[]) {
  filesToDelete.forEach(file => {
    try {
      fs.unlinkSync(file);
    } catch (e) {
      // do nothing
    }
    return null;
  });
}

function toHtmlPath(path: string): string {
  const index = path.lastIndexOf(".");
  const htmlFileName = path.substring(0, index) + ".html";
  return htmlFileName;
}

function convertFile(path: string): string {
  const content = fs.readFileSync(path, { encoding: "utf-8" });
  const html = marked(content).replace(/^\t{3}/gm, "");
  const htmlFileName = toHtmlPath(path);
  fs.writeFileSync(htmlFileName, html);
  return htmlFileName;
}

function removeFile(path: string): string {
  const htmlFileName = toHtmlPath(path);
  fs.unlinkSync(htmlFileName);
  return htmlFileName;
}

function _setup(
  options: JsonObject & Schema,
  context: BuilderContext
): Promise<BuilderRun> {
  return context.scheduleTarget({
    target: options.target,
    project: context.target !== undefined ? context.target.project : ""
  });
}

function _build(
  options: JsonObject & Schema,
  context: BuilderContext
): Observable<BuilderOutput> {
  // setup marked
  marked.setOptions({ headerIds: false });

  // setup markdown watcher and keep track of generated files
  const root = context.workspaceRoot;
  const watcher = chokidar.watch(path.join(root, "src", "**", "*.md"));
  let pathsToDelete: string[] = [];

  // add, update or remove html files on events.
  watcher
    .on("add", (path: string) => {
      const htmlFile = convertFile(path);
      if (options.log) {
        context.logger.info(`${htmlFile} added`);
      }
      pathsToDelete.push(htmlFile);
    })
    .on("change", (path: string) => {
      const htmlFile = convertFile(path);
      if (options.log) {
        context.logger.info(`${htmlFile} changed`);
      }
    })
    .on("unlink", (path: string) => {
      const htmlFile = removeFile(path);
      if (options.log) {
        context.logger.info(`${htmlFile} removed`);
      }
      pathsToDelete = pathsToDelete.filter(path => path !== htmlFile);
    });

  // when the task is killed, stop wtahcing files, and remove generated files
  process.on("SIGINT", () => {
    clearFiles(pathsToDelete);
    watcher.close();
    process.exit(0);
  });

  // wait for the watcher to be ready (after all files have been localized), then schedule the next target, and return its output. If the output completes (for example "ng build"), remove files, and stop watching markdown changes
  return fromEvent(watcher, "ready").pipe(
    tap(() => {
      context.logger.info("Markdown ready...");
    }),
    first(),
    mergeMap(_ => from(_setup(options, context))),
    mergeMap(target =>
      target.output.pipe(
        finalize(() => {
          clearFiles(pathsToDelete);
          watcher.close();
        })
      )
    )
  );
}

export default createBuilder(_build);

Finally, we need to set up our angular.json to run any other CLI command using the custom builder.

Final words

• Feel free to contribute to this project. There's a lot of room for improvement. (Language service does not work on markdown files) :(
• The code for the builder is located in this repository
• The sample app is located here
• The Angular custom builder starter project can be found in here