I've been doing some research on how to set up an asset pipeline using Broccoli, which is part of the toolset provided by Ember.js. The official website shows a good example of use, but I wanted to do something a bit more advanced. Here's the result.

At the end of this text, we'll have an asset pipeline able to read these inputs:

• ES6-flavoured JavaScript modules
• JavaScript packages from NPM
• Sass files

And generate these outputs:

• A single JavaScript file, in the dialect more commonly understood by contemporary browsers
• A single CSS file

I will be using Yarn instead of NPM, because it will create fewer headaches down the road. Also, it's 2017, happy new year!

Basic setup

Broccoli works as a series of filters that can be applied to directory trees. The pipeline is defined on a file named Brocfile.js, which at its minimum expression would look something like this:

module.exports = 'src/html';

A "Brocfile" is expected to export a Broccoli "node", which is a sequence of transforms over a directory tree. The simplest possible example would be just a string representing a filesystem path, so the above does the job. We could read it as "the output of this build is a copy of the contents of the src/html directory".

Note that I say Broccoli "nodes". There's a lot of literature out there referring to Broccoli nodes as Broccoli "trees". It's the same thing, but "node" seems to be the currently accepted nomenclature, while "tree" is deprecated.

Running a build

We have a very simple Brocfile. Let's run it and see its result. We need the Broccoli CLI and libraries for this, so let's first create a Node project, then add the required dependencies:

$ yarn init -y
$ yarn add broccoli broccoli-cli

Then we add the following entry to our package.json:

"scripts": {
  "build": "rm -rf dist/ && broccoli build dist"
}

Now we can run the build process any time with this command:

$ yarn run build

When we run this command, we run a Broccoli build. Since we are not doing much at the moment, it will simply copy the contents of the src/html directory into dist. If dist exists already, our build script deletes it first, as Broccoli would refuse to write into an existing one.

Did you get an error? No problem, that's probably because you didn't have a src/html directory to read from. Create one and put some files on it. Then you'll be able to confirm that the build process is doing what it is expected to do.

Using plugins

Most transforms we can think of will be possible using Broccoli plugins. These are modules published on NPM that allow us to transpile code, generate checksums, concatenate files, and generally do all the sort of things we need to produce production-grade code.

Now, in the first example above we referred to a Broccoli node using the string src/html, meant to represent the contents of the directory of the same name. While this will work, using a string this way is now discouraged. Current advice is to instead use broccoli-source, which is the first of the plugins that we will use in this walkthrough. Let's install it:

$ yarn add broccoli-source

Now we can require it into our Brocfile and use it. I'm going to use variables in this example to start giving this pipeline some structure:

var source = require('broccoli-source');
var WatchedDir = source.WatchedDir;
var inputHtml = new WatchedDir('src/html');
var outputHtml = inputHtml;

module.exports = outputHtml;

If we run the build, we'll get exactly the same result as before. We needed more code to get the same thing, but this prepares us for things to come, and follows best practices.

The development server

In the previous example, we referred to the input HTML as a WatchedDir. This suggests that, similarly to other build tools, Broccoli includes a development server that will "watch" the input files, running a build automatically when we save any changes. Let's create a command for this on our packages.json file, adding a new entry to the scripts section:

"scripts": {
  "build": "rm -rf dist/ && broccoli build dist",
  "serve": "broccoli serve"
},

Now we can start the development server with:

$ yarn run serve

Assuming you have a file called index.html in your src/html directory, you should see it at the URL http://localhost:4200. If the file changes, you can simply refresh the page and the changes will appear without you having to explicitly run the build.

Adding a CSS pre-processor

So far this isn't very exciting. The development server is just showing copies of the HTML files in our project. Let's add a proper transform.

For this we can use a CSS pre-processor. For example, we can install the Sass plugin:

yarn add broccoli-sass

Require it at the start of our Brocfile:

var sass = require('broccoli-sass');

And add it to our pipeline on the same file:

var inputStyles = new WatchedDir('src/styles');
var outputCss = sass([inputStyles], 'index.scss', 'index.css', {});

This example will:

• read files from src/styles.
• start processing from the file index.scss, which must be in the first node given in the first argument.
• leave the result in a file called index.css in the output location.

There's a problem now. We have an HTML pipeline and a SaSS pipeline. We have to merge the two into a single result.

Merging Broccoli nodes

When you have several sources of code, to be treated in different ways, you get separate Broccoli nodes. Let's merge the ones we have into a single one. Of course for this we need a new plugin:

$ yarn add broccoli-merge-trees

Now we can perform the merge and export the result:

var MergeTrees = require('broccoli-merge-trees');

// ...process nodes...

module.exports = new MergeTrees(
  outputCss,
  outputHtml,
);

Now ensure that your HTML points to the produced CSS, which in the above example we have called index.css. Reload the develpment server and check the results.

From modern JS to one that browsers understand

All that was quite easy. Dealing with JavaScript took some more figuring out for me, but eventually I got there. Here's my take on it.

We are going to transform some ES6 files into a more browser-friendly flavour of JavaScript. For this, we need Babel, and there's a Broccoli plugin that provides it for us. We start by installing the appropriate package, as well as as a Babel plugin that provides the transform we need:

$ yarn add broccoli-babel-transpiler babel-preset-env

And now we alter our Brocfile.js to look like this:

 var babelTranspiler = require('broccoli-babel-transpiler');
 
 // ...etc...
 
 var BABEL_OPTIONS = {
   presets: [
     ['env', {
       targets: {
         browsers: ['last 2 versions'],
      },
    }],
  ],
};

var inputJs = new WatchedDir('src/js');
var outputJs = babelTranspiler(inputJs, BABEL_OPTIONS);

// ...etc...

module.exports = new MergeTrees(
  outputCss,
  outputHtml,
  outputJs,
);

The BABEL_OPTIONS argument can be used to tell Babel what platforms its output should target. In this case, we specify that we want code compatible with the last 2 versions of current browsers. You can find the list of supported browsers at https://github.com/ai/browserslist#browsers.

Write some JavaScript that uses modern features of the language, and put it in src/js, then check the results. Remember to restart the dev server and reference the JS files from your HTML. The output will consist of files of the same name as those in the input, but converted to JavaScript compatible with current browsers.

Local JavaScript modules

The one thing Babel is not doing there is handling module imports. If your project is split into several modules, and you use import in them, these lines will have been transpiled into require lines but these won't actually work on a browser. Browsers can't handle JavaScript modules natively, so we will need a new step that will concatenate all files into a single one, while respecting these module dependencies.

I have figured out a couple of ways of doing this, so I'll explain the one I like best. First we are goint to need a new Broccoli plugin:

$ yarn add broccoli-watchify

Watchify is a wrapper around Browserify. In turn, Browserify reads JavaScript inputs, parses them, finds any require calls, and concatenates all dependencies into larger files as necessary.

Let's update the lines of our Brocfile that dealt with JS to look as follows:

var babelTranspiler = require('broccoli-babel-transpiler');
var watchify        = require('broccoli-watchify');

// ...

var inputJs = new WatchedDir('src/js');
var transpiledJs = babelTranspiler(inputJs, BABEL_OPTIONS);
var outputJs = watchify(transpiledJs);

The watchify transform assumes that you will have a file index.js that is the entry point of your JavaScript code. This will be its starting point when figuring out all dependencies across modules. The final product, a single JavaScript file with all required dependencies concatenated, will be produced with the name browserify.js.

Note that imports are expected to use relative paths by default. This is, the following won't work as it uses an absolute path:

import utils from 'utils';

But this will (assuming the module utils lives in the same directory as the one doing the import):

import utils from './utils';

That is the default behaviour. If you use different settings, you can pass some options in. For example, say that you want Browserify to:

• Use a file called app.js as entry point
• Put the results in a file called index.js

To achieve this, you invoke it with these options:

var outputJs = watchify(transpiledJs, {
  browserify: {
    entries: ['app.js']
  },
  outputFile: 'index.js',
});

Using modules from NPM

The best thing about Browserify though, is that it can pull NPM modules into your project. For example, say you want to use jQuery. First you have to fetch it from NPM:

$ yarn add jquery

Then you would import it in a module in your own code:

import $ from 'jquery';

// ...

And finally you tell the Watchify plugin where it can find it, passing an option pointing to your local node_modules as a valid place to pull modules from:

var outputJs = watchify(transpiledTree, {
  browserify: {
    entries: ['index.js'],
    paths: [__dirname + '/node_modules'],
  },
});

In this example, jQuery will be pulled into the final file, where your code can use it freely.

A complete example

I have a GitHub repo that I'm using to experiment with build tools. At the time of writing, there are two working Broccoli examples that you can check out. I may add others in the future, as well as examples with other tools.

Check it out at pablobm/build-tools-research. I hope you find it useful.