strugee.net

So a couple days ago I published generator-stratic@1.0.0-beta.1 to npm. Since Stratic is now officially in beta, I thought I'd write up a guide to converting a regular, static site to a Stratic-powered blog.

Each step in this blog post (part 1 of 2[?]) will take you closer to having a fully-functional blog, but because of Stratic's decoupled design, you can work through them at your own pace. Each step will leave you with a functional environment (i.e. nothing will be "broken" such that you can't work on your website anymore).

You can see the steps in this post in action at straticjs/static-to-stratic. Each commit corresponds to a step in this post.

Let's get started!

Initial setup

The site we'll be converting is currently pretty simple. It has an index.html and a projects.html. Each of these includes /css/main.css and /js/main.js. Also, they both have a navigation section and a footer that are duplicated across each page. Each time Alyssa P. Hacker - the website's owner - makes a change to these (for example to fix the copyright year in the footer), she has to change both HTML files. The best way for her to add a new page will be to copy an existing HTML file and then change it. This is a little unideal.

Alyssa tracks her website on GitHub (in the example repository mentioned above). Here are links for the index.html and the projects.html we'll be working with.

Here's a visual of the project layout:

% tree .
.
├── css
│   └── main.css
├── index.html
├── js
│   └── main.js
└── projects.html

2 directories, 4 files

When Alyssa needs to preview her website, she manually runs http-server ..

Since Alyssa uses GitHub she publishes her website on GitHub Pages, so her website is in the master branch of her git repository. (Here we're assuming that the repository is called aphacker.github.io or something, instead of static-to-stratic.)

In addition to adding blog support, we'll improve Alyssa's website by reducing duplication while still allowing her to publish to GitHub Pages.

Step 1 - adding gulp

Before we do anything else, we need to add a build system. Stratic is designed to work with [gulpjs][], so that's the one we'll be using.

Adding gulp is super easy. First, we need to create a package.json, so we do npm init:

% npm init
This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sensible defaults.

See `npm help json` for definitive documentation on these fields
and exactly what they do.

Use `npm install <pkg> --save` afterwards to install a package and
save it as a dependency in the package.json file.

Press ^C at any time to quit.
name: (static-to-stratic)
version: (1.0.0)
description: Personal website of Alyssa P. Hacker
entry point: (index.js)
test command:
git repository: (https://github.com/straticjs/static-to-stratic.git)
keywords:
author: Alyssa P. Hacker <alyssaphacker@example.net>
license: (ISC) AGPL-3.0+
About to write to /Users/alex/Development/static-to-stratic/package.json:

{
  "name": "static-to-stratic",
  "version": "1.0.0",
  "description": "Personal website of Alyssa P. Hacker",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/straticjs/static-to-stratic.git"
  },
  "author": "Alyssa P. Hacker <alyssaphacker@example.net>",
  "license": "AGPL-3.0+",
  "bugs": {
    "url": "https://github.com/straticjs/static-to-stratic/issues"
  },
  "homepage": "https://github.com/straticjs/static-to-stratic#readme"
}


Is this ok? (yes) yes

A couple things to note here: in general, the defaults are fine to accept. We've provided a description and an author, but these are optional since this isn't actually going to be published on the npm registry. They're just kind of nice to have.

The same goes for the license, which in this case is the Affero GPL 3.0 or above - however, as the copyright holder you are of course free to choose whatever license you want. (Or no license, although I'd discourage you from doing that.)

Once we have a package.json, we can go ahead and install gulp and another module we'll need, ecstatic:

% npm install --save-dev gulp ecstatic

If you haven't used gulp previously, you'll also need to install gulp-cli:

% npm install -g gulp-cli

At this point, we'll need to move some files around. Now that we have a build system, we can organize our repository however we want instead of putting stuff exactly where we want it in production.

You can do this however you want. The organization that you'll find most projects using, though, is to put stuff in a src directory. Let's make that right now.

% mkdir src
% git mv *.html src
% git mv css src/styles
% git mv js src/scripts

Finally, create a file named gulpfile.js and put the following in it:

var gulp = require('gulp'),
    http = require('http'),
    ecstatic = require('ecstatic');

gulp.task('build:html', function() {
    gulp.src('src/*.html')
        .pipe(gulp.dest('dist'));
});

gulp.task('build:css', function() {
    gulp.src('src/styles/*')
        .pipe(gulp.dest('dist/css'));
});

gulp.task('build:js', function() {
    gulp.src('src/scripts/*')
        .pipe(gulp.dest('dist/js'));
});

gulp.task('watch', ['build'], function() {
    gulp.watch('src/*.html', ['build:html']);
    gulp.watch('src/styles/*', ['build:css']);
    gulp.watch('src/scripts/*', ['build:js']);
});

gulp.task('serve', ['watch'], function() {
        http.createServer(
                ecstatic({ root: __dirname + '/dist' })
        ).listen(8080);
});

gulp.task('build', ['build:html', 'build:css', 'build:js']);

gulp.task('default', ['serve']);

This gives us a pretty good starting point. This gulpfile defines a couple tasks that simply copy source files into dist. The watch task watches for changes and rebuilds when they occur, and the serve task starts up a server, replacing Alyssa's usage of http-server. This provides exactly the same workflow as before: Alyssa runs one command and then she can look at her site on localhost:8080. You can use different task names if you want (for example, html instead of build:html, etc.), but these are what generator-stratic gives you.

However, there's one problem: Alyssa can't deploy her site anymore. If she pushed like this, visitors would have to visit e.g. https://aphacker.github.io/src/projects instead of https://aphacker.github.io/projects! That's no good.

In order to rectify this, we'll create a new git branch, src. src will contain the source files, and we'll put the final, built site in master, which is what's served by GitHub Pages. So:

% git checkout -b src
% git push --set-upstream origin src

Great. Now, we need to add something to put the built files (i.e. the contents of dist) in master. We'll use the gh-pages module for this. First install it and a dependency we'll need:

% npm install --save-dev gh-pages gulp-util

Next, make it available in the gulpfile by adding a line at the end of require() statements:

var gulp = require('gulp'),
    http = require('http'),
    ecstatic = require('ecstatic');

And finally, add a deploy task somewhere in the gulpfile:

gulp.task('deploy', ['build'], function(done) {
    ghpages.publish(path.join(__dirname, 'dist'), { logger: gutil.log, branch: 'master' }, done);
});

Now whenever Alyssa wants to deploy a new version of her website, she just runs gulp deploy and it'll be taken care of for her. (ProTip™: change the default branch to src on GitHub. That way visitors and new clones see the source files, not the build files generated by a program.)

The very last thing we need to do is add a .gitignore file since we're installing Node modules and have a build directory now. We'll just use GitHub's, adding a line for dist/ at the end:

% curl https://raw.githubusercontent.com/github/gitignore/master/Node.gitignore > .gitignore
% echo "\ndist/\n" >> .gitignore

Now we've got a functionally equivalent development setup based on gulp. Success!

Step 2: converting HTML to Pug

The next step is to convert the HTML to Pug. Pug (formerly known as Jade) is a language that compiles to HTML. It lets you do useful things like inherit from a common layout as well as looping over JavaScript variables. If you're not familiar with Pug, you should go take a look at its syntax now.

The easiest way to do this conversion is to get a program to do it for you. Here's the one I used way back when; you may be able to find a better one. The generated Pug will be valid but not the prettiest - you may want to wait to clean it up since we're going to do some work to reduce the duplication soon.

Once you've got the converted Pug, you should rename the relevant HTML file to have a .pug extention, then replace the contents with the Pug. Do this for each HTML file.

The last step here is to make gulp build the Pug. Install gulp-pug:

% npm install --save-dev gulp-pug

Then add pug = require('gulp-pug') to the end of the var declaration at the top of your gulpfile. Finally, change your html task to look like this:

gulp.task('build:html', function() {
    gulp.src('src/*.pug')
        .pipe(pug({pretty: true}))
        .pipe(gulp.dest('dist'));
});

We'll also need to fix the watch task so it has:

gulp.watch('src/*.pug', ['build:html']);

which will watch Pug files instead of HTML files.

That's it! Alyssa's site is now building with Pug instead of HTML.

Step 3: splitting out the layout

Pug's looping and flow control constructs will be very useful to us later on, but we can get some immediate productivity wins by splitting out the site layout so it's not duplicated across every Pug file.

There's one tricky thing about this: the navigation is mostly the same between pages, but not quite - the page the user is currently on shouldn't be a link. We'll solve this by using a block directive for each link. That way, we can override just what needs to be changed, while introducing no duplication.

You'll have to figure out exactly what parts of your personal layout make sense to be split out. In Alyssa's case, there are three main things that are mostly or fully duplicated across pages:

1. The navigation bar
2. The footer
3. Invisible metadata and script/style includes

These are what we'll split out. First, we'll make a copy of index.pug and put it in src/includes/layout.pug. (Again, you can organize your files however you want - but in projects generated by generator-stratic, utility Pug files go in src/includes.) Next, edit out the page-specific content and replace them with block directives. Finally, edit each navigation bar item so it has its own block directive, leaving the old code as the default for the block directive.

Here's what this looks like when we do this to Alyssa's site:

doctype html
html
  head
    meta(charset='UTF-8')
    link(href='/css/main.css', rel='stylesheet', type='text/css')
    block head
  body
    block heading
    nav
      ul
        block nav-homepage
          li
            a(href='/') Homepage
        block nav-projects
          li
            a(href='/projects') Projects

    block body

    footer
      p © Copyright 2016 Alyssa P. Hacker.
    script(src='/js/main.js', type='text/javascript')

Note how I've replaced the h1 element (the contents of which vary per-page) with block heading, I've added a block head directive so we can specify the title per-page, I've made a block for each navigation link so we can override them if we want to individually (otherwise they'll have the default of being a link), and I've added block body for the main content. I've also cleaned out a bunch of the cruft the automatic converter put in there.

Now, we can edit index.pug so that it inherits from layout.pug - we'll use the extends keyword for this. Then we just fill in the content we want using block. Here's what this looks like after we're finished with Alyssa's site:

extends includes/layout.pug

block head
  title Alyssa P. Hacker's homepage

block heading
  h1 Alyssa P. Hacker's homepage

block nav-homepage
  li Homepage

block body
  p This is the homepage of Alyssa P. Hacker. You can check out the projects I've worked on #[a(href='/projects') here].

You'll note that I've cleaned out some cruft here, too. We have one last thing to fix: if we change the layout, nothing will get rebuilt. We can fix this by changing the watch task again so that the line for watching Pug files reads:

gulp.watch(['src/*.pug', 'src/includes/*.pug'], ['build:html']);

Sweet! index.pug is way shorter than what we had before and includes just the content now. We can do the same thing to projects.pug. Then Alyssa can, for example, correct the copyright year in layout.pug - i.e., once - and that change will go into both index.html and projects.html. I've gone ahead and made the change for her.

To give a high-level overview, here's what Alyssa's site looks like now:

% tree -I node_modules .
.
├── dist
│   ├── css
│   │   └── main.css
│   ├── index.html
│   ├── js
│   │   └── main.js
│   └── projects.html
├── gulpfile.js
├── package.json
└── src
    ├── includes
    │   └── layout.pug
    ├── index.pug
    ├── projects.pug
    ├── scripts
    │   └── main.js
    └── styles
        └── main.css

7 directories, 11 files

Next time...

This post is long enough already, so I'll stop here. We've converted Alyssa's site to have a really solid base, so next time we'll build on top of this work to add superpowered blog features, powered by Stratic.

Now go apply this to your own site!

Whooooooooooo!

I am so, so, so thrilled to announce that the first part of Stratic is complete! And you can see the result right here on strugee.net, since this blog post was generated with Stratic!

tl;dr:

var rename = require('gulp-rename');
var markdown = require('gulp-markdown');
var parse = require('stratic-parse-header');
var straticToJson = require('stratic-post-to-json-data');
var jadeTemplate = require('gulp-jade-template');
var dateInPath = require('stratic-date-in-path');

gulp.task('posts', function() {
    return gulp.src('src/blog/*.md')
               .pipe(parse())
               .pipe(markdown())
               .pipe(dateInPath())
               .pipe(straticToJson())
               .pipe(jadeTemplate('src/blog/post.jade'))
               .pipe(rename({ extname: '.html' }))
               .pipe(gulp.dest('dist/blog'));
});

How gorgeous is that?? Let me explain how it works. (I'll assume the reader is familiar with Gulp and Node.js.)

So the gulp.src() call is pretty obvious. We just read all the blog posts into the stream. Note, however, that gulp.src() doesn't stream text, per se - it streams Vinyl file objects. This will become important later.

Now, the first piece of custom Stratic code that we use is the stratic-parse-header module. This module takes a Markdown file with a standard Stratic header (see my original announcement for details), parses the header, strips it out, then returns the new, headerless Markdown. However, the new Vinyl file object has a couple of new properties from the parsing phase - specifically, file.title, file.author, file.time, and file.categories now exist. This is why the fact that Vinyl is used is important - now any Gulp plugin downstream from where parse() is run can use all of these values in whatever way it wants. (See the README for more details.)

Now our Vinyl file object is only the content of the post, and it has additional Stratic metadata attached to it. Awesome! The next thing that we do is render the Markdown, just using a standard Gulp plugin for this. Easy breezy. After that, we pipe to the stratic-date-in-path module, which adds the year and month to paths. For example, without stratic-date-in-path, this blog post would be at https://strugee.net/blog/stratic-part-one. However, since I do use stratic-date-in-path, the post lives at https://strugee.net/blog/2016/05/stratic-part-one instead. Nice, right? Eventually I'll write code to generate pretty indexes for each year and month - that's what Stratic part 2 is for.

The next thing we do is pipe to the stratic-post-to-json-data module. This module is specifically designed to work with the gulp-jade-template module, which expects the file contents to be some JSON that will be given as data to a Jade template, whose rendered HTML becomes the new file contents. What sets up that JSON? You guessed it - stratic-post-to-json-data. That's all it does. It just creates an object that contains the metadata and the actual post text, runs it through JSON.stringify(), and sets the file contents equal to the result. Just how gulp-jade-template likes it.

And with that, we've successfully rendered a blog post. Whooooooooooo! I'm so pumped about this software. The call to rename() is just a little housekeeping, and then we write the whole thing back to disk with gulp.dest(). Awesome.

It's worth noting that the real beauty in this code isn't what the code actually does, but the extreme modularity of the whole thing. Unlike projects like Jekyll or even Wintersmith, this isn't a giant, monolithic framework. It's all standard Node and Gulp. Note how (for example) we didn't need a custom plugin for Markdown - we just used the standard gulp-markdown. Don't like Markdown? No problem. Write something to extract post metadata from your preferred format, replace parse() with that and markdown() with a different renderer, and you're golden. All the rest will continue to work the exact same - adding dates to paths, rendering the template, etc. - because everything's decoupled from everything else. Each component can be trivially swapped out and replaced with something new and better, and the rest of the system continues to work. Gorgeous.

I've got to go now, but I'm not done blogging. I'll be back soon to talk about the work going on in pump.io, and I'll be back (much?) later to talk about Stratic part two (aka, pretty indexes).

Whooooooooooooooooooooo!

I'm back! Sort of. Very sort of.

I've known for a while that I'm going to ditch Blogger. That's a large part of why I delayed posting stuff to my blog for so long: I didn't have blog software that I really wanted to use, but I didn't want to just put more data into Blogger. Eventually, though, I realized that it would take me a while to write software that a) built an actual blog from Markdown and b) worked the way I wanted it to, so I decided that I would just start cranking out posts and write the software to build them later. And then of course there was the week or three that I spent procrastinating on writing... oh well. I'm here now.

Anyway, it's two three AM as I write this, so I should go to sleep soon. I have a lot of stuff I want to write about, so I'll be brief.

Summer

I haven't written anything publicly since Reset the Net, way back in June. So I should probably cover some of the things I've been doing.

First, I ran a CryptoParty! It was hosted at Black Coffee's old location on Pine, and it was absolutely fantastic. We had a small group of people but it was really fun, anyway. The slides are on GitHub - and speaking of which, I've switched to Bespoke.js for all my presentations. Hell yeah HTML5!

Right after the CryptoParty was over, I actually had the opportunity to drive down to Portland for DebConf '14, which was one of the most fantastic experiences of my entire life. I met a lot of really cool people there, I got my new key (also generated over the summer) signed by a lot of Debian folks, and I played a lot of evening games (not Werewolf - the other one). One of the coolest parts was the fact that I actually got to meet two of my heros - Linus Torvalds came and did a Q&A with us (video of it is available here) and John Sullivan, Executive Director of the FSF, did a BoF-style talk on how we can get to a point where Debian is on the FSF free distribution list. I got a chance to talk to both of them afterwards, which was undoubtedly one of the coolest things that has ever happened to me thus far.

Robotics

Yes, the school year started and I'm a junior now. It's rough.

I'm on 5619 this year at robotics (Gabe, our main mentor, actually called me at DebConf to talk about it). I was a little annoyed to not be on 2856 at first, but it's worked out for the best. We're actually doing really well this year, and I'm so proud of the work we've been doing on the robot. It's difficult - it's much more complicated this year than last year, and there's basically no room on the thing. We had to move wires out of the way to make room for our scissor lift to come down properly. Speaking of which, we have a working scissor lift, which has never been accomplished at SAAS before, at least not during a season! So that's awesome.

We had a rough time our second competition, due to a lot of things - the Field Control System lag was bothering our driver, Wilson, so I wrote up something quick to fix it. I went to test it, and as soon as I ran our scissor lift up, one of the bars - which (of course) we made out of wood - snapped. So we had to rush to fix it before we had to go on the field in three matches or something. And because the bar broke, I didn't get to test the new teleop, and when Wilson tried to drive with it on the field, it broke horribly. Luckily we did pretty well in the first competition, and that gave us a nice buffer to make up for our losses the second time. I can't claim that we're doing well, but we're not doing horribly, either. Here's a video showcasing the first competition, and here's a second one of the work the club did beforehand to prepare. Please excuse the weird camera angle of me intensely working on the code because of the time crunch of FTC competition.

Patching Firefox

Over Thanksgiving break, I wrote my first Firefox patch! There was a bit of the DevTools that was bugging me, so I fixed it, in true free and open source software fasion. Unfortunately I'm having some trouble writing tests for it, and I haven't had time to track down the information that I need, so the patch hasn't made it into the tree yet. Soon, though! It's on my list of things to do during break. You can see everything over in bug 1106353.

Mail

I've spent hours of work, spread out over a number of months, working on steevie's mail subsystem. And I'm proud to say that as of a couple weeks ago, I'm finally done. I had to buy a block of static IPs for it, which I felt really cool for doing. There's still a lot of work to do - SPF, outbound DKIM signing, better TLS, Roundcube, ManageSieve, antispam, moving to LDAP from MySQL... the list goes on and on and on, but the system works. And I'm really proud of it. Anyway, I have a new email now: alex@strugee.net. I've even set up the customary names to forward to me: you can email postmaster for email trouble, webmaster for problems with the web server, and even hostmaster for general stuff. Or root, if you're APT or cron or somesuch. It'll all reach me.

Christmas

It was Christmas yesterday! Merry Christmas, Internet! I got a bunch of books, including The C Programming Language, Second Edition (yes, this is the book that K&R C is sort of named after), which was very exciting for me. I also got a budget for steevie approved, so now I can buy a bunch of hardware that I need. Hello, RAID 10 array!

Stratic

So, finally onto the juicy development part. Stratic is the name of my new pet project. Stratic is the STReaming stATIC site builder. It's like Wintersmith, except that it runs on Gulp, which I've fallen in love with over the summer. Because of that, it's a little weird - there'll be some custom components used to support it, but the main body of code is actually... a Yeoman generator.

There hasn't been much activity in the repo because I'm using the strugee.github.com repository, which still runs strugee.net even though I'm not on GitHub Pages anymore, as a testbed for Stratic. Once I've ironed out all the kinks, then I'll land all my work in the generator-stratic repository.

I'm very tired, and I want to go to bed, but before I do I figure I should explain the format of this post. Yes, this is Stratic format. It's pure Markdown, but with some additional semantics that Stratic will use to build out the blog. I figure that not many people will use <h1>s in their posts, so the Markdown equivalent (#) is used to distinguish the actual post text from what is essentially a header. I did something unusual, though, because even though the header is essentially for Stratic - who authored the post and when, what it's called, etc. - I wanted the Markdown to be at least somewhat readable in source form. Therefore, you're actually allowed to put anything you want in the header section. The values are distinguished by double quotation marks. Stratic will figure out what they mean based on their position. The first set of quotation marks contain the title, the second set contains the date, the third set contains the author, and the fourth set contains a comma-separated list of categories that the blog post should go in. The date looks a little weird - it's seconds since the epoch plus an optional UTC offset. It could be made more human-readable, but then you've got to parse stuff and it just turns into a nightmare. So I opted to sacrifice readability for elegance.

So! This is a blog post without a blog. Soon, I'll finish up Stratic and this will no longer remain solely in source form. I'm excited! I've already rewritten strugee.net in Jade, and used the opportunity to refresh the services page. Time to get crackin'.