strugee.net

So I promised a (long) while ago that I'd blog about all the stuff going on in pump.io. And there is a lot going on. Where to even begin?

LFNW talk

I think the first thing I should mention is the talk I gave at LinuxFest Northwest this year. It went really, really well (even though I finished half the slides the night before), and people seemed to be really engaged, especially during questions. It starts off by covering the history behind pump.io: that includes the relevant protocols, like OStatus and ActivityStreams, but also the motivation behind abandoning StatusNet (aka current GNU Social) in favor of a brand-new network. Next I move on to the way that pump.io itself works, namely, its function as a generic ActivityStreams distribution engine. To put this another way, I explain why this quote from the README:

I post something and my followers see it. That's the rough idea behind the pump.

is a pretty accurate way of describing what pump.io actually does. (I quote that quite a few times in the slides themselves.) I end with a discussion of the recent developments in the community, which are of course wonderful, and a short call-to-action for people to contribute - either directly to the software, or by joining the network and spreading pump.io.

Oh, and by the way - the talk was recorded! So you can watch it on YouTube or, even better, on my personal MediaGoblin. Fitting, since (as I mention in the talk) MediaGoblin federation will soon be released, and it's based on (and fully interoperable with) the pump.io protocol!

Conservancy application

Pump.io is applying to the Software Freedom Conservancy! This is super fantastic for us for a number of reasons. One of the biggest advantages is the fact that inside Conservancy (assuming our application is accepted), we'll be able to take donations much, much easier. This is primarily important because nodes in the pump.io network are volunteer-run, but they still require funds to operate. We're thinking about models wherein people will be able to donate to "pump.io", and then some (most?) of those funds will be used to pay for the network. (In particular, they'll be used to pay for the existing E14N nodes that Evan currently runs, which will become extremely useful once we transition those nodes to community owners). Conservancy also provides useful miscellaneous services, like owning our logo and making sure that if we encounter license violations, the license is properly enforced. But perhaps most importantly, becoming part of Conservancy cements pump.io even more in the community - both the group of people working on the software & network, but also the larger free software world.

ReadTheDocs

We've started a dedicated space for project documentation, hosted on ReadTheDocs. We're plannning to move a bunch of content from the GitHub wiki into this project, and hopefully it'll become a thorough and central place for pump.io documentation - both for users and for deployers.

Triage

We've spent quite a bit of time going through open issues and prioritizing them. A lot of issues have a release target now, and it feels really nice to feel like our issue tracker is a bit more organized.

Special thanks to Laura Arjona for driving this work.

Various minor code improvements

There have been a bunch of small bugfixes and improvements that have gone into the master branch - some of them user-facing, and some of them making the development experience better. Notable changes include:

• Migrating from Connect 1.x to Connect 2.x (this is just a start - Connect 2.x is still deprecated, but it gets us closer than we were to relying on a fully non-deprecated stack)
• JSHint is now automatically run against bin/ and routes/ when npm test is run. This makes it super obvious when there are regressions in code quality, especially in Pull Requests (since Travis CI will fail if JSHint doesn't succeed.)
• JSCS is now used to enforce code style. It's automatically run against the entire codebase (whoo!) when you run npm test, and it's awesome for the same reason - much of the style-related feedback that would've previously ended up in a Pull Request can now be dealt with directly on a local development machine, reducing PR review time for both the reviewer and the contributor.
• LibreJS is now supported
• Tests now pass! Whooooooooo! (Thanks to Menno Vossen for sending the enormous Pull Request that made this happen.)

Those are just the bigger ones, of course - there are a bunch of even smaller problems that got squashed as well. I'd also like to point out that quite a few of these were long-standing PRs which finally made it into core, which is awesome for everyone.

Express 3.x migration

I've been putting in a lot of work to migrate pump.io to Express 3.x. It's a huge amount of work, but when complete, it will bring us very, very close to being able to migrate onto Express 4.x, which is modern and fully-supported by upstream. Basically what I've been doing is just running the app, seeing where it crashes, going to the exception site, and fixing the problem. Rinse, repeat. You can check out this work on the express-3.x branch - currently, this branch can successfully start up the app, but will crash pretty soon after you try to do almost anything else.

This work, unfortunately, is on hold while another important project is completed: converting all the templates from utml to Jade.

utml to Jade transition

This is basically what it sounds like. Previously, the templates in pump.io were based on utml, which is essentially a thin wrapper around Underscore.js's _.template() function. However, utml doesn't work with Express 3.x (and it's not really worth making it work), plus it's not the prettiest to work with. Jade is an extremely popular templating language in Node-land nowadays, so a couple months ago I spent somewhere between 14 and 18 hours going through and rewriting all the utml in Jade, which was absolutely brutal - but necessary. Then, of course, I had to fix the client-side templating logic to handle Jade instead of Underscore templates, which took quite a while, along with the fact that I made a very large number of minor (largely cosmetic) errors in my conversions.

As I said above, this was kind of awful work (especially the beginning), but it's necessary and great, as it paves the way for Express 3.x and massively improves the contributor experience.

You can check out the gory details of this work in PR #1170, and the original reasoning behind why we're doing this in issue #1167. This work is actually done, but I'm going to write a separate blog post about it, calling for testers.

Upcoming 1.0.0 release

Last but certainly not least, we're gearing up for our 1.0.0 release! From a codebase standpoint, this is really just a small bugfix release (although it will make a lot of things less broken and - if I recall correctly - fix the actual installation process), but more importantly, it means that we're now committing to semantic versioning, which is a win for everyone (but especially administrators). The main thing that needs to be fixed before this goes out the door is the behavior of the XSS scrubber, which was accidentally made a little too aggressive. This is being tracked in issue #1169.

As a bonus, I'm also designing a t-shirt that (if there's sufficient interest) we may print as a celebration of this release - but more on this in my next post.

I realized I forgot to blog about all the work I've done on the strugee.net build code recently! I want to get back to working on Stratic part two (aka pretty indexes), but I thought I'd really quickly take the time to point out what's been done.

First of all, even though I rewrote the code in Jade about a year and a half ago, there was still a huge amount of boilerplate code in each Jade file. Why? Because essentially the way I did that was by running each HTML file through an HTML to Jade converter, then manually fixing up the result to be, you know, less ugly. That meant that common stuff, like the navbar, footer, includes, etc. were all duplicated across all Jade files. And that sucks, because Don't Repeat Yourself is a thing.

Luckily, refactoring is a thing too! So a few weeks ago, I went and split out the common site layout into its own Jade file, then made all the individual pages inherit from that common layout. The most difficult part of this was undoubtedly the navigation bar - I was pretty sure I'd have to just deal with the fact that the navbar was duplicated across pages (because each page has to have a different link deactivated), but then inspiration struck, and I ended up witht the solution I have now. Instead of treating the entire navigation as one block, I treat each link as a block that is overridable via Jade's block directive. And, since Jade blocks can have defaults, each page only overrides exactly what it needs to, and the rest of the links are automatically there and properly clickable.

This has a couple nice side effects. The first is that the codebase that generates strugee.net is now extremely easy for me to change, regardless of what part of the site I touch (whereas before, individual pages were pretty easy but changing e.g. the navbar require editing each and every individual page). The second (and clearly the most important) is that Night Mode, which I implemented in a 45-minute class over two years ago now (back when I was still in high school - what a crazy thing to think about not being in high school anymore!) now works on every page across all of strugee.net. Whooo! Silly hacks are at the core of the web, right? Might as well make them work well.

Finally, I'd be remiss if I didn't mention that I now have a proper deploy task, so I can abandon the sketchy and genuinely terrible deploy process that I used before. (For those curious: gulp build && git checkout master && mv dist/*.html . && git commit -m "Some bad commit message" && git push && git checkout src - not shown here are the several attempts where I made some typo or possibly didn't think of some files in dist/ that changed and so should have been mv'd, but weren't.) And, I now use root-relative includes now, e.g. /js/main.js instead of js/main.js, which is another win for DRY because now I don't have to maintain different header contents for subpages (which used to be the exact same, just with ../ prepended to all paths).

In conclusion: yay for progress! And yay for the fact that my codebase actually looks like someone competent wrote it! \o/

(Final note: I haven't forgotten my promise to blog about stuff happening in pump.io-land. But, not now.)

Whooooooooooo!

I am so, so, so thrilled to announce that the first part of Stratic is complete! And you can see the result right here on strugee.net, since this blog post was generated with Stratic!

tl;dr:

var rename = require('gulp-rename');
var markdown = require('gulp-markdown');
var parse = require('stratic-parse-header');
var straticToJson = require('stratic-post-to-json-data');
var jadeTemplate = require('gulp-jade-template');
var dateInPath = require('stratic-date-in-path');

gulp.task('posts', function() {
    return gulp.src('src/blog/*.md')
               .pipe(parse())
               .pipe(markdown())
               .pipe(dateInPath())
               .pipe(straticToJson())
               .pipe(jadeTemplate('src/blog/post.jade'))
               .pipe(rename({ extname: '.html' }))
               .pipe(gulp.dest('dist/blog'));
});

How gorgeous is that?? Let me explain how it works. (I'll assume the reader is familiar with Gulp and Node.js.)

So the gulp.src() call is pretty obvious. We just read all the blog posts into the stream. Note, however, that gulp.src() doesn't stream text, per se - it streams Vinyl file objects. This will become important later.

Now, the first piece of custom Stratic code that we use is the stratic-parse-header module. This module takes a Markdown file with a standard Stratic header (see my original announcement for details), parses the header, strips it out, then returns the new, headerless Markdown. However, the new Vinyl file object has a couple of new properties from the parsing phase - specifically, file.title, file.author, file.time, and file.categories now exist. This is why the fact that Vinyl is used is important - now any Gulp plugin downstream from where parse() is run can use all of these values in whatever way it wants. (See the README for more details.)

Now our Vinyl file object is only the content of the post, and it has additional Stratic metadata attached to it. Awesome! The next thing that we do is render the Markdown, just using a standard Gulp plugin for this. Easy breezy. After that, we pipe to the stratic-date-in-path module, which adds the year and month to paths. For example, without stratic-date-in-path, this blog post would be at https://strugee.net/blog/stratic-part-one. However, since I do use stratic-date-in-path, the post lives at https://strugee.net/blog/2016/05/stratic-part-one instead. Nice, right? Eventually I'll write code to generate pretty indexes for each year and month - that's what Stratic part 2 is for.

The next thing we do is pipe to the stratic-post-to-json-data module. This module is specifically designed to work with the gulp-jade-template module, which expects the file contents to be some JSON that will be given as data to a Jade template, whose rendered HTML becomes the new file contents. What sets up that JSON? You guessed it - stratic-post-to-json-data. That's all it does. It just creates an object that contains the metadata and the actual post text, runs it through JSON.stringify(), and sets the file contents equal to the result. Just how gulp-jade-template likes it.

And with that, we've successfully rendered a blog post. Whooooooooooo! I'm so pumped about this software. The call to rename() is just a little housekeeping, and then we write the whole thing back to disk with gulp.dest(). Awesome.

It's worth noting that the real beauty in this code isn't what the code actually does, but the extreme modularity of the whole thing. Unlike projects like Jekyll or even Wintersmith, this isn't a giant, monolithic framework. It's all standard Node and Gulp. Note how (for example) we didn't need a custom plugin for Markdown - we just used the standard gulp-markdown. Don't like Markdown? No problem. Write something to extract post metadata from your preferred format, replace parse() with that and markdown() with a different renderer, and you're golden. All the rest will continue to work the exact same - adding dates to paths, rendering the template, etc. - because everything's decoupled from everything else. Each component can be trivially swapped out and replaced with something new and better, and the rest of the system continues to work. Gorgeous.

I've got to go now, but I'm not done blogging. I'll be back soon to talk about the work going on in pump.io, and I'll be back (much?) later to talk about Stratic part two (aka, pretty indexes).

Whooooooooooooooooooooo!

The other day I described programming to someone. I pointed out that it's actually pretty easy to teach yourself programming languages, especially since after a while you start to carry over concepts from other languages. But what surprised me most about my own explanation(!) was when I compared programming to art: it's the kind of thing where you can just try stuff out and see what works and what doesn't, with no real consequences.

Since I said that, I've actually been thinking about it quite a bit. Programming is traditionally described as an activity closely related to mathematics, and to a certain extent, this makes a lot of sense, because of the logical skills that go into programming. You have to be able to reason your way through situations in order to effectively debug a program, which means logically eliminating possible points of failure. This is where math skills become very important.

But coding isn't just about logic. At OSBRIDGE this year, I attended a session about the beauty of code - it's hard to describe to someone who doesn't live and breathe code, but we all know it when we see it. We as a community value elegance in code; clever algorithms; thinking outside the box, and as I said in my Just Do It slides, the mere existance of Ruby proves this. So when I described programming as being like art, part of where I was getting that is the analogy I actually said (being able to easily mess around), but part of it was coming from my appreciation of the beauty of code. Part of it was coming from my sense of the aesthetic properties of programming.

I want us, as a community, in both our regular coding but also our educational outreach, to stop pretending that programming is so logical that it is math. Yes, there are elements of mathematics in coding. Lots of it, even. But to treat programming as a branch of mathematics is doing a disservice to the practice. So in addition to treating programming as a form of math, I want us to start treating programming as a form of art. There is such a thing as ugly code. The entire concept of refactoring would barely exist if that wasn't true. So let's start truly appreciating the aesthetic beauty of code, and let's start teaching that. To be honest, I'm not sure how you would teach that. But it couldn't hurt to try.

But even if we can and should treat it as both of those things, that doesn't mean that we should make that the be-all-end-all of how we describe programming. I truly believe that programming is not a branch of mathematics. And it's not an art form, either. Programming is neither of those things and both of those things; it is something entirely new, and we should treat it as such. If this isn't true, why do people swear by certain software? Why do people (including myself) aggressively sticker their laptops to showcase what software they use? And if this isn't true, how is it possible that people love their code?

I'm back! Sort of. Very sort of.

I've known for a while that I'm going to ditch Blogger. That's a large part of why I delayed posting stuff to my blog for so long: I didn't have blog software that I really wanted to use, but I didn't want to just put more data into Blogger. Eventually, though, I realized that it would take me a while to write software that a) built an actual blog from Markdown and b) worked the way I wanted it to, so I decided that I would just start cranking out posts and write the software to build them later. And then of course there was the week or three that I spent procrastinating on writing... oh well. I'm here now.

Anyway, it's two three AM as I write this, so I should go to sleep soon. I have a lot of stuff I want to write about, so I'll be brief.

Summer

I haven't written anything publicly since Reset the Net, way back in June. So I should probably cover some of the things I've been doing.

First, I ran a CryptoParty! It was hosted at Black Coffee's old location on Pine, and it was absolutely fantastic. We had a small group of people but it was really fun, anyway. The slides are on GitHub - and speaking of which, I've switched to Bespoke.js for all my presentations. Hell yeah HTML5!

Right after the CryptoParty was over, I actually had the opportunity to drive down to Portland for DebConf '14, which was one of the most fantastic experiences of my entire life. I met a lot of really cool people there, I got my new key (also generated over the summer) signed by a lot of Debian folks, and I played a lot of evening games (not Werewolf - the other one). One of the coolest parts was the fact that I actually got to meet two of my heros - Linus Torvalds came and did a Q&A with us (video of it is available here) and John Sullivan, Executive Director of the FSF, did a BoF-style talk on how we can get to a point where Debian is on the FSF free distribution list. I got a chance to talk to both of them afterwards, which was undoubtedly one of the coolest things that has ever happened to me thus far.

Robotics

Yes, the school year started and I'm a junior now. It's rough.

I'm on 5619 this year at robotics (Gabe, our main mentor, actually called me at DebConf to talk about it). I was a little annoyed to not be on 2856 at first, but it's worked out for the best. We're actually doing really well this year, and I'm so proud of the work we've been doing on the robot. It's difficult - it's much more complicated this year than last year, and there's basically no room on the thing. We had to move wires out of the way to make room for our scissor lift to come down properly. Speaking of which, we have a working scissor lift, which has never been accomplished at SAAS before, at least not during a season! So that's awesome.

We had a rough time our second competition, due to a lot of things - the Field Control System lag was bothering our driver, Wilson, so I wrote up something quick to fix it. I went to test it, and as soon as I ran our scissor lift up, one of the bars - which (of course) we made out of wood - snapped. So we had to rush to fix it before we had to go on the field in three matches or something. And because the bar broke, I didn't get to test the new teleop, and when Wilson tried to drive with it on the field, it broke horribly. Luckily we did pretty well in the first competition, and that gave us a nice buffer to make up for our losses the second time. I can't claim that we're doing well, but we're not doing horribly, either. Here's a video showcasing the first competition, and here's a second one of the work the club did beforehand to prepare. Please excuse the weird camera angle of me intensely working on the code because of the time crunch of FTC competition.

Patching Firefox

Over Thanksgiving break, I wrote my first Firefox patch! There was a bit of the DevTools that was bugging me, so I fixed it, in true free and open source software fasion. Unfortunately I'm having some trouble writing tests for it, and I haven't had time to track down the information that I need, so the patch hasn't made it into the tree yet. Soon, though! It's on my list of things to do during break. You can see everything over in bug 1106353.

Mail

I've spent hours of work, spread out over a number of months, working on steevie's mail subsystem. And I'm proud to say that as of a couple weeks ago, I'm finally done. I had to buy a block of static IPs for it, which I felt really cool for doing. There's still a lot of work to do - SPF, outbound DKIM signing, better TLS, Roundcube, ManageSieve, antispam, moving to LDAP from MySQL... the list goes on and on and on, but the system works. And I'm really proud of it. Anyway, I have a new email now: alex@strugee.net. I've even set up the customary names to forward to me: you can email postmaster for email trouble, webmaster for problems with the web server, and even hostmaster for general stuff. Or root, if you're APT or cron or somesuch. It'll all reach me.

Christmas

It was Christmas yesterday! Merry Christmas, Internet! I got a bunch of books, including The C Programming Language, Second Edition (yes, this is the book that K&R C is sort of named after), which was very exciting for me. I also got a budget for steevie approved, so now I can buy a bunch of hardware that I need. Hello, RAID 10 array!

Stratic

So, finally onto the juicy development part. Stratic is the name of my new pet project. Stratic is the STReaming stATIC site builder. It's like Wintersmith, except that it runs on Gulp, which I've fallen in love with over the summer. Because of that, it's a little weird - there'll be some custom components used to support it, but the main body of code is actually... a Yeoman generator.

There hasn't been much activity in the repo because I'm using the strugee.github.com repository, which still runs strugee.net even though I'm not on GitHub Pages anymore, as a testbed for Stratic. Once I've ironed out all the kinks, then I'll land all my work in the generator-stratic repository.

I'm very tired, and I want to go to bed, but before I do I figure I should explain the format of this post. Yes, this is Stratic format. It's pure Markdown, but with some additional semantics that Stratic will use to build out the blog. I figure that not many people will use <h1>s in their posts, so the Markdown equivalent (#) is used to distinguish the actual post text from what is essentially a header. I did something unusual, though, because even though the header is essentially for Stratic - who authored the post and when, what it's called, etc. - I wanted the Markdown to be at least somewhat readable in source form. Therefore, you're actually allowed to put anything you want in the header section. The values are distinguished by double quotation marks. Stratic will figure out what they mean based on their position. The first set of quotation marks contain the title, the second set contains the date, the third set contains the author, and the fourth set contains a comma-separated list of categories that the blog post should go in. The date looks a little weird - it's seconds since the epoch plus an optional UTC offset. It could be made more human-readable, but then you've got to parse stuff and it just turns into a nightmare. So I opted to sacrifice readability for elegance.

So! This is a blog post without a blog. Soon, I'll finish up Stratic and this will no longer remain solely in source form. I'm excited! I've already rewritten strugee.net in Jade, and used the opportunity to refresh the services page. Time to get crackin'.