strugee.net

Posts categorized as "development"

pump.io 5.1.1, Docker images, and sunsetting Node 4 support

It's been a (relatively) long time since we've put anything on this blog, and I think it's high time for an update - especially since there are so many exciting things afoot! Not only is pump.io 5.1.1 now on npm, but we have new experimental Docker images! With upstream having already dropped security support, we're also planning to drop support for Node 4 soon.

Let's take these one at a time.

pump.io 5.1.1

Several months ago I landed a patch from contributor Camilo QS fixing a bug in pump.io's session handling in a route serving uploads. This bug made it so that non-public uploads would always return HTTP 403 Unauthorized, even if the user actually was authorized. Clearly, this makes uploads unusable for people who don't like to post everything publicly. Evan suggested that we should backport this bugfix since it's so high-impact, and I agree. So that's what pump.io 5.1.1 contains: a bugfix for uploads. Since it's a patch release 5.1.1 is a drop-in replacement for any 5.x pump.io release, so I'd highly encourage administrators to upgrade as soon as it's convenient. We'd also love it if you file any bugs you find, and feel free to get in touch with the community if you need help or have questions. As a reminder, you can subscribe to our low-volume announce mailing list to get email when we put out new releases like this. Also, I would be remiss if I didn't mention that my signing key setup has changed temporarily - see here if you want to cryptographically verify the 5.1.1 release.

If you're on an npm-based install, you can upgrade with npm install -g pump.io@5.1.1. If you're on a source-based install, you can upgrade by integrating the latest commits in the 5.1.x branch. See here for the changelog.

But that's not all. pump.io 5.1.1 also includes another exciting change: with this release, we've integrated automation to relase pump.io Docker images too.

Docker images

We've wanted to release pump.io Docker images for a long time. But Docker has a well-known problem: security vulnerabilities in Docker Hub images are rampant. Even though we've had a Dockerfile in the repository for a while thanks to contributor thunfisch, we didn't want to release official Docker images if we weren't sure we could always provide security support for them.

Unfortunately, Docker the company has done very little to address this problem. Most of their solutions are aimed at image consumers, not authors. Docker Hub has some capacity for automatically rebuilding images, but unfortunately, it's not enough and you end up having to roll everything yourself anwyay. Pretty disappointing - so we had to get creative.

Our solution to this problem is to utilize Travis CI's cron functionality. Every day, Travis will automatically trigger jobs that do nothing but build pump.io Docker images. These images are then pushed to Docker Hub. If nothing has changed, Docker Hub recognizes that the "new" images are actually identical with what's already there, and nothing happens. But if there has been a change, like a native dependency receiving a security update, then the image ID will change and Docker Hub will accept the updated image. This cronjob is enabled for the 5.1.x branch and master (which as a side effect, means that alpha Docker images are published within 24 hours of a git push), and in the future it will be enabled on all branches that we actively support. Thus, Docker users can easily set up automation to ensure that they're running insecure images for, at most, 24 hours.

If you're interested in trying out the Docker images, we'd love to know how it goes. They should still be treated as experimental at the moment, and early feedback would be super useful. You can read more details in our ReadTheDocs documentation.

Note that there are still more changes that we'd like to make to the Docker images. These changes didn't make it into the 5.1.1 packaging since they felt too invasive for a patch release. Instead we plan to make them in the next release, which is planned to be semver-major. Which brings me neatly to the last topic...

Sunsetting Node 4, 5, and 7 support

We had a good run, but it's time to say goodbye: Node.js upstream has marked Node 4.x as end-of-life, and in accordance with our version policy, we're doing the same. Since this is a semver-major change, we're also taking the opportunity to drop support for Node 5.x and Node 7.x. These changes have been made as of commit 32ad78, and soon we'll be ripping out old code used to support these versions, as well as upgrading dependencies that have recently started requiring newer Nodes.

Anyone still on these versions is encouraged to upgrade as soon as possible, as Node.js upstream is no longer providing security support for them. Administrators can use the NodeSource packages, or they can try out our new Docker images, which use a modern Node version internally.

Please reach out to the community if you need any help making the transition. And good luck!


Winter break retrospective & spring semester goals

Tonight I'll have been back at college for a full week, and I wanted to write up a little retrospective of winter break to see what I accomplished (and in particular, which goals I completed or missed).

You may wish to skip directly to the executive summary.

Resolved goal: Node.js manpage PR

I didn't complete this goal per se, but I did at least resolve it by closing the Pull Request. I felt pretty bad about it (especially because I kept saying I intended to finish it) but honestly, it became clear to me that I'd just lost the motivation to keep going with it. I would love it if this was included in Node.js core, but I just consistently have higher priorities. So rather than leave it hanging and cluttering up the Pull Requests view, I just closed it to reflect reality. I made sure not to delete the branch though, in case someone (distant future me?) wants to pick it up again.

Failed goal: deal with GPG keysigning

I did nothing to push this goal forward. While I made numerous improvements to my GPG setup, I did not actually sign anyone's key, which was what this goal was about. This feels unfortunate. (I also do not have access to the private key material in college, and am certainly not about to ask that it be shipped to me.)

Partially completed goal: push Debian packaging work forward

There were two components to this: Profanity packaging upgrades and getting the new filter-other-days packaging into Debian. I made no progress on the Profanity packaging. However, I did fix a misconfiguration in my .reportbugrc (which annoyingly had previously sent my incredibly detailed email about Profanity packaging to /dev/null) and then submitted an ITP (Intent To Package bug, which is a Debian thing) for filter-other-days. I then used that ITP bug number to fix the last .deb Lintian warning (although see below). Then I paired with Anja who is, as always, an angel, and we figured out the weirdness that is dput and mentors.debian.net. Finally I was able to upload filter-other-days(!) BUT I was in for a rude awakening - apparently Lintian checks for .debs and .dscs are different. So while my binary package was Lintian-clean, my source package unfortunately wasn't. This is something I will need to work on in the weeks to come. That being said, I'm still pretty proud of what I've accomplished here! I've made significant progress on this front.

Completed goal: lazymention v1

One of the first things I did was ship lazymention 1.0.0 - and I wrote an announcement blog post to accompany it! (In fact, I syndicated that blog post to IndieNews using lazymention, which felt pretty damn awesome.) I got some great feedback on it from the IndieWeb community, and my lobste.rs submission - which also got some great engagement - even made the front page, which was pretty unreal! I still have a lot more work to do with lazymention - in particular, it doesn't actually respect edits (so it'll resend Webmentions with every job submission) - but for now it works well. I'm super pleased with it, and have integrated it into my site deployment workflow. I even wrote a module so other people can do this, too!

Failed goals: ActivityPub in pump.io core, pump.io PR review

No progress on this front. I did start hacking on a telemetry server which will eventually be helpful for our ActivityPub rollout, but that did not in any way directly help fulfill these goals. I also released 5.1 stable, but that's pretty routine by this point.

Partially completed goal: two blog posts per week

I stuck with this goal all the way up until the final week, where I didn't write any. (Although I wrote about my GPG keys around the time I actually flew back to college.) The first week, I wrote about my thoughts on shell script and about lazymention; the second, I wrote about the pump.io 5.1 stable release and about talking to Pull Request reviewers if you think they're wrong.

Failed stretch goal: paper editing

I did absolutely no editing on the paper I intend to get published (which I originally wrote for a writing class). This was a stretch goal though, so that's totally fine.

Additional activity: steevie maintenance

After I finally found the cable I needed, I swapped out the cable that connects steevie's motherboard with the drives' SATA ports. This seemed to significantly improve disk performance, although there are still noticeable performance problems. I'm very, very happy to have finally gotten this done.

Additional activity: Tor relay migration from Amazon EC2 to DigitalOcean

After getting some advice on tor-relays, I finally sat down and looked into moving my relay away from Amazon Web Services. This is because AWS bills by usage, which for a Tor relay ends up being incredibly inefficient. It turned out to actually be way easier than I thought, which only served to make me mad that I hadn't done it sooner. In any case, I now save approximately $240/year AND I can push 1000GB/month as opposed to the 10GB/month I pushed before. In the words of the commit where I made this change: "this change made possible by the fact that I'm no longer getting billed up the wazoo by Amazon Web Services." Here's a of a Tor Metrics graph (captured today) that shows the jump:

Tor Metrics graph

Anyway, I'm super happy I can contribute more to the Tor network and save lots of money in the process. That being said I am pretty damn salty I didn't realize this in the four years I've been running a Tor relay.

Additional activity: offandonagain.org maintenance

After turning on NodeSecurity monitoring for offandonagain.org, I found out that the module that underlies it, node-serve-random, had some vulnerable dependencies. Not only did I fix this, I wrote a large test suite for the module and found (and fixed!) several bugs in the process. Writing a test suite also allowed me to turn on Greenkeeper for the module, which will be a huge help to me in keeping the module up-to-date.

Additional activity: Stratic work

First off, I released beta 7 of generator-stratic! Nothing major, just some polishing stuff. Stratic is getting very close to the point where I might want to start promoting it more aggressively, or declaring it stable, and (with a lot of super-helpful feedback from my family) I worked on something that's super important before we get there: a logo!

Here are two of my favorites so far:

Yellow background with a centered black file icon and a asteroid coming up from earth in the midddle and a pipe to the right Yellow background with a centered black file icon and a rocket coming up from earth in the midddle and a pipe to the right

These are based off the JS logo, in case you hadn't seen it before:

Black JS text in the bottom-right corner of a yellow background

Anyway, I have to post another iteration in the GitHub issue based on some feedback from Saul (who I had a lovely lunch with) - he thinks I should reverse it so the pipe is on the left, so it looks like the file is coming out of the pipe. But anyway you should comment there if you have feedback!

Additional activity: IndieWeb stuff

I attended Homebrew Website Club in San Fransisco, which was incredible. I got to meet a bunch of new people, as well as say hi to Ryan and Tantek again, which was so nice - it's always just better to talk in real life. Tantek said (at least if I recall correctly) that my laptop was one of the best-stickered laptops he'd ever seen, which made me feel just unbelievably special. He also gave me a microformats sticker (and helped me figure out where to put it), which I had on my old laptop and had been really missing, as well as a Let's Encrypt sticker. The latter was so special I elected to put it on the inside of my laptop, which I reserve only for really special things (basically a Recurse Center refucktoring sticker and a sticker of Noah in this video, which he handed to me like a business card the first time we met). Anyway, every time I look at the Let's Encrypt sticker I just feel so happy. I love Let's Encrypt so damn much.

Homebrew Website Club was super inspiring, so when I got back to where I was staying (at my mom's boyfriend's house) I started implementing an IndieWeb-style social stream for strugee.net. It still needs some polishing but is actually pretty close to being done. Who knows when I'll have time to finish it, but it's getting there! I'm so freaking excited, honestly. Also, I added proper timestamp mf2 metadata to my site, as well as a visual display for post edits, and I expanded what type of Webmentions the display code recognizes too!

Executive summary

I resolved or completed 2 goals, partially completed 2 goals, failed 3 goals, and failed 1 stretch goal. Additionally I did significant work in 5 other areas. Out of the goals I set for myself, I completed 51% (Debian packaging work is ~2/5 complete; blog posts were written 2/3 of the time); not counting the stretch goal, I completed 61.2%. I'm pretty happy with what I got done during this period; however, while I was productive, the numbers show that I did a mediocre job sticking to my goals. In the future I should focus on making more realistic goals and then sticking to them (though not too much - it is a break, after all!).

Speaking of which, partway through break I felt like I was on the edge of burnout, which to me was a very clear sign that I was pushing myself way too hard during a time I should have been unwinding. Because of that I cut what I was doing a lot, which helped pretty dramatically. In fact, I think without that I wouldn't have been able to do some of the later stuff, like all the IndieWeb work. So that's another reason I have to find a way to balance sticking to goals and just relaxing (which doesn't necessarily mean not coding, just doing whatever coding I feel like in the moment) - I feel like I was pushing myself too hard to meet my goals (and then getting distracted and not meeting them) and that's what led to the feeling. Obviously there are different constraints for e.g. schoolwork; here I'm referring only to free time like breaks.

Spring semester goals

With that in mind, I want to set some very broad goals for spring semester. I may update this list as time goes on, but for now I have four overarching goals I want to accomplish (besides the usual day-to-day code maintenance stuff):

  • Finish editing the paper I wrote last semester on freedom-respecting software and intersectionality, and get it published
  • Make some measurable progress on my Push-based Two-factor Authentication IETF draft
  • Get access to the University of Rochester darkroom and start developing/printing photos again
  • Start pushing the University of Rochester library (and maybe the journalism department?) to start adopting Tor technologies

I'm excited to see how it goes!


pump.io 5.1 stable published to npm

Last night I officially published pump.io 5.1 to npm as a stable release!

As I wrote in the beta announcement, this release contains a variety of improvements:

  • Zero-downtime restarts, which allows administrators to seamlessly roll over to new configurations and codebases
  • The daemon now generates startup log warnings on bad configurations, including insecure secret values and internal parameters
  • An official Dockerfile is now included with the release
  • The logged-out mobile homepage's menu icon is no longer incorrectly styled as black
  • An authorization problem with SockJS connections has been fixed

5.1 stable does include one change the beta didn't: a bump to the version of the gm npm package which we depend on. This bump was done as a precautionary measure, as previous versions of gm depended on a version of the debug module which was vulnerable to denial-of-service security bugs.

As a project, we addressed these bugs back in October when we issued security releases for all supported release branches, and at the time we confirmed that the vulnerable function wasn't used by gm. Today's gm bump does not constitute a security release; instead, we're just bumping the version as a precautionary measure in case we missed something in October's assessment of the situation.

Aside from the gm bump, there are (as usual) miscellaneous version bumps included in this release. We've also started tracking test suite coverage information as well as overhauled our documentation on ReadTheDocs, moving most of the in-repository documentation there.

If you want even more details of this release, you can also check out the changelog.

pump 5.1 is a drop-in replacement for 5.0. That means if you're using our recommended installation method and installing from npm, you can upgrade with npm install -g pump.io@5.1. If you have a source-based install, you should merge and/or switch to the v5.1.0 tag. And as always, if you encounter any problems, please feel free to reach out to the community or file bugs you find.

Finally, I would be remiss if I didn't point out that pump.io has a brand-new announcement mailing list! While the blog is great for announcing new releases, not everyone finds it convenient to check. Also, if we issue new betas in the middle of a release cycle, these aren't typically announced on the blog. Therefore in the future all new releases will be announced on the mailing list, not just initial betas. If you want to subscribe to the mailing list, you may do so here - you'll get announcements of new features only, not e.g. feature announcements as seen on this blog. I hope people find this service useful!


Announcing lazymention: elegant outbound Webmention for static sites

This post also appeared on IndieNews.

Last night I hit publish on version 1.0.0 of a new project, lazymention! Whoohoo!

tl;dr: lazymention exists to add Webmention support to static sites!

To elaborate a little bit, I developed lazymention because I had a problem with this site: I wanted to send outbound Webmentions when I link to things, but my website is completely static. (Webmention, in case you didn't know, is a way to notify another website that you linked to them, so the other website can display some UI about your reply or whatever.) The page builds happen on my local machine, not on the server. One option would be to just send Webmentions from my local machine too, but this isn't really a good solution for a couple reasons. First, I couldn't do it automatically at build-time because the built pages wouldn't have been deployed to the server yet, so receivers of my Webmentions would reject the mentions due to the source being nonexistant. That meant that I would have to have a separate step, which wouldn't really be that big of a deal (lazymention requires pinging a server too) except for the second reason: I would need some way to keep track of where I'd already sent Webmentions to, and that would require synchronizing across computers. Probably the only decent way to do that would be to check it into Git, but having a program's data store checked in right next to the source code just feels kinda ugly. Plus, then it can't be shared with other people as a service.

So instead of doing it locally, I elected to build a server instead. Here's how it works: you mark up your stuff with h-feed and h-entry, and whenever anything happens (e.g. you publish a new blog post or whatever), you ping lazymention with the URL (either the feed or the post itself). lazymention will use your microformats2 markup to find the canonical location for a given post, then it will find all the links in the post and send Webmentions for them. And presto! You've just sent Webmentions for your blog. lazymention also records when it's sent mentions, so if you ping it again, nothing will happen unless you've updated your content. I'm also planning to add WebSub support to lazymention, too, and that'll work in the exact same way.

lazymention is super easy to get started with, especially because I've provided thorough documentation in the README. If you find anything that's confusing or missing, please let me know by filing an issue! I'd love to get it fixed. In fact, I'd be thrilled to hear about both positive and negative installation experiences.

Oh, and one more thing - lazymention is reusable in other applications. If you're writing a Node.js app and want to reuse its HTTP API, you can use its embedding API to get at the Express application and Router used internally. I'm not sure if people will actually find this useful, but I wrote it just for kicks anyway. See the embedding documentation for more!

Cheers, and happy mentioning! Elegant outbound Webmention for static sites is here.


Webmention test post

Once again I need to test whether lazymention - and more importantly, the underlying libraries (node-send-webmention and node-get-webmention-url) - complies with the W3C spec. I'll be using webmention.rocks, so, here's a large list of links:

Test 1, Test 2, Test 3, Test 4, Test 5, Test 6, Test 7, Test 8, Test 9, Test 10, Test 11, Test 12, Test 13, Test 14, Test 15, Test 16, Test 17, Test 18, Test 19, Test 20, Test 21, Test 22, Test 23

Hopefully it works \o/


~