Posts from 2018

Show only posts from January February April May September


Recently I turned 20 years old. As the days continue to go by, I have been struggling with the concept of time.

Probably this feeling started with me seeing photos of this year's SAAS robotics teams. I was on the robotics team for four years (I've written about it a little before) but recently I found that all of a sudden, I know barely anyone in the photos. Fifteen people in robotics graduated[1] high school last year, people I've largely known since they were freshmen (or in the case of Jesse and Matt, even longer). Some of them are very close friends. And I have found that the way I think about Redshift Robotics has changed: I don't quite think of it as something I used to be a part of, at least not that simply. Instead I think of it as my legacy.

legacy: n. something handed down from an ancestor or a predecessor or from the past

- from The American Heritage® Dictionary of the English Language, 4th Edition, via Wordnik

I don't belong there anymore. FIRST Tech Challenge has changed since I've left. They play a different game than I did, they run their GitHub differently than I envisioned when I set it up, they have different mentors and leadership than we did, and I'm sure they practice and compete differently than we did. That's okay. But it's weird. I know exactly what they're thinking because I thought it too, years and years ago - to them, I am a vague echo in someone else's past; an accumulation of events that has shaped what the club is today but that they would never be able to name. I'm someone they've vaguely heard of, maybe, but never really known. What happens next is all in their hands and there's nothing I can or want to do about it; I witnessed a former mentor of SAAS robotics turn the entire club around and that's history they'll never have seen, and maybe will never even know about. But whatever is happening there now is something I'll never see, either.

Pretty soon there will be no students left at SAAS I still know. Two more very close friends graduate this year. It'll only take a year or two more after that before the last student I know personally is gone.

There are places I remember all my life
Though some have changed
Some forever, not for better
Some have gone and some remain.

- The Beatles, In My Life

I think maybe as time goes on I'm getting better at watching people grow and change. I'm on year two of college and already I see it happening - last year a close group of friends and I were just freshmen, bright-eyed and bushy-tailed. This year one's an RA, two are First-year Fellows[2] and I'm a D'Lion[3]. The rest are living their lives in upper-class housing. As I was sitting in the dining hall earlier tonight I thought, how lucky we are that we even met last year. We're all scattered around now, generally in wildly different departments, and we're all focusing on different next steps. Most of us live in different buildings from all the others, and I can't help but wonder if we would all have met later. I think the answer is probably no. Causality is a strange thing to me; the circumstances that brought us together are dead and gone and yet the friendship lives on. Legacy.

I see the world changing too. There was a point in my life where it seemed like nothing big and cosmic would ever change; it seemed like the world would adjust itself only in small details. But at some point that stopped being the case, when I wasn't watching. James Porter, who's part of the faculty at the Recurse Center, once told me that whenever new batches came in he was always super sad to see so many people he'd made friends with leave to be replaced by strangers he didn't know yet. But, he said, he never quite noticed the point where people in-batch switched from strangers to friends. That feeling of not noticing is how I feel about the world right now. And it just boggles the mind that I'm at an age now where I can actually say yeah, I've seen those things change. I see people even just a year or two younger than me so much more attached to their phones; I say "back in the day" unironically when I talk about tech things.

The traffic in Seattle has gotten worse, almost unbearable compared to what it once was. We didn't used to hear about wildfires like we do now; the country and the planet are burning. And above all else, the country is angrier. So much angrier.

I turned 20 today. A question for people older and wiser than I: I feel incredibly weird to not be a "teenager" anymore, what the hell do I do???

- me, a couple days ago, on and on Twitter


[1]: and we all know how I feel about graduations.

[2]: upper-class student at University of Rochester who lives with first-years and helps them with academic stuff (especially course registration)

[3]: like a First-year Fellow except instead of academic stuff, they help bring their hall together and make sure everyone's having a good time and making friends, etc. 5.1.1, Docker images, and sunsetting Node 4 support

It's been a (relatively) long time since we've put anything on this blog, and I think it's high time for an update - especially since there are so many exciting things afoot! Not only is 5.1.1 now on npm, but we have new experimental Docker images! With upstream having already dropped security support, we're also planning to drop support for Node 4 soon.

Let's take these one at a time. 5.1.1

Several months ago I landed a patch from contributor Camilo QS fixing a bug in's session handling in a route serving uploads. This bug made it so that non-public uploads would always return HTTP 403 Unauthorized, even if the user actually was authorized. Clearly, this makes uploads unusable for people who don't like to post everything publicly. Evan suggested that we should backport this bugfix since it's so high-impact, and I agree. So that's what 5.1.1 contains: a bugfix for uploads. Since it's a patch release 5.1.1 is a drop-in replacement for any 5.x release, so I'd highly encourage administrators to upgrade as soon as it's convenient. We'd also love it if you file any bugs you find, and feel free to get in touch with the community if you need help or have questions. As a reminder, you can subscribe to our low-volume announce mailing list to get email when we put out new releases like this. Also, I would be remiss if I didn't mention that my signing key setup has changed temporarily - see here if you want to cryptographically verify the 5.1.1 release.

If you're on an npm-based install, you can upgrade with npm install -g If you're on a source-based install, you can upgrade by integrating the latest commits in the 5.1.x branch. See here for the changelog.

But that's not all. 5.1.1 also includes another exciting change: with this release, we've integrated automation to relase Docker images too.

Docker images

We've wanted to release Docker images for a long time. But Docker has a well-known problem: security vulnerabilities in Docker Hub images are rampant. Even though we've had a Dockerfile in the repository for a while thanks to contributor thunfisch, we didn't want to release official Docker images if we weren't sure we could always provide security support for them.

Unfortunately, Docker the company has done very little to address this problem. Most of their solutions are aimed at image consumers, not authors. Docker Hub has some capacity for automatically rebuilding images, but unfortunately, it's not enough and you end up having to roll everything yourself anwyay. Pretty disappointing - so we had to get creative.

Our solution to this problem is to utilize Travis CI's cron functionality. Every day, Travis will automatically trigger jobs that do nothing but build Docker images. These images are then pushed to Docker Hub. If nothing has changed, Docker Hub recognizes that the "new" images are actually identical with what's already there, and nothing happens. But if there has been a change, like a native dependency receiving a security update, then the image ID will change and Docker Hub will accept the updated image. This cronjob is enabled for the 5.1.x branch and master (which as a side effect, means that alpha Docker images are published within 24 hours of a git push), and in the future it will be enabled on all branches that we actively support. Thus, Docker users can easily set up automation to ensure that they're running insecure images for, at most, 24 hours.

If you're interested in trying out the Docker images, we'd love to know how it goes. They should still be treated as experimental at the moment, and early feedback would be super useful. You can read more details in our ReadTheDocs documentation.

Note that there are still more changes that we'd like to make to the Docker images. These changes didn't make it into the 5.1.1 packaging since they felt too invasive for a patch release. Instead we plan to make them in the next release, which is planned to be semver-major. Which brings me neatly to the last topic...

Sunsetting Node 4, 5, and 7 support

We had a good run, but it's time to say goodbye: Node.js upstream has marked Node 4.x as end-of-life, and in accordance with our version policy, we're doing the same. Since this is a semver-major change, we're also taking the opportunity to drop support for Node 5.x and Node 7.x. These changes have been made as of commit 32ad78, and soon we'll be ripping out old code used to support these versions, as well as upgrading dependencies that have recently started requiring newer Nodes.

Anyone still on these versions is encouraged to upgrade as soon as possible, as Node.js upstream is no longer providing security support for them. Administrators can use the NodeSource packages, or they can try out our new Docker images, which use a modern Node version internally.

Please reach out to the community if you need any help making the transition. And good luck!

New temporary signing keys

So unfortunately, recently I have lost my Nitrokey, which I liked very much. In addition to this being fairly upsetting, I am now left with the sticky situation of not being able to sign code - while I have a master key (from which I can generate new subkeys), I'm currently at college and my copy of the key is sitting 3,000 miles away at home.

To get around this situation, I've generated a temporary signing keypair. This keypair is set to expire after 3 months (and I don't intend to renew it). When I have access to my master keypair, I will revoke the old subkeys, generate new subkeys (it was probably time, anyway) and revoke the temporary keypair.

The new fingerprint is D825FD54D9B940FF0FFFB31AA4FDB7BE12F63EC3. I have uploaded the key to GitHub as well as the Ubuntu keyserver and (just as my original was). The key is also incorporated into my Keybase account so that you can bootstrap trust in it, if you want to verify software signatures or whatever.

How to evaluate domain registrars: a DNS story

Adriel: Any recommendations of where I should buy domain names from?

Me: I've heard is good and they support free software projects and organizations - I'm on Namecheap at the moment but it's janky and sometimes unreliable, and basically the only reason I'm still with them is inertia

Adriel: What makes the difference? Why does it matter where it comes from? I guess a better question is, what am I actually buying when I buy a domain?

Narrator: DNS is fraught with peril and complexity, and many hours would go by before Adriel got his answer...

This is a cleaned-up version of an explanation of how the DNS works and what owning a domain name really means. The original was much worse, because I typed it on my phone into a WhatsApp group chat and didn't edit at all. But you, dear reader, get the new-and-improved version! I'll start with Adriel's original question - why does it matter where my domain name comes from - and then transition into talking about how the DNS works and how the domain registar plays into that picture.

Here is, in my highly-opinionated opinion, the golden rule of registrar shopping - the #1 thing you have to know: all registrars are sketchy. Therefore you are looking for the least terrible option.

Once you understand this, you're looking at four things registrars have to provide:

  1. DNS services
  2. Whois data
  3. The web UI for managing everything
  4. Support

These are the most important, but depending on your usecase you might also want to examine value-add services registrars provide. Most registrars will also host your email and provide shared hosting, or sometimes a virtual private server, or VPS. A VPS is a box where you get root and can do whatever you want, including hosting a web server (but you have to do it yourself); shared hosting is where you get a managed web server installation that's shared (get it?) with other people. You get write access to a particular directory on the box that the web server is configured to serve from. (Registrars often also provide TLS/HTTPS certificates, but now that Let's Encrypt exists, why you'd pay for one unless you need an EV cert is beyond me.)

The third and fourth responsibilities are pretty easy to understand. Is the web UI butt-ugly or buggy, and is the support staff friendly and responsive. So I want to focus on the first responsibility, DNS, because that can be super confusing (I don't really understand the second, Whois, and anyway this post is long enough already). Here's the tl;dr: the registrar provides the servers that are responsible for answering DNS queries. Even if you use a third-party provider your registrar is involved, because they have to serve NS records that basically say "hey look over at this other provider for the REAL records."

Let's break down exactly what that means. Before I start I should note that if you've ever heard people say something along the lines of, "it'll take up to 24 hours for DNS to propogate," you should forget that. It's a convenient lie that people (including myself sometimes!) use to explain DNS' caching behavior more easily.

DNS works by recursively resolving each component in a domain name. It's probably easiest to demonstrate how this works by walking through the steps clients like browsers take when they're trying to resolve a domain name into a numeric IP address. So say we're trying to resolve, with no help from anybody else.

Our first step is to look up the authoritative nameservers for the com component. In other words, we're looking up the servers that have the absolute final word as to what DNS records com has. (More on exactly how this first step works later.) Once we've found the com nameservers, we issue a DNS query asking them where we can find the nameservers for The answer we get back will point to's registrar. Always. Even if they're using a different DNS service - the registar just points to the other service with NS records.

Let's pause in our journey to resolve for a second to consider the implications of this. This means that the registrar is always involved in DNS lookups for a domain, which is important for two reasons:

  1. If the registrar's DNS goes down so does your website
  2. If you want to use DNSSEC on your domain (which of course I have Opinions™ on but whatever) your registrar has to support it, because the way DNSSEC works is that EVERY link in the lookup chain MUST be signed, or the entire chain of trust falls apart[1]

Just things to bear in mind.

Anyway, we're almost done resolving We've found its nameserver, so all we have to do is issue our original query. Maybe we need an IPv4 address, in which case we'd ask the nameserver for all A records for; maybe we want IPv6 addresses instead in which case we'd ask for AAAA records. (If you want to know more about DNS record types, Wikipedia has a nice list!) If the registrar is the authoritative nameserver for it'll respond normally, if uses a 3rd-party DNS host, the registrar will respond with NS records. In the former case, we've gotten the data we originally set out to get; in the latter, we simply issue our query again to the nameserver the registrar pointed us at - which we now know to be authoritative - and if all goes well, we'll get a regular response and be done. As a reminder, the "authoritative nameserver" for a given domain is whatever nameserver contains the authoritative data for that domain. So we say e.g. "such-and-such a registrar's nameservers are authoritative for" For the authoritative nameservers could be completely different.

The overarching goal of the recursive resolution procedure I just laid out is simply to find that veeery last nameserver in the chain which can speak authoritatively for the domain we're interested in. Your registar's job, as far as DNS is concerned, is to put the domain in the nameservers for the top-level domain (or TLD - com in's case) and to either serve regular DNS records or point recursive DNS resolvers at the authoritative nameserver that will. There's also some other paperwork involved I think, but I wouldn't know much about that.

As an aside, I should note that normally, your computer doesn't do all this recursive stuff. There will be some DNS server upstream - perhaps running on your router or run by your ISP - that does it for you. This is to alleviate your computer from having to know how to do this and also because it makes stuff like caching work better. Speaking of caching, let's talk about the real explanation behind the admittedly super-convenient "24 hour propogation" lie. Now that we know how DNS resolution works the idea of "DNS propogation" is pretty simple to understand - it comes from caching. All that recursion stuff is expensive in terms of time (and nameserver load), so we want to avoid doing it whenever possible by generating responses from a local cache. This is accomplished by a Time To Live (TTL) associated with every DNS record response. The TTL basically says how long the record is valid for, and therefore how long it can be cached. When you change your DNS records and wait for it to "propogate", really you're just waiting for caches to expire. That means, though, that if you plan ahead you can lower your TTL to a few seconds, wait until everybody's cache expires (so everyone sees the new TTL), and then make your change[2]. This would lower downtime to a minimum. To be polite you'd then raise your TTL again. If you want to know more about this, here is an excellent Server Fault question that rehashes this explanation and then describes an exponential backoff strategy you might use to make such a change as efficiently as possible.

And with that, we've covered most of what you need to know about how DNS works, except for one thing - that mysterious first step. How did we get the nameservers for com?

The answer is the DNS root. The DNS root servers are at the absolute top of the DNS hierarchy, and they're responsible for resolving TLDs. In fact, if I was to nitpick, I'd point out that I lied earlier when I said we were trying to resolve In reality, we're trying to resolve (note the trailing .). If we read this new domain name, which is now fully qualified, from right-to-left, we get the chain we used to lookup the final authoritative nameserver. It goes: empty string, indicating the root "zone" (to use DNS parlance) -> com -> example. DNS is technically distributed but organizationally highly centralized around the DNS root, and trust in the DNS root, and people who don't like that tend to run alternative DNS roots (your friendly author is sympathetic to this position). But that's a story for another time.

Hopefully you now have a much better idea of how the DNS works and why your registrar has to be involved in it. I'm 95% sure this post is accurate, but I could always be wrong so if you spot a mistake, please feel free to contact me either by the usual ways or by Zulip if you're a Recurser. Good luck!


[1]: FWIW this is how Namecheap screwed up my domain - I turned on their DNSSEC option, but they had messed up their signing so that suddenly any resolver which validated DNSSEC would reject the legitimate responses as invalid. This was made extremely difficult to debug by the fact that you don't really know if a resolver is doing this for you, and often if DNSSEC validation fails that failure won't be passed directly on to the client (the client instead will get NXDOMAIN or something, which is basically the DNS way of saying "there's nothing there"). So resolution would fail on half the networks I tested from, with no clear difference. To make matters worse, when I went to turn of Namecheap's DNSSEC support because my website was down for a basically semi-random sampling of my visitors, the change didn't actually propogate to production! Like, I flipped the switch in the admin panel and it literally did nothing. So I had to escalate a support ticket to get them to purge all the DNSSEC stuff from production. Kinda ridiculous!

[2]: do note, however, that people sometimes don't follow the standard and set a lower bound (i.e. minimum) on TTLs. So if you make a change make sure stuff running on your old IP address won't wreak havoc if it gets production traffic.