strugee.net

Recently I backed up and restored both my Android phone and my Android tablet. There were a couple reasons for both of these. The tablet had been borked for quite a long time (any time I tried to upgrade it from the Android 5.0 build it was running, it failed - also the thing just froze randomly). The phone was on the CyanogenMod nightly channel and I wanted to switch to the snapshot channel, plus within the past couple days both WiFi and the cell network data connection straight up stopped working so it was pretty unusable. At first I wanted to switch to CopperheadOS on both devices. CopperheadOS doesn't support GApps and will probably never support GApps (for very good reasons), and I said, maybe I can make it work. Sadly, I couldn't - I still regrettably need stuff from the Google Play Store. (The tablet ended up back on stock because I want fast upgrades, and the phone ended up on CyanogenMod because they have the fastest upgrades while still offering root.)

The whole experience made me think, though - what would it take to create something that functioned like GApps, but respected your freedom? I'm sure some people reading are already scrambling to link me to their favorite Google Play Services reimplementation, but this isn't the only thing that's in GApps. You gotta think about the user experience, too. Such a system should be able to:

• Provide the nice APIs that Google Play Services does
• Store your photos in the cloud, like Google Photos
• Related to the above, automatically backup and restore apps and their data
• Transfer from other devices, similar to the above item
• Support functionality like Google Now
• Ditto for Google Assistant
• Integrate into the initial device setup to configure all this stuff

I'm sure there are more that I've missed.

Honestly, we're actually quite close to this. The first bullet can be mostly accomplished with something like microG. Automatic backup and photo storage needs a UI, but fundamentally can be accomplished with any generic WebDAV implementation. F-Droid can be used as the app store. Imagine this: you take a bunch of photos and install a bunch of freedom-respecting apps on your phone. Then, you get a new one. When you set it up, the phone prompts you to sign in to your WebDAV account (which could be e.g. ownCloud, or a WebDAV implementation on Sandstorm) and then automatically reinstalls all your apps from F-Droid, as well as retrieving their data from ownCloud. When you open the built-in gallery app, all your photos are already there because they're seamlessly backed up to the cloud. Your cloud. Transfer can be accomplished in a lot of ways, but I can easily see it building on the above.

Supporting something like Google Now is non-trivial, but I've even already proposed a feature for Huginn that would make this possible. Google Assistant would be very, very difficult, but even without that, we'd have come a long way.

This reality is not that far off. What's missing is some UI pieces and a nice ZIP that can be flashed on top of ROMs, similar to how GApps are flashed today. So who's going to put it all together?

(I suppose I've just volunteered myself - oh well... I'll just add it to my endless list of projects.)

Pump.io 1.0.0 is officially available! Whoooo!

I just wanted to write up an announcement real quick to celebrate. Here's a sample what's gone into this release:

• Node 4.x support
• Lots of security improvements including a better cross-site scripting scrubber and security-related headers that help protect the web UI (most notably, the web UI now declares a Content Security Policy)
• Minor improvements to the API to make it (slightly) smarter
• LibreJS support
• Numerous dependency upgrades, most notably Connect
• And of course, tons of minor bugfixes and improvements

For more details, see the brand-new change log.

And of course since we're now past 0.x.x releases, we're now officially making a commitment to the community to make only API-compatible changes going forward (or at least, until 2.0.0!).

As this release does improve security and fixes a lot of bugs, node administrators are encouraged to upgrade as soon as possible. If you have a global, npm-based install, you can upgrade with:

sudo npm install -g pump.io

And with a source-based install:

git pull
git checkout v1.0.0
npm install --production

If you're upgrading from 0.3.0, everything should Just Work(tm). Don't forget to restart your daemon!

One final note - the rumors are true. While we're not doing so yet, we are, in fact, planning to deprecate running under Node.js 0.10 and 0.12 very soon. Also, if you upgrade to Node.js 4.x early, the new, better XSS scrubber will be enabled - however, be aware that pump.io is far less tested under Node.js 4.x and you are likely to run into more bugs than you would under 0.10 or 0.12. This is an unfortunate situation, but sadly there's really nothing to be done about it. :(

Special thanks to Menno Vossen, Laura Arjona, Evan Prodromou, Jan Kusanagi and all the other volunteers who did so many different things to make this release happen. It truly wouldn't have happened without you.

Enjoy the release!

With <3,

AJ

So in my last post, I mentioned that I'd left a couple things for a second blog post. This is that post, and instead of being about all the cool stuff going on in the pumpiverse, it's about stuff that you - yes, you! - can do! It's super easy too.

Call for testers

So as I mentioned, the utml-to-jade branch is basically finished (see PR #1170). Since switching templating languages is a huge, huge change, by definition touching every single part of the Web UI, we want to make sure it's well-tested. This is especially critical given the fact that the Web UI unfortunately has very little test coverage.

That's where pump.io system administrators come in. If you're a sysadmin and you're willing to test this change on your node, we'd very much appreciate it. There's very little risk, since I think I've squashed all the regressions that happened, but you should be willing to report bugs if you do run into them. In particular, you should look out for:

1. Links that seem to encompass too much text
2. Missing spaces - e.g. Alex Jordanat [date] instead of Alex Jordan at [date]
3. HTML code showing up on the page - e.g. Test note<br /> instead of Test note followed by a line break

Note that the utml-to-jade branch incorporates all changes in the master branch, so you may want to check out the advice in Running from Git master.

Sound interesting? Want to take part in the development of pump.io? Installing is super easy:

$ [sudo] npm install -g e14n/pump.io#utml-to-jade

This will work even if you already have a (non-source) install of pump.io - just make sure to restart the server afterwards.

Note that this command has some semi-terrible logic to build Jade templates on install (this is a workaround for a deficiency in npm). If you get a scary warning message from npm, please file an issue, making sure to include the full log.

Call for design feedback

The other big thing that's happening is the pump.io 1.0.0 tshirt we're designing! I've spent quite a bit of time working on a variety of candidate designs which can be viewed in this ownCloud share. Obviously we want the coolest tshirt possible, so we're looking for any design feedback that people have. Anyone with some spare time can glance through the designs, and we'd be thrilled to get everyone's opinions.

If this sounds interesting, I'd welcome you to check out the drafts. As always, get in touch with the community through our chatroom or if you'd prefer, you can email me directly at alex@strugee.net.

Thanks so much! :)

So I promised a (long) while ago that I'd blog about all the stuff going on in pump.io. And there is a lot going on. Where to even begin?

LFNW talk

I think the first thing I should mention is the talk I gave at LinuxFest Northwest this year. It went really, really well (even though I finished half the slides the night before), and people seemed to be really engaged, especially during questions. It starts off by covering the history behind pump.io: that includes the relevant protocols, like OStatus and ActivityStreams, but also the motivation behind abandoning StatusNet (aka current GNU Social) in favor of a brand-new network. Next I move on to the way that pump.io itself works, namely, its function as a generic ActivityStreams distribution engine. To put this another way, I explain why this quote from the README:

I post something and my followers see it. That's the rough idea behind the pump.

is a pretty accurate way of describing what pump.io actually does. (I quote that quite a few times in the slides themselves.) I end with a discussion of the recent developments in the community, which are of course wonderful, and a short call-to-action for people to contribute - either directly to the software, or by joining the network and spreading pump.io.

Oh, and by the way - the talk was recorded! So you can watch it on YouTube or, even better, on my personal MediaGoblin. Fitting, since (as I mention in the talk) MediaGoblin federation will soon be released, and it's based on (and fully interoperable with) the pump.io protocol!

Conservancy application

Pump.io is applying to the Software Freedom Conservancy! This is super fantastic for us for a number of reasons. One of the biggest advantages is the fact that inside Conservancy (assuming our application is accepted), we'll be able to take donations much, much easier. This is primarily important because nodes in the pump.io network are volunteer-run, but they still require funds to operate. We're thinking about models wherein people will be able to donate to "pump.io", and then some (most?) of those funds will be used to pay for the network. (In particular, they'll be used to pay for the existing E14N nodes that Evan currently runs, which will become extremely useful once we transition those nodes to community owners). Conservancy also provides useful miscellaneous services, like owning our logo and making sure that if we encounter license violations, the license is properly enforced. But perhaps most importantly, becoming part of Conservancy cements pump.io even more in the community - both the group of people working on the software & network, but also the larger free software world.

ReadTheDocs

We've started a dedicated space for project documentation, hosted on ReadTheDocs. We're plannning to move a bunch of content from the GitHub wiki into this project, and hopefully it'll become a thorough and central place for pump.io documentation - both for users and for deployers.

Triage

We've spent quite a bit of time going through open issues and prioritizing them. A lot of issues have a release target now, and it feels really nice to feel like our issue tracker is a bit more organized.

Special thanks to Laura Arjona for driving this work.

Various minor code improvements

There have been a bunch of small bugfixes and improvements that have gone into the master branch - some of them user-facing, and some of them making the development experience better. Notable changes include:

• Migrating from Connect 1.x to Connect 2.x (this is just a start - Connect 2.x is still deprecated, but it gets us closer than we were to relying on a fully non-deprecated stack)
• JSHint is now automatically run against bin/ and routes/ when npm test is run. This makes it super obvious when there are regressions in code quality, especially in Pull Requests (since Travis CI will fail if JSHint doesn't succeed.)
• JSCS is now used to enforce code style. It's automatically run against the entire codebase (whoo!) when you run npm test, and it's awesome for the same reason - much of the style-related feedback that would've previously ended up in a Pull Request can now be dealt with directly on a local development machine, reducing PR review time for both the reviewer and the contributor.
• LibreJS is now supported
• Tests now pass! Whooooooooo! (Thanks to Menno Vossen for sending the enormous Pull Request that made this happen.)

Those are just the bigger ones, of course - there are a bunch of even smaller problems that got squashed as well. I'd also like to point out that quite a few of these were long-standing PRs which finally made it into core, which is awesome for everyone.

Express 3.x migration

I've been putting in a lot of work to migrate pump.io to Express 3.x. It's a huge amount of work, but when complete, it will bring us very, very close to being able to migrate onto Express 4.x, which is modern and fully-supported by upstream. Basically what I've been doing is just running the app, seeing where it crashes, going to the exception site, and fixing the problem. Rinse, repeat. You can check out this work on the express-3.x branch - currently, this branch can successfully start up the app, but will crash pretty soon after you try to do almost anything else.

This work, unfortunately, is on hold while another important project is completed: converting all the templates from utml to Jade.

utml to Jade transition

This is basically what it sounds like. Previously, the templates in pump.io were based on utml, which is essentially a thin wrapper around Underscore.js's _.template() function. However, utml doesn't work with Express 3.x (and it's not really worth making it work), plus it's not the prettiest to work with. Jade is an extremely popular templating language in Node-land nowadays, so a couple months ago I spent somewhere between 14 and 18 hours going through and rewriting all the utml in Jade, which was absolutely brutal - but necessary. Then, of course, I had to fix the client-side templating logic to handle Jade instead of Underscore templates, which took quite a while, along with the fact that I made a very large number of minor (largely cosmetic) errors in my conversions.

As I said above, this was kind of awful work (especially the beginning), but it's necessary and great, as it paves the way for Express 3.x and massively improves the contributor experience.

You can check out the gory details of this work in PR #1170, and the original reasoning behind why we're doing this in issue #1167. This work is actually done, but I'm going to write a separate blog post about it, calling for testers.

Upcoming 1.0.0 release

Last but certainly not least, we're gearing up for our 1.0.0 release! From a codebase standpoint, this is really just a small bugfix release (although it will make a lot of things less broken and - if I recall correctly - fix the actual installation process), but more importantly, it means that we're now committing to semantic versioning, which is a win for everyone (but especially administrators). The main thing that needs to be fixed before this goes out the door is the behavior of the XSS scrubber, which was accidentally made a little too aggressive. This is being tracked in issue #1169.

As a bonus, I'm also designing a t-shirt that (if there's sufficient interest) we may print as a celebration of this release - but more on this in my next post.

I realized I forgot to blog about all the work I've done on the strugee.net build code recently! I want to get back to working on Stratic part two (aka pretty indexes), but I thought I'd really quickly take the time to point out what's been done.

First of all, even though I rewrote the code in Jade about a year and a half ago, there was still a huge amount of boilerplate code in each Jade file. Why? Because essentially the way I did that was by running each HTML file through an HTML to Jade converter, then manually fixing up the result to be, you know, less ugly. That meant that common stuff, like the navbar, footer, includes, etc. were all duplicated across all Jade files. And that sucks, because Don't Repeat Yourself is a thing.

Luckily, refactoring is a thing too! So a few weeks ago, I went and split out the common site layout into its own Jade file, then made all the individual pages inherit from that common layout. The most difficult part of this was undoubtedly the navigation bar - I was pretty sure I'd have to just deal with the fact that the navbar was duplicated across pages (because each page has to have a different link deactivated), but then inspiration struck, and I ended up witht the solution I have now. Instead of treating the entire navigation as one block, I treat each link as a block that is overridable via Jade's block directive. And, since Jade blocks can have defaults, each page only overrides exactly what it needs to, and the rest of the links are automatically there and properly clickable.

This has a couple nice side effects. The first is that the codebase that generates strugee.net is now extremely easy for me to change, regardless of what part of the site I touch (whereas before, individual pages were pretty easy but changing e.g. the navbar require editing each and every individual page). The second (and clearly the most important) is that Night Mode, which I implemented in a 45-minute class over two years ago now (back when I was still in high school - what a crazy thing to think about not being in high school anymore!) now works on every page across all of strugee.net. Whooo! Silly hacks are at the core of the web, right? Might as well make them work well.

Finally, I'd be remiss if I didn't mention that I now have a proper deploy task, so I can abandon the sketchy and genuinely terrible deploy process that I used before. (For those curious: gulp build && git checkout master && mv dist/*.html . && git commit -m "Some bad commit message" && git push && git checkout src - not shown here are the several attempts where I made some typo or possibly didn't think of some files in dist/ that changed and so should have been mv'd, but weren't.) And, I now use root-relative includes now, e.g. /js/main.js instead of js/main.js, which is another win for DRY because now I don't have to maintain different header contents for subpages (which used to be the exact same, just with ../ prepended to all paths).

In conclusion: yay for progress! And yay for the fact that my codebase actually looks like someone competent wrote it! \o/

(Final note: I haven't forgotten my promise to blog about stuff happening in pump.io-land. But, not now.)