Introducing 1bin.org - an easy way to download and run CLI tools on Linux and Mac without installing them

This post is about how and why I built nikvdp/1bin and it’s companion site 1bin.org. If you just want to download static binaries of CLI tools, check out 1bin.org. If you’re interested in learning how to create your own self-extracting 1bins, take a look at the repo.

1bin.org is a scratch-my-own-itch project that’s been almost 5 years in the making. Long long ago in a younger nikvdp’s career I spent most of my days doing infrastructure engineering. In my case this involved building internal tools to ease common internal tasks, containerizing things and configuring our CI pipelines, and making sure our dev envs were easy to get set up so that new engineers could onboard easily and everyone else could unhose hosed dev envs when needed. Since much of our stack was Python, in the process of all this deploying I got introduced to Conda and got to get my hands very dirty in docker-land. Docker did a decent enough job of getting us consistent and reproducible builds in prod on Linux, but most of the team used macOS for their daily dev work and keeping everyone’s packages and tooling in sync quickly became a headache. We experimented with running our dev environments under docker on each dev’s mac, but this led to even more issues, mostly around performance and issues with docker’s shared folders. Eventually I dug into conda’s documentation and learned how to distribute out internal tools and libraries as conda packages by writing conda recipes for them and set up a private conda repo to distribute them to the rest of the team. This approach ended up working far better than our attempts to dockerize our dev envs had, and had the side effect of making me realize that conda, despite not being on anyone’s hype list and having some quirks, is a powerful and underrated tool.

Fast forward a few months and I was reveling in the glory of building everything in Conda and looking down my nose at Docker. I started to wish that I had conda installed in every docker container I accessed. Why? Because then I could use all the command line tools I was used to from my dev machine when I was kubectl exec-ed into a remote container attempting to debug something nasty. Since none of the standard CLI tools or text editors I was used to were available while debugging remotely (because they’d been stripped out to keep the docker image size down) it often felt like working with my metaphorical hands tied behind my back since. I began to dream of a site where I could download a single file that had everything I needed to spin up a copy of neovim or jq while shelled into a remote container so that I could actually debug effectively.

Fast-forward a few years and I once again found myself attempting to deploy packages with complex dependencies to end-users machines on multiple platforms (this time as part of building Uncloud), and came across the wonderful conda-pack. A few months after that I stumbled upon the last piece of the puzzle: warp, a tool that can turn any directory of source files into a self-extracting executable, and with that everything clicked into place. With these tools together, it should be possible to take anything that can be packaged up as a conda package (which is pretty much everything), and convert it into a single self-extracting binary that could be deployed and run on any Mac or Linux machine without any additional installation steps. Since Github Actions now provides free CI minutes for open-source projects and free hosting for binary artifacts through their Releases, this meant that it should be possible to build a hefty collection of CLI tools for free, and host them on Github, also for free.

Unfortunately, the URLs to the artifacts in a Github Release don’t exactly roll off the tongue (keys?), and while having a website I could go to to right click and copy+paste long URLs into containers is certainly better than being unable to install these tools at all, it wasn’t cool because it required me to touch the mouse, and I am nothing if not a keyboard snob. I wanted my tools to be available from an easy to remember URL so that I could just curl or wget them down on the fly and start using my tool without having to break out of context and go on a Github google quest. Luckily, the Github API makes it relatively easy to pull down a nice JSON representation of all the artifacts attached to a Github release, and since Next.js makes it easy to build websites that can pull in external data at build time and there’s no shortage of places to host static Next.js sites for free these these days, a little website that redirects URLs to the Github Release artifacts ought to be easy to build and free to host too.

So, I built it. I set up some scripts and a Github Actions workflow to build the CLI tools I use regularly for Mac and Linux, pack them up into single self-extracting executables with conda-pack and warp, and then upload them all to a Github Release. Then I built a small Next.js app and used Next’s getStaticProps() feature to hit Github’s API and get a JSON representation of the latest release over at nikvdp/1bin at build time. Sprinkle in some URL redirection logic to automatically set up redirects based on the contents of the Github Release metadata so that I could have relatively easy to remember URLs and a single command to remember that would work on both macOS and Linux (ie wget 1bin.org/$(uname)/jq) possible and this project was starting to feel pretty close to my long-sought CLI nirvana. 1bin.org now had most of the functionality I’d originally wanted out of this project, but as a last step I decided to build an actual web frontend and add a dab of fuse.js fuzzy search magic and copy-to-clipboard functionality so that users could navigate to 1bin.org, type the first few keystrokes of their desired tool’s name, and copy the download+install commands to the clipboard in one go.

There are still many improvements to be made, especially on the backend, but 1bin.org is now mostly feature complete and my five-years-in-the-making itch is now officially scratched. I hope you’ll find it useful as well.

If you’d like to see more tools added to 1bin.org PRs are very welcome, please feel free to hop on over to the repo and open a PR to add new tools. Many popular tools are already available as conda packages thanks to the wonderful people over at conda-forge. If the tool you’re interested in is already available there then a PR may be as simple as adding the package’s name to build.sh. If no conda-forge package exists then a little more work is needed, but it should be possible to write a conda recipe for almost any tool and add it to 1bin.org as a custom recipe.

I’m also experimenting with a method to package Nix derivations into conda packages, if you’d like to help with that please feel free to get in touch on Github or Twitter! Adding nix support would greatly increase the selection of packages that could be easily added to 1bin.org since nixpkgs provides a quite comprehensive selection of packages.