Packages or binaries?

As much as possible, on my Fedora systems (which runs on my desktop, laptop and a handful of Raspberry Pis), I use what dnf provides me with. The same goes for machines I run other operating systems on, like my pihole server (which runs Raspberry Pi OS).

Sometimes though, that is not enough, when software is not packaged or recent versions are not available over all the platforms I use.

Obviously, I could package all software that's not currently in Fedora myself, but alas, I simply do not have time to always do so.

Downloading binaries

Some binaries therefore, I simply choose to download directly from GitHub. Easy enough, until you amass a couple of dozen of those. Then it becomes a burden to keep them up to date.

A long time ago, I wrote a script to grab the latest binary of a project for my OS and architecture and drop it into ~/.local/bin.

Over time, this script has become more complex (as scripts do), so recently I dropped it into git and made it configurable and capable of batch downloading a whole bunch of binaries at once. It's called 'ghdl', for GitHub Downloader. Very poetic.

Intermezzo: the sad state of releases on GitHub

Binary releases on GitHub are a complete and utter mess. Virtually every project out there uses different schemes for naming their binaries.

• Some projects put the architecture of the binary in the name, some don't. Some projects call it amd64, some call it x86_64. Some projects call the ARMv7 architecture - quite generically - arm, some call it armv7l. The trailing 'l' is dropped off some times.

• Some projects call the aarch64 architecture arm64, which I have more than once downloaded by accident instead of amd64.

• Some projects release binaries with 'darwin', 'windows', and 'linux' in the name. Some do the first two, but leave the OS out of the name for the binary meant for Linux.

• Some projects provide plain binaries, some provide zip files, some provide tar balls. Some provide all three. Some have yet other ways of packaging their binaries. There is one project, for example, that only offers a single zip file with binaries for all architectures in it. Yay?

• Some projects do not offer actual binary downloads of their software in releases at all, but just provide source code in zipfiles and tarballs (which is quite useless for me).

• Some projects do not actually release anything (like the 'kn' binary for knative), because all of their releases are labeled as 'pre-release', which makes the releases page in the API show up empty.¹

It would be great if GitHub could provide some guidance on file naming. Right now, all GitHub does it nudge people towards semantic versioning - which is nice, don't get me wrong - and tell people to prepend a 'v' in front of their version number.

It would be awesome if GitHub would help people all follow the same template to name binaries. Something like mybinary-linux-amd64 and mybinary-darwin-aarch64.

It should actually be relatively simple for GitHub to detect the executable format and deduct the platform for the binary. GitHub could have people upload a binary, ask for a name ('mybinary', in my example above) and do the rest in the background: rename the file to mybinary-linux-amd64, and provide optional tarballs and zipfiles. All the same for every project. One can dream, right?

But anyway, GitHub is not doing that, (and maybe there are downsides to it that I didn't think of) so I tried hacking around the plethora of different naming schemes and came up with a script that does the trick for me. It batch downloads some 40 binaries from GitHub for me without error. You can easily specify your own list. The Readme explains how to do that.

Intermezzo: So what about the gh binary?

What about it? I don't want to rely on yet another a binary to just talk to the GitHub API. Apart from that, it actually does very few useful things for me. It doesn't select the right binary for me, and it doesn't unpack it. I could use it to query the API, but that's the easy bit of the script to be honest. The tricky bit is selecting and downloading the right binary, and it doesn't help there ;)

Trying out the script

If you want to give it a go though, you can go to GitHub and grab it. I just pushed version v0.5, which has the ability to parse a YAML file of projects and organizations and download the latest binary from all of them.

The script will auto-detect your OS and architecture, but truth be told, I have only tested it on Fedora (mostly), and Debian (a bit) for operating systems and mainly x86_64 for architectures.

To show a bit of an example, downloading the Digital Ocean commandline (doctl) would work like this:

./ghdl.py --org digitalocean --project doctl
URL:  https://api.github.com/repos/digitalocean/doctl/releases/latest
Latest version found:  v1.54.0
Release URLs found:
 -  https://github.com/digitalocean/doctl/releases/download/v1.54.0/doctl-1.54.0-checksums.sha256
 -  https://github.com/digitalocean/doctl/releases/download/v1.54.0/doctl-1.54.0-darwin-amd64.tar.gz
 -  https://github.com/digitalocean/doctl/releases/download/v1.54.0/doctl-1.54.0-linux-386.tar.gz
 -  https://github.com/digitalocean/doctl/releases/download/v1.54.0/doctl-1.54.0-linux-amd64.tar.gz
 -  https://github.com/digitalocean/doctl/releases/download/v1.54.0/doctl-1.54.0-windows-386.zip
 -  https://github.com/digitalocean/doctl/releases/download/v1.54.0/doctl-1.54.0-windows-amd64.zip
Filtered for linux, x86_64:
 -  https://github.com/digitalocean/doctl/releases/download/v1.54.0/doctl-1.54.0-linux-amd64.tar.gz
Filtered for tarballs / zipfiles:
 -  https://github.com/digitalocean/doctl/releases/download/v1.54.0/doctl-1.54.0-linux-amd64.tar.gz
Downloading and installing file:
Downloaded file: /tmp/tmpa0j5yav8/doctl-1.54.0-linux-amd64.tar.gz
Largest file: /tmp/tmpa0j5yav8/doctl
Cleaning up temporary directory
Symlink exists. Removing and recreating it.
Symlinked /home/myuser/Sync_Workstations/files/doctl-v1.54.0 to /home/myuser/Sync_Workstations/links/doctl

The script is a bit verbose at the moment. It's on my todo list ;)

I hope this helps some people that are tired of keeping their minikube, starship and doctl binaries up to date!

Caution!

A small word of caution here. The script will probably break for you. It works for me, on my systems, but there is no guarantee whatsoever that it will work for anyone else!

Also, right now, this script is a prototype and a huge mess, but I decided to release it into the wild because it might help others.