NotAwfulTech @awful.systems froztbyte @awful.systems 12mo ago

better tools thread

this is in part because it's for (yet another) post I'm working on, but I figured I'd pop some things here and see if others have contributions too. the post will be completed (and include examples, usecases, etc), but, yeah.

I've always taken a fairly strong interest in the tooling I use, for QoL and dtrt reasons usually (but also sometimes tool capability). conversely, I also have things I absolutely loathe using

wireguard. a far better vpn software and protocol than most others (and I have slung tunnels with many a vpn protocol). been using this a few years already, even before the ios app beta came around. good shit, take a look if you haven't before
smallstep cli. it's one of two pieces of Go software I actually like. smallstep is trying to build its own ecosystem of CA tools and solutions (and that's usable in its own right, albeit by default focused to containershit), but the cli is great for what you typically want with certificate handling. compare step certificate inspect file and step certificate inspect --insecure https://totallyreal.froztbyte.net/ to the bullshit you need with openssl. check it out
restic. the other of the two Go-softwares I like. I posted about it here previously
rust cli things! oh damn there's so many, I'm going to put them on their own list below
zsh, extremely lazily configured, with my own little module and scoping system and no oh-my-zsh. fish has been a thing I've seen people be happy about but I'm just an extremely lazy computerer so zsh it stays. zsh's complexity is extremely nonzero and it definitely has sharp edges, but it does work well. sunk cost, I guess. bonus round: race your zsh, check your times:

% hyperfine -m 50 'zsh -i -c echo'
Benchmark 1: zsh -i -c echo
  Time (mean ± σ):      69.1 ms ±   2.8 ms    [User: 35.1 ms, System: 28.6 ms]
  Range (min … max):    67.0 ms …  86.2 ms    50 runs

magic-wormhole. this is a really, really neat little bit of software for just fucking sending files to someone. wormhole send filename one side, wormhole receive the-code-it-gives the other side, bam! it uses SPAKE2 (disclaimer: I did help review that post, it's still good) for session-tied keying, and it's just generally good software
[macos specifically] alfred. I gotta say, I barely use this to its full potential, and even so it is a great bit of assistive stuff. more capable than spotlight, has a variety of extensibility, and generally snappy as hell.
[macos specifically] choosy. I use this to control link-routing and link-opening on my workstation to a fairly wide degree (because a lot of other software irks me, and does the wrong thing by default). this will be a fuller post on its own, too
[macos specifically] little snitch. application-level per-connection highly granular-capable firewalling. with profiles. their site does a decent explanation of it. the first few days of setup tends to be Quite Involved with how many rules you need to add (and you'll probably be surprised at just how many things try to make various kinds of metrics etc connections), but well worth it. one of the ways to make modern software less intolerable. (honorary extra mention: obdev makes a number of handy pieces of mac software, check their site out)
[macos specifically] soundsource. highly capable per-application per-sink audio control software. with the ability to pop in VSTs and AUs at multiple points. extremely helpful for a lot of things (such as perma-muting discord, which never shuts up, even in system dnd mode)

rust tools:

b3sum. file checksum thing, but using blake3. fast!. worth checking out. probably still niche, might catch on eventually
hyperfine. does what it says on the tin. see example use above.
dust. like du, but better, and way faster. oh dear god it is so much faster. I deal with a lot of pets, and this thing is one of the invaluables in dealing with those.
ripgrep. the one on this list that people are most likely to know. grep, but better, and faster.
fd. again, find but better and faster.
tokei. sloccount but not shit. handy for if you quickly want to assess a codebase/repo.
bottom. down the evolutionary chain from top and htop, has more feature modes and a number of neat interactive view functions/helpers

honorary mentions (things I know of but don't use that much):

mrh. not doing as much consulting as I used to, using it less. quickly checks all git(?) repos in a path for uncommitted changes
fzf. still haven't really gotten to integrating it into my usage
just. need to get to using it more.
jql. I ... tend to avoid jq? my "this should be in a program. with safety rails." reflex often kicks in when I see jq things. haven't really explored this
rtx. their tagline is "a better asdf". I like the idea of it because asdf is a miserable little pile of shell scripts and fuck that, but I still haven't really gotten to using it in anger myself. I have my own wrapper methods for keeping pyenv/nvm/etc out of my shell unless needed
pomsky. previously rulex. regex creation tool and language. been using it a little bit. not enough to comment in detail yet

28 comments

Back when I was an intern at a small ads company, there was one guy hyping fish really hard, and as a pathological contrarian I was compelled to write it off completely as hipster brogrammer trash. Anyone here use it? Any reason to switch to it from, say, zsh?
- A couple of my friends use it, it seems nice. I’m just a stubborn lazy ass
  
  More seriously: I’m not entirely certain it’s what I want in my shell, tbh, and I haven’t really had the spoons to try figure that out either
These recommendations are lovely and I will try most or at least some of them, at least the ones relevant to my use cases.

I adore wireguard and I use it for couple of personal and small org things. I would hate to set it up for a double-digit number of people. OpenVPN and IPsec are a pain in the ass and not just once, but in some use cases I would absolutely prefer them to wireguard.

Zsh is my main shell and I love it. Sometimes it breaks in bizarre ways through odd interactions between plugins. Sometimes I just spawn a different terminal with a more primitive shell because zsh's tab completion shits itself attempting to list a dirtily dropped remote filesystem.

I don't particularly like language-specific package managers. It's not too bad to install rust packages through cargo instead of apt when I'm writing rust, but if I just want to put a thing on a Debian box, why do I have to give a shit if it's implementes in C, Python, Rust, JS or Fortran? Fully a case of sysadmin brain over programmer one.

I risk playing the heel because I believe these tools, lovely as they are, have weaknesses that can sometimes be addressed and sometimes constitute inherent tradeoffs that need to be considered. This is not synonymous with discounting them or considering them inferior to other, possibly older solutions.

For a more positive contribution I will second ag as a grep/ripgrep alternative, though I will refrain from comparing it to rg itself.

And for a more old school recommendation, awk has been invaluable in my career and I would heartily recommend any unixist to learn it a little further than { print $n }.'
- re vpns: yeah ipsec is just human-hazardous. even few technical people get it correct. never mind the mistakes with cipher choices and inter-vendor/-kit fuckups. something that might be worth you knowing about: pritunl has wireguard support, and they generally make an okay product for wrapping ovpn et al. wireguard distribution also improved a lot in recent years, in the form of being able to load tunnel configs from files (and even QR codes, in mobile apps). if you've only learned your wg from the early "edit a config" or "wg-quick" era, might be worth a review
  
  re zsh completion: do you mean completion on e.g. filenames in something like a directory that was linked over sshfs/nfs/etc? admittedly I don't use remote filesystems much in my workflow (in part because I think it's fundamentally broken, likely to lead to such issues, and thus avoid it in my control loop), so I'm unlikely to hit that pain point in a lot of cases
  
  lang-specific managers: yeah, I know where you're coming from. I'm on both sides of that fence. languages are generally at most equipped to progress their own tooling as fast as they can manage (and even then not so fast, e.g. npm still being dogshit at dependency solving, instead of just learning from the other kids), so I do understand the why of lesser integration. but it irks my inner sysadmin/automationist just as much as you, because holy shit are some of those things badly thought through. the amount of implicit "oh I'm just gonna drop 19 envvars everywhere, and some undocumented magic files" involved in a lot of these things is... 🤬. I would like if languages would work with distro tooling teams, to see how they can end up making some of that smoother. python (and them interfacing with the debian lang teams) is notorious here. ah well. at least rust and clojure don't suck that much
  
  (aside: still getting to know the nix thing. being on an m1 makes it a bit spicy)
  
  ag: get that metal out of your blood, it's bad for you~ :D but yeah, I will say that ag was what I used for a few years before I found out about ripgrep
  
  awk: the sheer amount of heavy lifting I do with cut, tr, awk, etc over the years has definitely amounted to sizable amounts of compute hours saved. solid recommendation
  
  funnily enough i never thought about recommending awk, because it was so obvious to me that it's incredibly useful.
  
  (i just realised that i learned awk almost thirty years ago, and that for most people it might be just that slightly dusty thing that's lying there, unused)
jql looks like exactly the kind of thing I’ve been looking for, as someone who bounced right off of jq.

some quick recommendations:

amaranth is the only good hardware description language (and I’ve tried many), though its docs are lacking. pair with symbiyosys formal verification and a Lattice FPGA for a very advanced, fully open source hardware design environment

doom emacs is a modernized, easy to pick up emacs config distro that runs a lot faster than spacemacs

gnu parallel lets you easily parallelize arbitrary tasks on the command line. it’s flexible enough to do everything from devops tooling to job handling on massively multicore machines

I’ve got many more recommendations, I’ll come back to this when I’ve got more free time
- I really need to get to trying out doom sometime. still on spacemacs, and yeah fuck it is slow. I don't have the desire to write my own emacs config from scratch either.
  
  parallel: yeah. I.... it's mostly sorta good? but god I hate it. the way the args layout works sucks. I've looked for alternatives and haven't really found any. I'll often still just go for xargs -P -n 1 over parallel because parallel just ugh
  
  doom’s compilation model is great! it AOC native compiles everything it can, then the JIT runs for everything else. it’s all pinned to specific versions of packages too so breakages are relatively rare. there’s even a package I use that replaces doom’s build mechanism with Nix, though it’s in desperate need of updates for newer versions of doom
  
  I agree on parallel, it’s a good tool to have but I do wish there was a good alternative. parallelizing tasks like it does feels like something that should maybe be built into my shell, especially given how good desktops are at multiprocess workloads now — even something like a multiprocess for loop that operates sensibly could be a massive improvement over anything we have now
OK this is totally niche but QPDF is a nice tool for doing most "content preserving" PDF operations and it's maintained by a single person who writes good code and is always super nice when I send pull requests and really positive to people who file bugs even if they're kinda basic questions.

I use it at work for PDF linearization but really I just wanted to share how drama free the project is talk about a breath of fresh air.
- I’ve actually been looking for something in that space, will give it a whirl
i shall only mention that this type of thread is not complete without mentioning this julia evans' blogpost.
- …the little tools i'm personally using without too much complaining (and haven't seen mentioned so far) include:
  
  ag, aka the silver searcher which is probably like ripgrep,
  
  gopass, which we're currently using as the secret storage for our ansibles,
  
  gojq, which is just like jq, but also has --yaml-output and --yaml-input built-in,
  
  vcsh for my config file management,
  
  himalaya, the cli (…not tui) email client,
  
  direnv for location-dependent shell configuration,
  
  desk for more general task groupings.
  
  I don't follow evans so I wasn't aware of that post existing. cool to see some overlap I guess
  
  aka the silver searcher
  
  just switch to ripgrep:
  
  froztbyte@bambam ~ % cd ~/code froztbyte@bambam code % dust -n 0 10G ┌── .│████████████████ │ 100% froztbyte@bambam code % hyperfine -m 15 -i 'ag somerandomstring' 'rg somerandomstring' Benchmark 1: ag somerandomstring Time (mean ± σ): 8.728 s ± 0.149 s [User: 4.112 s, System: 12.585 s] Range (min … max): 8.585 s … 9.133 s 15 runs Benchmark 2: rg somerandomstring Time (mean ± σ): 2.359 s ± 0.095 s [User: 0.935 s, System: 3.690 s] Range (min … max): 2.285 s … 2.665 s 15 runs Warning: Ignoring non-zero exit code. Summary rg somerandomstring ran 3.70 ± 0.16 times faster than ag somerandomstring
  
  for our ansibles
  
  I'm sorry. e: this wasn't said in snark - ansible sucks so much, and it sucks that you have to use it
  
  gopass: why ... is this its own thing in go? it doesn't seem to advocate why it's a thing. not that I particularly care, I'll almost certainly avoid it (on the basis of it in go), but it's weird that it just exists as a plural copy without advocating why
  
  gojq: see above
  
  vcsh: heh I forgot this exists. I recall when richi first started working on that. worth mentioning indeed, some people might use that (I don't (my hg ~ repo has been going since the dark times and I really like that I can still just hg clone . ssh://host//path/to/dest and have it just dtrt))
  
  himalaya: heh, ran across that a while back when looking into rust mail implementations (also when I found vomit/vmt). guess I should setup a trial sometime just to see how it rolls
  
  direnv: years ago I first wrote this off because on spinny rust it was just awful (and even running into it on AWS hosts it was annoyingly slow because people run systems on ext3/4). recently reinstalled it on a host but haven't found myself using it much
  
  desk: cute. how does it handle/where does it store state? how does it deal with e.g. path moves? not sure I'd ever use it (see aforeposted grump about miserable piles of shellscript), but cute
a downthread thing reminded me, so: moreutils by joeyh. has a couple things that are quite handy on a lot of unixy systems. check it out
zsh and ohmyzsh

install powerlevel10k as a theme. Do it now. Thank me later.

https://github.com/romkatv/powerlevel10k#oh-my-zsh
- You missed the key indicator on that deliverable
  
  NOT oh-my-zsh
  
  Shit’s janky and slow as hell
  
  but it does free me from having to care about few things which for me is worth these 100ms on startup; anyways: what would you recommend instead?

28 comments