In NPM I set a proxy host 192.168.box.IP to forward to 100.jellyfin.tailscale.IP:8096. I tested it by going to box.IP and jellyfin works.

I'm not surprised this worked, numbers are allowed in FQDNs, but an IP address is not entirely equivalent.

I tried "box.IP:8096" as a domain name and NPM rejected it. I tried "box.IP/jellyfin" and NPM rejected that too (I'll try Locations in a bit)

I would strongly suggest you to read up on the OSI model.

Nginx only understands HTTP and HTTPS requests at Layer 7 (implicitly and strictly ports :80 and :443), and forwards or redirects them to Layer 4 destinations. (Nginx can technically handle other protocol requests via plugins, but that isn't what you are looking for.)

In NPM, the proxy host name should at least contain the Raspberry Pi's hostname, e.g. jellyfin.your-rpi-name. Or you could use the path location option, e.g. your-rpi-name with location /jellyfin. (I think the second option might work with network hostname auto-discovery, in which case pihole as a DNS may not be strictly necessary.)

I personally use rdfind as it has an option to replace duplicates with hardlinks instead of deleting them outright (if on the same filesystem). This is useful if you do still need a file to exist at multiple paths.

I then use Czkawka for everything else, especially for similar, non-duplicate files.