constant transcoding is the issue. Doesn't really seem to matter how powerful the server is. I've found the best option is to standardise on a single codec format and a set of bitrates based upon resolution.. EG: older DVD based 720P I use AV1 at 800Kbps.... Then all my clients will direct play

Transcoding the library outside of Jellyfin allowed me to use an 4th gen i3 with 12GB RAM for the jellyfin server itself as it now just streams everything, no transcoding on the fly whatsoever