It may be on the older side, but I've been using my Pixel 4A for three years or so, no complaints. I also use the headphone whenever possible. Also using GrapheneOS.
You may want to check out the Pixel 5A (also has a headphone jack if I'm not mistaken) for a bit of a newer phone, but the one I got used seems to have a band issue in which it would forcefully shut down after some time with my SIM card.
I think the fragility is either overstated or there were some lemons in the batch. I've had mine since day one, don't treat it with kid gloves and have never had an issue.