I think the solution lies in deleting their photos. For a multitude of reasons. I'm almost at that point, personally. I don't need to see that face anymore, ya know? It's been years.
You can filter out dates from showing memories. If you get a photo you don’t want to see, tap the three dots, access memories settings and pick dates you want to hide
Yes let me just scroll through years of photos and spend hours combing through them all to find the few specific pictures they decide to randomly throw in your face.
I have pictures on my phone from over ten years ago. The reality is no one asked for this "feature"