For me it’s hard to say for sure, because my scariest book wasn’t meant as horror, but Communion by Whitley Streiber is definitely responsible for the most sleepless hours.
I just finished The Thing Between US by Gus Moreno. I've read a lot of horror and I've never jumped or turned my face away or slammed the book shut, but I did all of these things at different points in this book. Something about the way Gus Moreno describes things really freaked me out lol. It was a phenomenal book and I can't stop thinking about it.
Penpal by Dathan Auerbach, was a book that really succeeded at scaring me personally. It managed to make me feel pretty vulnerable and it just filled me with dread the whole time I was reading it.
Probably isn't really horror but I've never felt more uncomfortable than when I read House of Leaves. Just being alone in an apartment felt creepy while reading certain parts.