What happens to the posts and comments made in an instance if that instance shuts down?

To my understanding, it gets deleted in that instance, but a portion of it may be preserved if another instance that is federated already holds that data.

Doesn't it just fall under whateve are that records retention policies for that instance?

Two things: the data on the instance will be gone.

The posts/comments that were replicated on other instances will stay, but they will be defederated, meaning they will not ever be synced up woth other instances' replications of these posts/comments. Every instance will just have their own, insular copy.