I was fully expecting Denji to keep erasing devils until the world becomes barren, ending up with a finale similar to that of some of the author's older work, but I'm glad the plot is not heading that way. I'm excited for how the story will continue unfolding.
I think they knew it more as a concept than a fact. Because, if they had known from the beginning, they would have gone straight to extracting something instead of making him eat something and then throwing it up.
In the same way Makima and Yoru still knew about the things Chainsaw Man had eaten, perhaps. Apparently, the Devils don't have much difficulty in still remembering those things. Maybe there is still a Devil behind all Public Safety.