Here's the thing: falling off your bike is dangerous but not as dangerous as being hit by a car. A significant number of biking accidents are someone being drove over.
If you get hit by a car, you could be hitting your head a lot harder than you would on a normal fall, so it could also be argued that makes helmets more important.
There's some danger, but yeah, nowhere near the level of danger that motor vehicles pose. I wear a helmet even if I'm not going on roads though because why not?
I agree--I have always worn a helmet too. Then you look at places like Amsterdam where tens of thousands of people commute daily on bikes with no trouble, but I also guess that they have protected vias.