Funny enough, the Romans themselves didn't care too much for the toga. It was hot, heavy, and unwieldy. But it was like a suit - it was what you were SUPPOSED to wear on proper occasions.
Senators definitely did, I don't believe ordinary people would have. Though I could be wrong, I can't remember much on the details of Colosseum seating beyond division by social status.