I lean toward the live-and-let-live side myself
On formal nights I skip the cap, but for regular dinners I wear one without worry. For a clean, simple cap that looks fine at the table, I personally use
richardson hats wholesale for most of my casual outings
They fit well and hold up after many washes, so I feel comfortable keeping one on in relaxed settings