I didn't really base it on an age, but we started playing the "how long can you hold your breath under water?" game during bath time.
About the time he was able to hold his face under water to the count of ten-ish is when I started darting in and out (mostly staying in).
I completely understand that this isn't a gurantee that he'd hold his breath if he fell over over knocked his head, but it did up my comfort level that he was able hold his breath that long. Going to the pool and swim lessons upped my confidence that he was safe in the bath by himself for longer periods of time
I think he was about 4...almost 5 when he started being able to do that. Up till that time I just wasn't comfortable...the risk outweighted the benefit for me. If I needed to be out of the bathroom that long for something that couldn't wait then I'd either postpone or shorten bathtime. However, we happen to have a fairly flexible evening routine....you might not.
He takes a shower now, but even now if he does take a bath I'm not usually out of the bathroom for more than 2 mins.
Besides, we live in a super small house so I could be to the bathroom in 15 steps even at the furthest part.
And, really, the kid never stops talking so anytime he pauses longer than a breath it peaks my alarm.