When do they start to get this time thing down?

My son is almost three. He seems very bright to me, but the concept of time is really confusing for him. I don't mean telling time, but the whole morningtime, naptime, nighttime sequence. I wouldn't say I'm worried about it, but it seems incongruent with how quickly he picks up other things. I guess I'm just wondering what other kids out there grasp on this topic. He's nearly out of toddlerhood I guess, nearing three, but I'll ask here anyway.

For instance, he confuses the idea that meals at different times of day have different names, and gets confused between afternoon naps and nighttime sleep. He'll wake up after his nap and say it's morning time, or think nighttime is afternoon nap, or want reassurance that his afternoon nap is just a nap. It just surprises me because our routines are predictable and there are lots of cues, i.e. at naptime his dad's not home yet, we've only been up for a few hours, we've eaten lunch, and it's light out. Or he'll think we've just eaten when it's been hours. This is normal wonky toddler learning, right? He learned his colors and numbers and letters quite early just in playing together, and speaks so well. I guess it's hard for me to grasp how confusing this whole lapsing of time thing is. I think I read somewhere that babies have no concept of time, but I don't know when they start to grasp that.
What about your 2-3 year old?