We don't have a specific line item for babies, but the expenses associated come out of our other budget areas.
For instance, I exlusively breastfeed - often for nearly the entire first year (my kids so far haven't been too interested in solids earlier). That means I need a few more calories and that expense is met by our grocery budget line.
The baby needs diapers and clothing, but that comes out of the clothing budget line.
Any furniture needs would come out of our miscellaneous budget line - this time we added a twin bed to increase our family bed from a queen to a queen + twin pushed together.
We live very simply and our babies don't need much "stuff". I bought a used changing table and we were gifted a pack 'n play. That gives us two nice areas for changing diapers - one for upstairs and one for downstairs.
Since we already have kids, our health insurance doesn't increase - we already have the family premium. We've been very fortunate in that our kids have been extremely healthy and we rarely visit the doctor more than once every 2-3 years.
Dental is pretty much the same, with the exception of 2x a year cleanings. But babies don't use the dentist often.

The only "big" expenses I recall are car seats and a jogging stroller. I wear/carry my babies until they're about 2 years old, but after that the stroller is handy. I'll be able to use the car seats we used for DD for this coming baby and the stroller we already have will likely be used later on for this baby as well.
We received very generous cash gifts after my DD was born and put that money into a savings account to save for future expenses. It's a nice cushion so when we did need to move up in car seats to a differnt model, the money was on hand already.