Well, wyandot lake is just across the parking lot from the columbus zoo... if you guys feel like having a fun amusement/water park type day. COSI is supposed to be very good, but they might not have a lot of stuff for a preschooler and baby. I have not been to the new one, but it's downtown by the river. The days you will be there... I am thinking you *might* be able to catch, "red, white and boom." It is one of the best fireworks displays in the midwest. Last time DH and I went, we took a canoe and the fireworks seemed as if they were right above us... it was so cool! I can't wait to take my kiddos to it when they are older. You may also want to visit the franklin park conservatory, I don't know if they have any special exhibits right now, but it is a very nice park and not far from downtown.
If you feel like spending a day shopping, easton is the nicest to place to go, esp now that the weather is gorgeous. It's one of those outdoor type malls with lots of nice stores and if your 4 yr old likes water, ppl put their kids in suits (bring normal clothes too) and let them play in the water fountain/sprinkler area.
Don't know if you would be interested, but the ohio historical center. It's more like a museum, I'm not sure how interested your 4 yr old would be, but you might find it interesting. http://www.ohiohistory.org/places/ohc/
Also, there are a lot of cute shops in the short north area, which is in btwn downtown and the OSU campus. The area has cleaned up a LOT in the past few yrs, you will be surprised at the difference. It wouldn't be a bad place to window shop... BUT several are art galleries, so if you have a rambuncious preschooler you may re-think going into certain shops (those with lots of glass sculptures/vases, etc.)
I'm sure there are a lot of other things to do to. I just can't think of all of them. I miss columbus, we live over an hr away, but try to make day trips there a few times a month.