I know how you're feeling. I'm torn between letting my DD be (she's just 2) and offering her what I think she wants/can handle (which makes me afraid of pushing her). She already knows what preschool would cover. Shapes, colors, the alphabet, she can identify all the letters and the sounds they make, she can count to ten, etc. I can tell she's close to reading.
I've looked into some Montessori things, and that's encouraging. Check out the blog Chasing Cheerios for example. We've started doing some things from there. DD loves matching, puzzles, sorting into groups, and so on ... and it makes me feel good about her learning AND just having fun!
I've also kept doing what I was doing before. Point out numbers, letters, read out in the real world, etc. I'd also recommend the book Teach Your Child to Read, Easily and Naturally, Before the Age of 3 by Timothy Kailing. Even though he's already reading it has great ideas for games to play with letters and words, and a lot could be applied to numbers since your DS likes math. Helping with cooking/baking is life skills plus math and science, would he like that if you pointed out the math and science concepts in it?
I would also look at things like Before Five in a Row and Five in a Row. Simple and easy for you, and covers a lot, but I don't think anybody would object to using it for fun with a 4yo.
And to take an idea from Charlotte Mason about narration, DD enjoys telling me about things and making a book out of what she says. We might take pictures during a zoo visit, then glue them into a notebook, and write a little about each one underneath. DD loves to tell me about pictures, and really likes the books we make.
She also loves Signing Time. I would encourage you to start watching this if you don't already ... there's no limit to what they can pick up from these! The explicitly taught signs of course, but you can pick up more during the songs and such. DD loves Signing Time and ASL, and I love that she's learning a new language in such a FUN way!
What we try to do on a daily basis:
read a book
play letter games
play outside (point out animals, weather, etc and make it "nature study", but also just free play)
help mama with some cleaning/cooking
puzzles
some idea from Chasing Cheerios or another Montessori-inspired type of thing
watch Signing Time
We also try to make two weekly "field trips". One to the library, and another to a different place ... zoo, aquarium, children's museum, a new park, the beach, a hiking trail, etc. At the library I let her pick a few books on her own, and I choose a few based on things she's shown an interest in. I don't really screen these based on her age ... we've read about electricity, the water cycle, trains, England, whatever looks good. She picks up a lot that I would think is above her level. I don't quiz her on it or anything, I just let her soak it up.
Long and disjointed, sorry, but I'm NAK. I hope you find something useful!