I think 4 would be a magical age for Disney.
We took the kids when they were 1 and 3 (and the 1 yr old didn't get that much out of it, but I would still say he had fun)
Then to Disney World when they were 1, 3, 5, and 7 (had two more kids at this point). They all loved it, but especially the 5 and 7 yr old.
We went to Disneyland again this past summer when the kids were 2, 4, 6, and 8 and it seemed like a great age for all of them. Of course, the 2 yr old couldn't go on certain rides - but he loved the ones he could go on, he loved all the characters (especially meeting mickey), he loved the parade and light shows, fireworks, etc.
Personally, I really don't think they are ever too young to enjoy it (except maybe a newborn, who who have no clue what's going on). But if I had to pick an ideal age for a once in a lifetime visit - I would probably say between 4-6 years old. Any older and some of the 'magic' is lost as they know the characters aren't real (not a bad thing, just a different experience).
That all said, I went as a kid when I was 8, 13, 14, and 15 -- and had a blast each time for different reasons. So, I also think they are never too old to enjoy Disney, either.