I recommend the Museum of Natural History. Fun for all ages and you don't have to worry about keeping your younger kids' hands off the masterpieces. Also Central Park is right outside. Unless you're really looking for an art museum, which it isn't. In which case I would suggest the Met - they love kids there but you do have to keep their hands off the art. It's also right on the park.
Museum, Library and park sounds like an awful lot for one day, especially with travel at both ends. We frequently spend the entire day in Natural History and see about a quarter of it. You might want to scale back. Or replace the museum with a sightseeing activity, like the Empire State Building.