I'm not sure how I'd deal with that from my mom (though I like Dar's take on it). Often, you get some version of that from other mothers in your peer group. "Oh, my son and I would drive each other up the wall if we were together all day!" or "I don't know how you do it! Three days into summer vacation and I'm ready to send them back!", etc. I don't take offense because, first, they might be right! I don't know for a fact that Jane and her son *wouldn't* drive each other up the wall!
Second, no one likes to have their routine disrupted, no matter how much they love their kids. If your routine involves having the kids in school, the disruption of that takes some adjustment -- for parent and kids -- and can be rocky. My sister, who has kids in school, has said that it takes about 2 - 4 weeks for her kids to "decompress" at the beginning of summer break and to start behaving normally and not bickering a lot, etc. Then they fall into a summer routine, but for shorter breaks, things stay in a state of semi-upheaval until school starts again. For many working parents, the disruption of school breaks is even more acute.
Third, children who are out of their parent's control for a good portion of the day spend a lot of energy being their "public" selves during that time. They behave, they delay gratification, they share with their classmates, they eat when told to, etc. So when they get home at the end of the school day, guess who gets the brunt of all that pent up repression? Good, safe, Mom! Many moms assume that the "after school" child is the one they would have to live with *all day long* if they homeschooled. In reality, a child who is with Mom or Dad stays on a more even keel throughout the day. They may have their moments, but you don't live with a little demon all day, lol.
So when I get comments about "how I do it" or how exceptionally great my kids must be, I usually point out the last two things above. (Agreeing with them -- point #1 -- might be a tad rude!) When people stop to think about it, they realize that it's true. I don't need people thinking I'm a hero or that my children are angels. If my kids were in school full time, I imagine I'd be just as glad as every other parent to get back to the routine after spring break or whatever.