I've always only heard it for takeoff/landing (or if the fasten seatbelt sign is on, maybe) also.
I think there's a few reasons for this rule:
1) They've simulated crashes with babies in carseats and with babies in arms, but not with babies in every random carrier type on the market. They have NO IDEA what might happen, and while it might be better than in-arms, it also might be worse. It's a risk they can't take.
2) US tests of the belly belts that are popular on some European carriers show that lap children secured with them are more likely to be injured than those in-arms. It's not the same as a carrier, but it's the most similar circumstance that's actually been tested.
3) It's entirely possible that the carrier would dramatically worsen the nature of injuries. Your arms will give a LOT more in a crash; in an Ergo, for example, baby's head is free to flop about, but the body of the carrier will keep their torso with yours. That may result in much greater head excursion than the situation where their body *and* head will move a bit more naturally along with your arms.
4) FAA regulations prohibit it; they MAY NOT allow you to use a carrier during takeoff/landing, or risk their licensure. So if you don't like it, you have to take it up with the FAA... the flight attendant is in no position to negotiate it.