The reasoning is that infants and toddlers need direct interaction with caregivers to properly develop socially/emotionally/cognitively. Also, people tend to use it as a babysitter, and may rely on it more than they realize. See the AAP's website for their policy statements regarding TV/ media. Most of them are directed at older kids' viewing, talking about desensitization to violence or sexual themes, lack of moral values, and advertising, but there is one statement about viewing for kids under 2.
www.aap.org