My daughters started reading the series after I became hooked and they watched me reading them. I think they were in 2nd and 3rd grade at the time? It was right after the 5th book came out (that's when I purchased them, prior to that I'd read them from the library.)
I recently leant my copy of the first book to a friend to read aloud to her DS, the summer before he started 2nd grade. Her 4yo DD listened as well, which surprised her, but I'm not sure how much the 4yo really absorbed from it.
So the answer really depends on your particular child and his/her interest and level of comprehension. It does have a couple of scary parts- it opens with a double murder (of Harry's parents), then Harry is neglected by his aunt and uncle who are "caring" for him. Well, they don't actually show the Potters being killed, but you hear about it- and the thought of a 1yo suddenly being orphaned could be very frightening to some children, especially when coupled with his poor treatment by his foster family.
Later in the book you meet a mountain troll and a giant 3 headed dog, and during the "final showdown", Lord Voldemort's face is sticking out of the back of somebody else's head. That's a pretty freaky image- possibly the scariest image in the entire series.
Some children are going to find this story much too frightening before age 9 or 10. Other children won't have a problem with it- in general, things won't "look" any scarier to a child than he or she can imagine it, while images in movies can be a LOT more frightening than the same thing in a book.
