I don't know if age alone is a big issue for reading the book. However, I think that the book needs some serious context in order to be as powerful as it should be. When I read the book, I was in sixth grade, I think, and we read it out of the blue with no history to surround it. My classmates and I thought it was horrifically dull and lame.
When I was teaching, we devoted several months to a multi-subject unit about WWII and the Holocaust. Our kids (who were in eighth grade) were so aware of what was going on, and especially the consequences Anne's family would face if they were caught, that the book had a lot more meaning.
Eighth grade was a good age, I think, for kids to be able to handle information about the Holocaust that they needed to put the book into context. I might go a bit younger with kids who were not overly sensitive.