You definitely should attend the festivals -- harvest, May, Christmas. No one does festivals better. Also, the kindergarten is lovely and beautiful. My daughter goes there and we love her teacher and the community associated with the place. My son was in teh same kindergarten five years ago.
Before you change your life, though, think about where you'd like your kids to be in a few years and when they finish school. HV may not work out and you'd be stuck here. You'd be left with the public schools or private schools far away or homeschooling. In Woodstock or Berkshire County MA the public schools are likely better.