I suppose it depends on the child and his/her relationship with birth mom, but yes, I do think it becomes more difficult for the child as he/she gets older. My daughter is 5, and sees her birth mom once, maybe twice, a year. Her most recent visit was, as they often are, out-of-the-blue, impromptu, rushed, and a little dramatic. Birth mom announced she was leaving her long-term relationship, and setting off for another country, and, she said, "And maybe I just won't come back." While this might not have gone noticed by my daughter at 2 or 3, at 5 it definitely does. After the visit, my daughter asked, "Will she really not ever come back? Will I ever see her again?"
I admit to feelings of anger and frustration sometimes, but ultimately I know that: We love and care for her birth mom, even though we don't always agree with her choices; My daughter must ultimately define her own relationship with her birth mom as the years pass; My daughter has a solid foundation with us, her adoptive parents, and we must be open to helping her process her feelings about her birth family because those feelings *will* surface (and have surfaced), and all we can do is be committed to supporting her in those feelings.
I don't know if that helps at all, but I do sympathize with your feelings. It is one of the hardest things about being in an open adoption--for me, anyway. Navigating the ups and downs of birth family relationships. It can be very much out of our control, which can certainly be unsettling.