I've had to deal with grandparents rights (I'm in BC Canada) My case was quite different from yours because I'm actually okay with ex's parents seeing the kids, they just took issue with not being able to dictate when, where and for how long they visited the kids. They hadn't seen the kids in over a year because they didn't contact me after XH was charged with child assault. When they did make contact, it was in the form of a card, directly to the kids, informing them they'd pick them up at 9AM on Easter Sunday, and they'd have them back 'by nightfall'!! The kids were 3 and 6 at that time, so I just didn't read it to them, and called XH's parents to set up a more appropriate kind of visit. They met the kids in the park near our home for a few hours. I meant for the visits to work into them having unsupervised access, but the way things progressed from there made me more and more aware that they had zero respect for me. I thought that if I let them have the kids alone, they'd let XH be at the visits (there was a no-contact order in place)
It only took 2 or 3 months before they started threatening me with court. They started doing nasty things like giving the kids bikes at the start of a visit and then at the end took them away and gave me a note, saying the bikes are for use at their home only (by this time I didn't intend to let them have unsupervised access anytime soon) Eventually, we had a family case conference. The agreement we came to was 2 months of 4-hour supervised visits, 2 months of 4 hour unsupervised visits close to my home, and then 2 months of 6 hour visits (2 visits per month) They were required to visit the kids counselor to discuss how to appropriately interact with the kids, and it was made clear that if XH were to attend the visits he would get in very big trouble. Ex MIL made a comment about being concerned about the kids 'health and well-being' (they bough the house XH and I owned... it was pretty mess when we left, because I had broken my arm and was doing the bare minimum for 2 months before the move... moving 2 weeks after you get your cast off = not fun! I wasn't going to extra effort to get it spic and span for her, so calling CPS was her revenge. CPS never came to visit, that's how seriously they took her complaint)
The access visits went okay for a short time... MIL would wonder aloud as they approached the door whether I was home or not (great thing to say to a kid with separation anxiety!) Then they started dropping the kids off in the neighbours yard and I'd find out they were home when I heard their voices! Eventually I made the decision to reduce the visits to once per month, for 6 hours because the kids were missing out on seeing my family, and on birthday parties with friends, that sort of thing. I offered to let them take the kids for dinner on a weeknight, but they declined. We went back to court and had a trial that time. In the end, they got 6 hours, once a month and my lawyer says that's only because it's what I wanted anyway! XH never attended any of the court dates.
There are a ton more details obviously, but thats the extremely condensed version. XH's parents can be emotionally abusive... I've been asked things like "Do you wish you never had kids?" and told "Daddy is nice now!!" (which leads to DD emailing him, and him not responding to them) but I'd rather the kids grow up with their eyes wide open about the grandparents, because they will have to create their own boundaries with them when they're older. DS is already wise to when they're attempting to manipulate him, and he resents it. If/when they decide they don't want to go on visits anymore (when they hit their teens, I suspect) we'll probably end up back in court.