Need to know more to answer fully, but if he has court-ordered visitation than I think you might be held in contempt of court not to give him the visit...
Mom to two terrific kiddos, affirming every day that the Universe is unfolding as it should and all is well...
Yes you have to allow the visitation. Denying visitation, or requiring it to be supervised, when he has court ordered visitation, is very bad. Don't do it. Unless there is imminent danger to the children, and you can prove it with hard and fast evidence.