Also maybe add to the invites, or even a secondary to those who have not responed:
"Hello, "whomever" Its so very exciting to "name of your child" to know who will be at his/her party, that even if you even think you may not be able to make that date, to please let me know, as I would feel so bad if he/she thinks that your child may be coming, then to find out on his/her big day, that you're not here! If you cannot or do not want to make the date for any reason, we are not offended in any way, but it means a great deal for us to be able to tell him/her. So, again, please just drop a quick text, email or note that you will not or will be attending. Thanks again and hope to see you on (whichever date)!. All the best for your family!"
I have no problem responding with rsvp's, but it seems like many people just don't rsvp... sorry, its a fact that many don't bother.
Hope he and also you, have a great bday party for him!