I think that 3 yr old s are unpredictable also, and they can get a Little wild. I don't thik that the majority of what you are describing is "normal" however. This child is acting out at a very early age to me. He has proven that he cannot, CANNOT be left alone with the baby. He lacks the empathy skills to be trusted with a baby. I, personally, wouldn't watch this child again. What would you do if this was the nice lday down the street's kid? This is a clear test, IMHO, and it's one that you need to ppay extra close attention to this DC when he is around the baby.
Should you decide it's worth it to watch this child, I agreee with the PP about asking for more money. Just because they are family, doesn't give them the right to walk all over you. I would also lay out your rules and expectations in writing, and ask them to sign a contract. Then you have the hard job of teaching Dear Nephew *DN* to follow the rules at YOUR house. At three he Should be able to understand what you are saying and what you want, if you keep things simple. If you make your line in the sand, then you need to follow through with some very real consequences. We are talking about someone else's child, and that child needs to learn the rules of YOUR house. I am all for GD, but when a child is in danger, or totally out of controll theres needs to be some serious action, and I'm not talking about spanking because that's just more violence.
I hope you get some ideas that are more GD, because I am at a loss at the moment. IMHO you need to have a sit down with your SIL and BIL and figure out what you are going to do. Tell them that this is totally unexceptable, and that just because you are a SAHM does not mean you run a daycare. Tell them that you are OK, or not OK with watching DN, but that you are having problems with him. I wish you the best of luck, and from personal experience I hope you keep a close eye on your baby. I had my 1st injured by an out of controll 3yrold.
ETA: just read your update, and YEAH for you for getting things a little better! Glad that mom's changing things on her end, and poor baby about the pinworms! Glad that he was checked out by the doc. I hope things continue to improve for you and DN!