I see two separate issues here....
Issue #1: The love, safety and well being of the 3 year old. OP, I think you are handling this very very well. I think that keeping them separate from each other is the only thing which makes sense, considering how highly inapropriate this incident was...I think, mama, that your son really is going to be okay. For HIS age, this is really normal play (maybe beside the specific nature of it, I think a three year old is more likely t be interested in "touching pee pee" than "kissing pee pee") and TBH, if I came across two three year olds engaged in silly "pee pee play" I would redirect their play, but I wouldn't freak about it. So, what I'm getting at, is that from YOUR childs standpoint, this probably felt really normal and silly and not at all scary and it will probably become a mere vague memory and over time, slip away into his vault of forgotten childhood play. So...I know this must suck so badly for you...but I think your dear son will be okay, I really don't think he'll even remember it...
Issue #2: The love safety and well being of your nephew.
He is a child. What he did was so wrong on so many levels and for so many different reasons and I realize that for that reason it must be so hurtful...you love this boy and trusted him, you've watched him steadily improve over the years of patient and loving care and dedicated parenting from your sister and BIL....and now this. It must seem like a really huge step back. But remember where he comes from. He has a terrible background. I have three adopted family members who were born addicted to crack and while, after many many years of dedicated, loving parenting at the hands of two adults very skilled with high/special needs parenting...these kids are doing great, they will never be "normal" people. They struggle with urges and social ackwardness which will never be over for them, can never be "taught" or "loved" away. They are different. From a chemical level...they are different.
What my grandparents went through with these kids...it's just, unimaginable. At one point, when he was around 10/11, the boy (my uncle) actually burned down the family home...just had an irresistable urge to play with and be around fire...and it cost them their home and everything in it. This DEVASTATED my family....it was so hard...but this is your kid. You grieve, you process, you move on. What else can you do?
Children are highly sensitive and reactive to "YOU messages". I would highly caution your sister against sending the message to him: "YOU are a monster, YOU are a waste, YOU are an extreme disappointment". Again, OP, I think that you have to handle him differently, because you have your own youngling to protect, your obligation is to your son. But your sisters obligation has to be to HER son....I know she wouldnt outright say to his face "YOU are a monster and we hate you" - but a "you message" doesn't have to be a directly spoken statement. It can be in way they change toward him, the level of commitment they show him....and I'm just afraid, that if he is really trying to graple (at his young age) with intense urges and overwhelming thoughts that he knows are wrong....I just worry that sending te message to him that he is wrong, ugly, monsterous, could cause those statements to manifest into truths in his life.
He did something wrong....but HE is not wrong. HE is a kid who is really messed up inside and he needs a lot of help. Everything your sister has done for him is NOT a waste, he is just entering into a more difficult phase of life...he is hitting puberty...that is hard for kids without the problems he has...but for a kid who DOES have his sort of issues, I'm telling you...I wouldn't wish that pain and ackwardness on anyone....that is so much to come through.
He already feels different, he is old enough now to see that people think he is odd and that they dislike him and he can tell now, that your sister is beyond devasted and beyond angry....all I ask is that you guys separate the issues...I know that's hard, but you have to do it, because there are TWO little boys here who need love, caring and compassion here....
Your son is going to be okay. He didn't do anything wrong and I really would anticipate, judging his reaction, that he will forget about it completely, as a lot of kids forget about those first exploratory experiences. It's your nephew who, in this situation, is at real danger for heading down a path in his life that not many people come back from. In order to keep him from that, his parents will have to keep him connected, to prevent him from turning deeper into himself out of shame and confusion...if he turns into himself and starts living inside his head, which sounds to be a confusing and maybe harmful place right now....you may not be able to reach him again on an emotional level.
SHame is not good for the victim...but in this case, your nephew is also a victim of sorts...and he stands to lose even more than you could imagine if he is shamed, hated and emotionally cut off by family members.
My advice to you....love your kid, which you are obviously doing.
My advice to your sister...seek help. She cant go through this without a strong guidance from someone who can help her process.....she needs to love her kid too, she can't give up on him. He was born ot a mother who didn't care...the whole world could've given up on him...but look....it didn't....and your sister came, like an angel from the heavens, to scoop him up and show him a real life....he's got a shot at being something great, at giving and recieving a kind of love even his own mother couldn't give him....please, don't you all give up on this boy. He's just a little boy. 11, yes....but in his heart and mind, based upon what I know of children with these types of backgrounds...he is probably not really even an 11 year old. He is a messed up person inside and he deserves the continued love and support of the family who promised to always give him that, when they took him in, angry and confused and "damaged" as he was.
GL to you all...I wish you so much peace and I'm sorry this has happened....you are doing such a great job mama...honestly, you really sound like you're keeping it together with your kiddo and I commend you for it...