As a mom of two 35 weekers who spent 10 days in the nicu this season, I understand your paranoia. I had to take my DD to the pediatrician's sick waiting room while her sisters were in the nicu. When I talked to one of the nicu nurses about it, she told me that there were 145 babies there, and not one of them had flu/rsv despite being in a children's hospital, one of the germiest places ever. The handwashing system is very effective, and if you are anything like me, you are probably bathing in the hand gel that is everywhere. It's really unlikely that you passed anything to him. Are you still wearing a gown and gloves when you see him? If so, that's even more protection.
Get some sleep, it's a hard place to be and there are lots of things to worry about.