I'm not sure that SVS would let you stay with him all day? I would check on that, though, because that might be the ticket to helping him feel comfortable.
You could also see if they have any get-togethers that you could go to, information sessions or parent meetings or some way to get to know other families. I'd talk with the staff and see if they have any ideas or suggestions.
Have you talked to him about it? Ask him if he has any ideas on how to make things more comfortable for him. Maybe you could draw a picture on his arm so he could look at it or kiss it when he got nervous, or maybe a piece of paper or stone or small stuffed animal for his pocket. What about if you made an agreement for him to try it for a certain length of time, say a week, with the option of him calling you if he gets upset, or going home after x amount of time there? Also, you could try reframing things for him...4 hours on a rock isn't so bad. Little steps are ok, he did a brave thing by just trying it for those 4 hours, right? He is now already pretty familiar with the rock, eh? So he is comfortable with one place! Maybe he could try bringing a book or a notepad for drawing or a Nintendo DS and just sitting quietly on a couch for a couple of hours. My experience at my kids' school is that it's perfectly fine to ignore everyone if you want/need to. In fact, in one of the SVS books I read I remember there was a girl who said she just sat in the sewing room and didn't say anything for a long time (as in weeks/months, I wish I could remember more details) until she was comfortable and wanted to engage. I think it's important for your ds to know that he isn't weird for being afraid/nervous (I'm sure you tell him that already, I'm just thinking out loud!), and that Sudbury really is different from his regular school. OH! Just had a great idea! What about if you borrowed/bought a copy of the documentary Danny Mydlack made to help familiarize him with the environment? It's a different school, but I think it could still help. Link again: Voices from the New American Schoolhouse
When my ds had a hard time with separation anxiety (made worse by the fact that I was working and had no choice but to leave) there was a certain staff member that he could be comforted by--she had a calm, matter of fact attitude and he could go to her when he was feeling overwhelmed. If there is someone (staff or student) with the time/interest in getting to know your ds, that might help him.
Good luck, and I'd love to hear more about what happens! Keep us updated!