An SF-based blogger I read sometimes has a little category called Sites and Bites where she features a place to visit in SF, plus a restaurant nearby. Maybe you'll get a few ideas there? Here's a link: http://sayyestohoboken.com/category/sitesandbites
It's funny how hard it is to think of places when you're on the spot. I think if I had a quiet morning or afternoon, I'd first take a long walk on Ocean Beach, then cuddle up under a blanket by the fire at Park Chalet for some fries. From there, you can explore Golden Gate Park (it's so big, and there's so much to see, a whole day or more would be best if you really want to explore the park). The Botanical Gardens, the de Young museum, Academy of Sciences. You can come out of the park at the entrance to Haight street, if you want to walk and shop Haight-Ashbury district (it's a looong way from one end to the other, just so you know).
A small, romantic restaurant we like is Firefly in Noe Valley (fireflyrestaurant.com). 24th St has a little shopping too, no tourists.
My teen daughter really loves Japantown. It's less touristy than Chinatown, with some good little restaurants and Japanese imports.
I hear a little one calling, so I have to go, but if I think of anything else I'll come back later. Oh, and if you are thinking of drifting up north to Marin, I can give more recommendations -- that's our neck of the woods.