Could it be that some of you who are hesitant to speak (or your partner who is hesitant to speak) are self conscious? My DH speaks korean, but pretty much only exclusively with family member and only when necessary when going to the korean store or restaurant. He is very self conscious about his korean. He immigrated to the US from south korea when he was 7 yrs old and feels that his korean is at the skill of, "baby" korean. I have encouraged him to speak to our kids in korean and he seems to forget to do it. Usually, he will just teach them what x is called in korean. So, they are picking up some vocab, but don't hear conversational korean.
As for me, my parents spoke both taiwanese and mandarin at home. I learned mandarin, was sent to school not knowing any english at home. My parents freaked out and wanted us to only speak english at home (while they spoke mandarin to us). Then around jr high aged, my parents suddenly decided they wanted us to speak back to them in mandarin, but by then, we weren't used to it and just couldn't get into that mode. So, while I understand mandarin and taiwanese, I speak HORRIBLE mandarin and taiwanese (and they even sent me to chinese school on sundays, which was a torture and a waste of time). I regret that my parents did not have my siblings and I speak mandarin at home.
So, my question to all of you who want your spouse to speak another language with your child. Do you expect your child to speak back to your spouse in that language? If you don't make them do that, they will end up like me, and understand the language, but their speaking skills will not be very good.