I always felt safer using the base, though for a quick trip in someone else's car I did go without once (rather than switch because I was in a hurry).
Though a friend of mine had a older car seat where the base wasn't available and always used her's without.
To me personally, it always looked more sturdy with it, but those infant car seats never did feel all too sturdy to me anyways. Once I switched to the older seat (still rear facing until over a year and 20lbs though) which has a belt on the back of the seat that clips underneath the front of the back seat which holds that seat REALLY in place...and I mean, I can take that seat and shake it like a tornado is striking the car (without baby in it obviously) and the damn thing won't move, heh. But to me the older seat doesn't look comfortable enough for an infant and I'm still going to use the infant seat with #2 until about 7/8 months (like I did with Morgan).
Uhhh...sorry for the long winded response.