We do.
We live in a small college town, about 5,000 residents, in a rural area. We live right downtown, old house on a small lot. While definately not "city" it is a busy community.
Our second house is about an hour away in a state forest in an area of seaonal homes and cabins.
I remember feeling really nervous like what in the heck are we doing??!?!?!
To answer your questions.
If so, do you feel like you get to spend enough time in the weekend place, or do other things get in the way?
We spend almost every weekend at our place. Unless there is a specific reason for us to stay home, we go.
Not to sound flip, but I dont' let other things get in the way. Our son is young enough that he isn't in organized sports. When it comes to weekend activities, b-day parties and such, if it isn't a super close friend, we decline the invite so we can go to the cabin.
Do you find it worth it?
Absolutely worth it. We have made new friends in the community, we do tons of outdoor activities, camp fires at night, hiking, swimming, etc.
Any drawbacks that you didn't expect?
For me, it requires a hightened level of organization. For example, last night when we got home, I immediately started the laundry. I did the cabin stuff - sheets, towels - first. These items will go right back into dedicated tote bags so all is ready to go next time we visit. It keeps everything more organized and makes packing to leave easy.
Because we are miles (1 hour round trip) from a store, I need to plan food and meals prior to arriving. I have a standard packing list and I add ingredients as needed.
I purposely kept our cabin as simple as possible. It is easy to keep clean and organized.
It took us a little while to adjust to the lifestyle of nothing to do. After the intial excitement and work settled down, we were like "now what?" That is when I pulled out the trail maps, go a bike carrier, started planning for outdoor and nature activities.
I thought we would have all sorts of friends and family using our place but that hasn't happened yet. My aunt and her family wanted to use it last summer but then when I said the dog was not welcome, that put a damper on their desire to visit.

So far, no one else has taken us up on the offer but I think they will in the future.
Our cabin life is very different from our home life and we like it that way. No tv, no ac, no laundry to do (we don't have a washer/dryer) so when we are there, we are really
together as a family.
My one suggestion for you to consider -Are both of you on the same page when it comes to how you are going to spend your time when visiting? I could see that being a potential source of conflict that might prevent you from visiting your place.
For example, if I was the type that wanted to spend my weekend shopping, I might not be too thrilled to be in the middle of no where. Or if watching a game on TV was super important to my DH, he likely would not want to spend a weekend with no tv.