I've been camping with my two since the oldest was ~6mos. Their Dad isn't a camping kind of guy, so it was mostly car camping then, but once it was just me and the kids we'd do both car camping and backpacking. They laugh when they "camp" with their Dad, his wife and her kids - KOA all the way!
I have never found camping as the sole adult with children a safety concern. *Could* something happen? Sure - just as it could doing any other activity as a sole adult with children.
The one thing you cannot ever have enough of is socks. Other clothes? You can make do.
A ground sheet is imperative. I like to use one of the blue tarps for car camping, something less bulky and lighter for backpacking.
When they were younger, I made a habit of prepping the first dinner at home - sauce for pasta, stew, something of that ilk so dinner was easy once we got the tent set up, our gear stowed, etc. Also... the reason so many campers with kids get stuck with hot dogs over the fire? The kids LIKE it. It's different from the norm, it's fun, it feels like camping. And they taste darned good cooked over an open fire. That and marshmallows. So I always planned on one meal of dogs on a stick. When we backpack, that's usually the first night's meal.
Even when they're little, if they can walk? They can help set-up and break camp. The only time mine sat in the car while I was setting up camp was when the heavens opened and it poured as soon as we got there - and wasn't about to stop soon. I figured one of us soaking wet was better than all three of us! LOL That was the first time I took them on my own.By the time they were 8 & 10, they could put the tent up on their own, stow the gear that went in it, etc. So I could get the cooking area set, get the fire laid, and so on. Get them involved early - it may take a bit longer at first, but will serve you well in the long run!
As you gain more experience, you'll be able to pare down the extra stuff you need. Use the large plastic totes to store your gear. Make a list of everything as you pack, put it in a ziploc bag, and tape it (with a pen) to the inside of the lid. Towards the end of the trip, cross off what you didn't use, and otherwise mark what you used a lot and what you used, but could have likely left behind. When you get home, take out everything you didn't use, and decide what you used but could have done without - take that out, too. Clean the rest, and pack it back into the tote. Replace anything you need to (refill detergent, matches, sterno, fire starters, etc.). Lay out your tent, let it dry (if you leave in the morning, it will be damp from the dew), re-roll it. Shake and air out sleeping bags. Re-roll those, too. Etc. You'll be ready to go the next time w/o as much prep.
I keep our backpacks ready-to go, so all we need do is add the clothes we're taking and food - and we can be off in a very short amount of time. Whichever camping we're doing.