We did it, not in an RV though, on a boat, with young kids (infants and toddler) off and on for five years. So similar space-wise and also nomadic. It WAS a lot of work but so well worth it. Kids that young don't have to be tied down to one spot. They don't need school, preschool, sports teams, lessons etc at that age that you need to be home for. We would visit the local libraries in the various communities we ended up in, the kids would partake in storytimes and such, as well as the museums, we could often find free kids programs going on that the kids enjoyed. Plus there's playgrounds, the Y, parent-tot groups, etc, so lots of opportunity for socialization (which was the ONE thing people would ask me everytime - "don't your kids need to socialize???")
I wouldn't even worry about the kids, seriously, they're adaptable.
I wasn't clear - are you planning on selling your home? If so is that absolutely nessessary to this venture? Another option is to rent it to cover your mortgage, taxes and some basic upkeep.
Also, our marriage was heading down the toilet fast when we made our final descision to do this. Getting away from the norm and being in control of our own lives helped so much. We set our own schedule, worked together, problem solved together, helped each other with the kids, talked a LOT, (and argued too ) and learned how to have fun again.
When we were done (for the time being) we sold the house, built another one, started a business, put the kids in school and so on. It was tight for a while but it's getting much better now. And the knowledge and worldliness that our kids gained was incredible, it was such an education for them : )
Best luck with whatever you do !