[get ready for a long response...]
This is such a challenging topic. We struggle with it too. Coming from an upbringing where the parenting was coercive ("We the parents know best, so these are the things you must do, so do them, or else"), but instead desiring a parenting style where the CHILD internalizes these habits is so, so, hard!
My husband is an example. He grew up in a very neat house with a very neat mother. Unfortunately I think that she did all the cleaning up herself and required very little of him. I think you can guess the result. So I wonder, how can I do better with my own son?
This question comes up over and over again, not just with cleaning up, but with any "good habit," such as washing face, teeth brushing, practicing things that are hard to do. (we are homeschoolers that TEND toward unschooling but would be better described as eclectic)
With cleaning up, my son's room has always been a disaster area and I do tend to think that he gets overwhelmed by it. Ever since he was little, though, his room has been somewhat organized so that at least when I picked things up we would know where to find them later. Labeled bins and shelves. "Where does this go?" He'd ask. "It goes in the Trucks Department," I'd say. Or "the Space Toys Department" or "The Pajamas crate or the Sock Bucket" or whatever. LOL So he knows there's an underlying system. He sees how hard it is to find toys when he has not put them in their place. He sees me being real happy when I find things that I need, because I had put them in the right place. So in that sense I've been modeling. I also talk about how happy I am when I clean up the living room or how proud I am of myself when I've worked all day on my bedroom because the lack of clutter makes me feel so peaceful when I'm done. He can see that there is an effect.
I tend to think that nagging and forcing will make him turn away from the "good habits" in the long run. It was like this with me. I was forced and coerced all my life. So when I got older and left home, I felt a sense of rebellion against these parental figures. I would let my dental health go, or I'd let my place get messy, or my finances…..and in my head I would be hearing the parental voices telling me to do this or that, and I was conscious that my not doing them was a defiance of these parents..even though I was now an alleged adult and on my own. But the truth is, it was MYSELF I was hurting by neglecting these things!! So it always brings me back to the fact that my son must not be cleaning up because of me, or brushing his teeth because of me…..it is HE who must be convinced that these things are a good idea or it will never stick.
That being said, you may set limits on what YOU will do. For example, after an initial organization of their rooms, labeling totes and bins with them and basically providing them with an infrastructure and some clues on how to stay organized, you can then just refuse to help them clean up their rooms. If they start to look like something out of the show "Hoarders," then you can also go on strike in other ways. When they lose toys or can't find things, you don't help them find the things. Not out of spite, but because you're modeling that this is their responsibility, and you've really tried to help them by organizing their room and giving them good tips on maintaining it, but really after that point it's not your job. You have plenty of "mom things" (or "grownup things") to do! I often find myself saying that to my son. "When you were a little baby I did all these things for you because you could not do them. But now you are growing up and you are very competent. These are your jobs now. I am here to help within reason, but they're yours to do."
This story is a little bit of a tangent, but it speaks to kids internalizing what they need to do:
Last night my son and I were talking (he is 7, almost 8). It was 12:30 a.m. and he asked "Can I build a superhero in bed?" Now I know from experience that if I say No he's going to do it anyway. But that's because saying No would be completely nonsensical. The truth is, I want HIM to be able to judge. He can ask my opinion, of course. I said "Why do you ask me whether you can do this? You know that if I say No you are going to do it anyway. I think that's really disrespectful to do. I'd rather you just ask my opinion, and I will tell you 'I think that you need your sleep and at 12:30 a.m. it's not a good idea to start building superheroes in bed.' " He started to tell me that it's a battle between his Wants and his Needs. That his Needs (for sleep) were real weak, like Native Americans with bows & arrows, and that his Wants (to play all night) are strong, like modern-day soldiers with machine guns. That was his analogy, so I hope nobody is offended by that, but I thought it was a vivid one, and very self-aware for his age. Well I said good luck with that! He seemed distressed and did not want to end the night on that note. So I said, why don't your Wants and Needs work out a deal? You can play for a little bit, but then turn out the lights and go to sleep. And that's what he did. Today he had to go to Sunday school pretty darn tired. But he will learn over time…."When I stay up late, I feel dog-tired the next day." And a better habit will be born. I just have to be patient.
Back to the topic of room-cleaning. Sometimes I just see that my son is overwhelmed and I have to step in. I will say "I'm not going to do X (fun thing) with you until you clean off your desk. Your desk can't be used right now, it's so messy. And you have lots of projects you enjoy doing; you need a space to do them" And I point to a checklist I made for him and had stuck on his wall. The checklist is like so:
First, look for
Toys (and put away any toys you see in the pile)
then Art Supplies
(by going one thing at a time it helps him be less overwhelmed)
The point is, we're not on opposing teams here….I'm not disappointed or mad that he needs this help, so I just give him the pointers and let him do it. But if it is something that's gone on too long and really needs cleaning, I just put the Fun Activity With Mama right on the other side of it. He has to get through the cleanup to get to the activity. That's life.
Hope that helps. I really ought to write a book, for all the brevity I don't achieve here on this forum.