I came to unschooling from a philosophical space, before my kids started school. You're coming to it from a practical space and while you might find resonance in the unschooling philosophy, it's probably best to continue along the practical path for the time being. By which I mean, notice what works, and do more of that! So as to "where to start," I would say to start with your ds and with what he enjoys doing on a typical Saturday kind of day.
And really, it sounds like you're already doing that! The things you describe yourselves as doing since you "let your ds be" sound great.
As to having a rhythm to your days, you may find yourselves gravitating to a natural rhythm over time, and that's cool. If not, there may be good reasons to be somewhat intentional about creating a rhythm. Perhaps your ds appreciates knowing what to expect from each day and is happier that way. Perhaps a rhythm ensures that you and he both get the exercise and social time that you thrive on. Perhaps a rhythm ensures that you are able to consistently make yourself available to support and facilitate the things he wants to do. Perhaps a rhythm reassures him that he is having productive days.
On the other hand I'm not sure that reassuring *you* that *he* is being productive is enough of a reason by itself to add structure to his home learning. If he seems to thrive with things very loose and unstructured for the time being I think there are other ways to reassure yourself that he's learning well.
First: be a sensitive observer. I don't think an activity that really engages a child is ever devoid of important learning. An example: my eldest dd used to read books over and over and over again. It seemed so pointless to me. She was a great reader. She had intricate details of these stories memorized. What could she possibly get out of reading Harry Potter or The Hobbit for the ninth time? Well, it turned out that she was internalizing the poetry, the phrasing, the rhythm, the music, of well-written and not-so-well-written prose. At the time she didn't know why it engaged her, and neither did I at first. But I kept telling myself: she wouldn't be doing this if she wasn't getting something important out of it. And I kept asking what it could possibly be. I noticed that she sometimes read (quietly) aloud to herself -- which was much slower and not her normal way to read -- and I began to suspect that it was something about the way the prose flowed and the craftsmanship of the writing that she was internalizing.
Second: feast upon the experiences of other homeschooled families. When you have in your memory bank a harvest of reassuring outcomes from unstructured learning, you will be able to make peace more easily with the lack of structure in your own home.
Third: document your child's natural learning. Some people keep binders, or scrapbooks, or photo-journals, or just little notebooks, or digital versions of these. I've been keeping a general homeschooling blog for many years. I now also keep blogs for each of my kids' home-based learning. My general blog is public, and every once in a while I realize that people are getting a weirdly distorted view of our homeschooling as this über-productive, exceptionally creative, high-achieving, never-faltering endeavour. Which it's not: we have our crappy periods, and our ho-hum days and a lot of pretty boring stuff too. But the blog is serving its primary purpose which is to point out to me that among the daily ho-hum stuff there is a fair bit of impressive learning going on. And that gives me some peace of mind when I start to get anxious about needing more structure, more curriculum, clearer expectations. I can browse back through what I've documented and think to myself "Wow, he's actually doing a lot!"
So I guess in a nutshell I'd suggest starting with what's already working, and gently encouraging a bit more of the things that really seem to engage him. And to work towards a gentle daily rhythm if it seems to be something that benefits your ds. And above and beyond that to simply notice what he's learning and keep reassuring yourself through a variety of means of all the good that is coming, and can come out of natural learning. Think of it as a deschooling process that you may continue with if it's still serving him well in a few months, or perhaps not if it's not, but it's necessary for now as he detoxes from his school experience. Keep your feet beneath you, and your eyes on the horizon: work with what you're given today, for now, but remember that you have a really long time-line for getting "results." You no doubt want your ds to grow up to be a happy, capable and productive adult. There are a billion paths to that outcome.