These things happen from time to time.
I often got threatened with cps or police trigger happy people when my 3yo wouldnt put on his coat in winter. (hes now 5 and doesnt mind his coat) Once i was literally stopped in my tracks by these two women who claimed to be something or other, laid their hands on my stroller and wouldnt let me pass (because i had raised my voice at that same, then 2yo, who kept throwing his coat out of the stroller-he had a thing with coats i guess) I threatened to call the police on them, and kids started crying...
As irritating as it is, there are people out there with nothing better to do, than to invent hysteric tales in their minds, and particularly aim their wrath at mothers of young children. Once thing i miss about my prechild days, is that privacy most people have but mothers of young children do not, because young children attract attention. I have had to perfect the art of polite retort to the cascade of comments both positive and negative from random strangers.
On a positive note, i have discovered most people are good and caring people. They will lend a hand if needed, or simply say 'bless your beautiful children', or tell you you are supermom because you are baby wearing. Many people compliment me when they see me baby wearing, especially when i doned my babywearing coat in winter. Same for breastfeeding.
Im sorry you had to endure this....it is really scary