I was on 50 mg Clomid for 5 cycles, and only once did I produce two mature follicles. The time I did was when I finally got pregnant with a singleton. Your experience may be different, of course.
On two cycles with 5mg letrozole/Femara, I had one mature and one borderline follicle on the first cycle. Got pregnant and miscarried before I could see whether it was one or two. Assuming one, based on beta.
On the 2nd, had two mature follicles and got pregnant. Still waiting to see how that will turn out.
I feel the same as you about wanting a singleton, but I think you get less risk averse as you go along in the process. I'm not saying that's necessarily a good thing