That felt is plastic so it's not really better than plastic (the same is true of poly stuffing) Price is the top reason to compromise and have plastic toys of whatever sort, so it's the same old dilemma. But you are going to a lot of effort to make something nice by hand and that time and energy hardly seems worth it if the material is still a plastic. I try to go with fewer toys, or fewer total sheets of felt, and still get the nicer natural materials.
Or learn to make toys from cheap cotton fabrics--it won't act like felt in that you need to turn it to put seam allowances inside but you can still make nice things with some sewing skills. (I'd stuff with wool or cotton then!)
I'd skip the sweater-recycling idea if I were you. The cost of a few sweaters at thrift stores is generally more than a few sheets of new wool felt and the felting, handling, thickness will be unpredictable and the color choices very few.