I think if I wanted something that wouldn't fray and was soft enough for a hankie, I'd use old t-shirts or the like. Soft cotton knit.
You could use flannel (which is the material I prefer for hankies) and cut the edges with pinking shears, but you'll still get some fraying. You'd have to trim them and remove loose threads for quite a few washes, but it would work. Old flannel pjs or receiving blankets would be a good source for the flannel.