I think if I wanted something that wouldn't fray and was soft enough for a hankie, I'd use old t-shirts or the like. Soft cotton knit.
You could use flannel (which is the material I prefer for hankies) and cut the edges with pinking shears, but you'll still get some fraying. You'd have to trim them and remove loose threads for quite a few washes, but it would work. Old flannel pjs or receiving blankets would be a good source for the flannel.
I have used the old t shirts as well as the flannel. I didn't even use pinking sheers on them and they didn't fray that bad. I would pull a wad of threads off as it accumulated but then before you know it they have stopped fraying. I say try a few and see what you think.
~Katie~ married to J, mom to DD- A 13 yrs ,DS- L 7yrs , and my little nursling DD2- R 5yrs.