We each have our own towel, but it's no different from other towels...
We do not have "special" towels that only one person can use, but we also don't share towels when they are in use.
We wash the towels and get a new one, but it's not like we have specific towels that only one person can use after it's clean.
So long as I don't have to towel off with his wet from just used towel, it makes no difference.
Now, that being said, I do have two towels that I use most often and never he. But that's only because I've had them for years and they are those "body sheet" sized towel, I like to wrap myself in them and I've got more "bulk" than he, so he uses the "smaller normal" towels. But if he wanted to use one of my "special" towels it wouldn't make any difference to me so long as he has one and we don't have to share at the same time.