Dh said one thing to me about 4-5 years ago in the heat of the moment and completely apologized for it later. I let it go because I know he was sincere in his apology.
However, one time he accused me of lying to him about something. I don't recall EVER being so angry or hurt. Being accused of lying is NOT something I take lightly. I would NEVER lie to my husband about ANYTHING! I'm not a good liar in general. Well, this arguement went on for 3 days. I was not about to give in (like usual) and he was counting on it.
There is a bible verse Matthew 7:25 that reads: And the rain came down, the streams rose and the wind blew and beat against that house yet it did not fall because it had it's foundation upon a rock".
Now I know that that verse is not talking about marriage but I apply it to marriage. If you build your marriage (your house) upon a solid foundation, nothing that life throws at your marriage will be able to tear it down. HOWEVER, if the occupants of that house do things to destroy the foundation from within, then that house will fall.
Dh knows my theory and agrees with it - that only WE can weaken and destroy our foundation. And it's little things - like saying something hurtful or accusing the other of lying when she didn't- that slowly chip away at that foundation.
So on the 3rd day of our fight I said, "Congratulations on chipping away at the foundation of our marriage" that stopped dh and he realized what he was doing because we both promised not to do anything to destroy our foundation. The light finally went on in his head that I was NOT the kind of person to lie to her husband and he apologized.
Perhaps you could share my theory with your dh and get him to see that it's not always the big things - affairs, gambling, drugs - that destroy a marriage but that sometimes it's the little comments here and there that slowly chip away at your foundation causing your house to fall.
Sorry so long. I was just hoping to help. Btw, I'm not a religious person and am not trying to push any aspect of religion on to you. Really, that verse is only one of two that I even know! lol!
Hope you're feeling better. I wouldn't let his comment go. I definitly think you should say something to him about it.