In addition to your food diary, I would keep a mood diet. I've had IBS off an on my whole life, and it took me a long time to figure out that it was related to my anxiety.
Mine also went away during my pregnancies which may have been a physical side effect of my pregnancies, or could be because I felt mellow and not anxious when pregnant.
But I agree with seeing a GI doctor. If this has been going on for years, and it is effecting your life to this extent, it's not normal and should be addressed. Good luck with this!
I was thinking about the pregnancy thing and I think I know why I didn't have issues with it then. I was low on iron for all 3 pregnancies so I had to be on large doses of iron for all of them. I remember one side effect they warned me of was constipation, which I never had. So maybe the iron is what did it.