To get labels of glass jars, I usually use really hot soapy water. If that isn't enough to it off, I have been known to toss them into a pot of boiling water lol
I can't recall ever using anything stronger to take off labels/stickers.
I take the label off, and then boil the jar. The glue comes off in the boiling water. I assume it's the same for all jars. I've never used a baby food jar ... just pasta jars. But, I bet it's the same glue.