If your baby is having an intolerance to your milk, it is more likely to be a reaction to the protein, like casein or whey rather than something related to lactose. Right now your baby should have a lot of lactase to help break down lactose into galactose and glucose. Lactose intolerance, at least as I understand it, means lacking the enzyme that will break the bond between the two molecules, which leads to diarrhea, gas, bloating and abdominal pain. There are other things, like galactosemia that are related to lactose, but that is a metabolic disorder.
An actual allergy to the proteins in milk can present as a rash, swelling, hives or nasal congestion, coughing, sneezing, wheezing. This is from Web MD, so you can check that out for yourself.
Whether or not he is lactose intolerant later in life depends on his genetic background to some extent. Also from WebMD: "In rare cases, newborns are lactose-intolerant. Symptoms in newborns include severe foamy diarrhea, diaper rash, vomiting, dehydration, weakness and irritability, and slow weight gain."