We live in Colorado at 9600 feet. We are all, of course, well acclimatized to the altitude, but I do notice a few things when we travel to lower altitudes and return home. (We have family on the east coast and have vacationed on islands with dd when she was an infant, so have made the sea-level to 9600 feet switch a few times with a baby!)
Make sure you all, and especially baby, are getting plenty of fluids. LOTS!!!
I have noticed that, when dd was a young infant, she would be fussy the first night back home. She is 2.5 years now, and that is no longer the case, but I thought it might have to do with adjusting back to high altitude. The rockies are much, much drier than the east coast, as well as having thinner air - a humidifier in the room at night can make a huge difference for anyone, but especially for a baby.
I wouldn't be too concerned about visiting a high altitude - be attentive to your dc, and of course, if you notice any signs of respiratory distress, get help quickly. But serious problems are not all that common. Some discomfort is more common - headaches, etc - which goes away after a day or so. Lots of fluids help, and perhaps some infant motrin if you think it is appropriate.
Enjoy your trip!