We took DD1 on a 5-6 hour car drive (each way) for a wedding (my cousins) at 4 weeks old! I don't remember it as a hard trip. We stayed at my moms house and pretty much our entire extended family (my siblings, aunts, uncles, cousins, and DH's siblings) got to meet her. I think it was way less stressful than having all of our siblings (I have 5, DH has 3) come meet her here, even slowly over a period of several months.
We then went to another wedding (DH's brother's wedding and DH was the best man) at 7 weeks old. That one was about a 10-12 hour drive, which we split into two days, and stayed at my mom's house for 2 nights about half way there. It was a much more difficult and stressful trip, but not because of the drive itself. That was just the difference between my in laws expectations of babies (all formula fed) versus my family (all breastfed). Also, we were staying at a hotel and dealing with family drama.
We proceeded to do about once a month trips to our home town, a 5-6 hour drive, for about the first 18 months or so of DD's life. We went to more like every other month as the trips got to be draining for us and for DH's vacation time. During this pregnancy, we went at Thanksgiving (I didn't know I was pregnant yet, but I was a mess), Christmas, and then not again until Easter, and haven't been back since. I've missed going.
I clearly remember one very rough drive very early on, though I'm not sure which drive it was. We were trying to drive late at night after DH was off of work (which we'd always done pre-baby), but DD kept crying and we'd stop and nurse and as soon as I put her back in the car seat she'd cry again. I started to fear that we'd have to stop at some hotel for the night. Once we figured out that evening and night time, when DD did her normal cluster feeds, was a bad time to drive, it's gone much more smoothly. I know lots of people drive at night so the kids will sleep. We drive in the early afternoon so DD will sleep.