I can't come at this from a foster-adopt angle, just an adoption angle and a mom-to-special-needs-children angle.
I think your responses to raising a difficult child (whether bio or adopted, whether physical, mental, or emotional needs) is greatly affected by your own outlook and personality. That said, no matter how sunny or resilient or strong or optimistic you are, raising difficult children is HARD at times. And not just in "how it affects your happiness with parenting" ways. It takes a huge toll on your relationships, your marriage...how social you can be, how much freedom you have, your finances, your time...even your ability to relate to other people.
I've had three "difficult" experiences. My first children were special-needs twins, who went through quite a lot (lots of hospital stays, brain surgeries, medicine trials, ER visits, you name it). I was a stay-at-home mom to them, and only them, for the first four years of their life. Before having children, I was involved in my community, was pursuing big career goals, had lots of friends, and lived a very strong, optimistic, meaningful life. Those first years of having the twins gradually wore me down, until I finally had to admit that all the strains and stresses of being a 'special needs' mom had worn me out...my marriage was rocky, I had few community connections, and I was depressed. I could never, ever have predicted that turn-out for me, even if I had known the challenges I'd face. I didn't think I was the kind of person that could "let" life overwhelm me...certainly I'd faced difficult issues in the past (as a young adult and pre-parent), and nothing affected me as much as this parenting experience.
My middle child experience, ds3, has been a dream. I think he helped bring me back to life, honestly (I helped, too, but parenting him was almost like being reborn as a happy mom). He is a healthy, typical, bio child. Through parenting him, I finally experienced what it was like, the joy, of raising a typical child. I had experiences that could relate to others. I could join play groups! I could go to the park and not be the outcast. I experienced a public life without scrutiny, pity, or embarrassment. Those might sound like very shallow things to value, but I can't tell you how important they are. I can't tell you how amazing it is to parent without fear, without constant worry... it was just so EASY. So JOYFUL. I couldn't believe what I'd been missing all the previous years, even as much as I loved ds1 and ds2. I never knew what parenting could be until I parented ds3.
My youngest child, dd, was a difficult parenting experience. Two years later, it still is sometimes. DD came home young, at 9 months, but she had been through several caretaker transitions in her young life. From an early age in Korea, she showed anxiety around strangers and fear/grief at change...so no surprise, when we brought her home, it was like a bomb exploded in our lives. We were not prepared, despite years of reading and research, for how difficult it was to parent her. She grieved, she manipulated, she didn't sleep, she resisted us...again, it doesn't sound like much, but after months of frustration and severe (!) sleep deprivation, dh and I were at our limits and our other kids (not to mention dd) were NOT getting the life or parenting they deserved. I actually found a lot of our parenting experience with her MORE difficult than what we'd been through with the twins, which is saying a lot. Over the past two years, we've worked long and hard to improve our relationship with her in baby steps. We measure progress by the month, or the half-year, and over time it has gotten better. She was just incredibly, incredibly needy...and infuriating when she didn't get what she wanted (with anxiety and attachment coming into play as well). Parenting her was, and still is at times, exhausting. She eats up a lot of our emotional reserves.
So, your question...is your enjoyment of parenting more about the kids, or more about you? And I wonder if our happiness as parents really will depend on whether we get a "difficult" child?
I think it's naive to think that your general positive outlook, or ability to handle challenges, will mean you can parent a difficult child unscathed. You have to have a healthy respect, I think, for the statistics of marriages, depression, and other outcomes of difficult parenting situations. It's not as if all the people who experience divorce, or depression, or even shorter life spans, are just the weak ones, or the ones that didn't try hard enough. I've seen families and marriages...healthy, thriving, positive people with great community connections...fall. I've been close to that edge myself more than once.
For me, parenting joy is definitely affected my my outlook....enjoying the little moments, even in a challenging experience, brings great joy. Celebrating the milestones, and "inchstones" brings great joy. I love being a parent more than I ever thought I would, and have chosen to have a big (four now, five including our son that passed) family because of it. That ability to see the little things, to experience joy, was true of my life before parenting, and it's especially true now. What I didn't expect was that eventually, all the background noise and stress and strain of difficult parenting lessened my ability to enjoy the little moments. It took a lot of work, and therapy, to get healthy again. It still does. Stress wears you out. It wears your partner out. In some ways you can prepare yourself for how stress will affect you, but you can't always count on your partner having the same reactions, the same tolerance, or the same vision of how to fix things. It's tricky, no doubt.