I totally had this experience, too. We switched to drawing names and someone decided the limit was $50! I felt pressure that I had to give my person something they would really like since they would only be getting one thing. And it meant I couldn't give couple gifts or family gifts. Since Dh was also assigned someone, the $ was well above what we had spent previously. I just opted out after a couple years. We just give gifts to local relatives. Sometimes I skip the younger kids because they get so much and don't notice who gives them what. I'd rather give to adults who appreciate it and don't get so much.
Well to play devils advocate here, the LIMIT is $50 or $XX.xx. You do not need to spend that much. You can spend whatever amount you want. Plus you have no idea if your person is only getting one thing. I've found that in 'name drawing' situations family members still buy for others. Like you pull Aunt M's name. She is still going to get presents from her kids and her mom. (You are a distant cousin). Grandma is going to get presents from her kids and grandkids no matter what. The gifts just many not be given at this particular party or time.