You do need a specialist, and it should either be a urologist VERY familiar with fertility issues, or an endocrinologist.
My dh took clomid for three months in an effort to help his male factor issues because his urologist tested for testosterone and claimed his was low. Clomid is supposed to help with this. Several months later, a new SA showed WORSE results than ever.
By this time, we had gotten an appointment with an endocrinologist, who reviewed the prior paperwork and determined that the urologist had run the WRONG test on dh, and his testosterone was fine. Clomid then raised his testosterone even more, which then killed off his swimmers. This was the same urologist, btw, who said he didn't have a varicocile. After that, we got a second opinion. The new urologist saw it from across the room. It was that big.

So, please learn from my experience and call your local infertility clinic and get a recommendation from them for a urologist or endocrinologist who is familiar with male factor issues, so that the right tests get run the first time and the right diagnosis can be made. Yes, it is more time consuming and more money up front, but in the long run will save you time, money, and possibly heartache.