It's not good if it is turning brown. Is it turning brown at the ends of the branches or just in the shaded middle? Big dense rosemary will be dead looking inside where it isn't getting sun, but if it's turning brown at the outer branches and tips it's the sign of an unhappy rosemary.
How cold did it get last winter where you live? The hardiest cultivars of rosemary are only hardy to Zone 6 or 7, and many are only hardy in Zones 8 and 9, so it may have been damaged during the winter.
My rosemary turned brown presumably because I neglected it for quite awhile. I recently pruned it WAY back and began watering it more deeply and regularly and it is starting to do better. I was thinking about fertilizing it with some fish emulsion but not sure if that is good for rosemary. (?)