I would think as long as you are limiting pacifier use to sleep, his speech is developing normally, and his teeth aren't affected then the paci can stay. Most children give up their comfort/ transitional object by fall of their kindergarten year.
We used the same approach to wean from the paci as we did to wean from the breast (" the paci is going to go bye bye in 5 days") and didn't have any issues. Paci use continued for another 10 months after breastfeeding ended just because I didn't want to throw too many things in the mix at once. We chose to wean from the paci because his teeth were shifting (thanks to a fall he'd had that loosened two teeth). I chose to wean from the breast because after 2 years I just felt done.