My older boy didn't want anything to do with the potty and by the time he was 3 1/2 he had only pee'd in it a handful of times. The day he turned 3 1/2 I packed up all of his diapers and put them away and that was just the end of it. I told him they were gone so we were going to have to learn how to use the potty. I don't think he ever had a pee accident, but for several days he'd go hide somewhere in the house and poop on the floor. And then somehow I got him to poop on the toilet one day. I think I found him sneaking off to poop and I just plopped him on the toilet and held onto him and talked him through it. After that we had no more accidents at all (unless you count the completely freak thing that happened 2 years later at the worst possible time, gah!). It was not like my friends who's kids were out of diapers at 2 1/2 but had to have their whole outfits changed a couple of times a day. I was so happy to skip the accidents!
Now my second ds is 3 years and 4 mos old. I think I'd be working on it now except I'm taking the california bar in july and I think it would be a pretty big distraction... But he seems to me more resistant than his brother, though he's shown a bit more aptitude for it. lol If I tell him I'll give him a piece of chocolate if he pees in the potty, he'll march right over and do it. But if there's no chocolate involved, he wants nothing to do with it and he cries for a dipe. I think we are going to have to pack his dipes up too, one day, probably in august so that he can start school in sept (his teacher-to-be is a good friend of mine and she keeps telling me I don't have to worry about the dipes, but I'd feel bad because it's really not her job to be changing him).
So.... I don't know if there're any words of wisdom in there, but you're not alone. I think maybe you're feeling more antsy because your first one was out of dipes younger, but up until 3 1/2 and maybe a couple of months later I'm not out of my comfort zone. Too much past that and I'd probably start worrying about it. But the truth of the matter is that it's not going to go on for too much longer.