If it were me, I would worry under a few circumstances...
1) if the fever lasts longer than two days, sometime into the third day I would become concerned.
2) if my child became lethargic I would be VERY concerned.
3) if the wet diapers decreased and my child showed signs of dehydration (dry eyes when crying, dry mouth), I'd be very concerned.
But you know what? We all have different levels of tolerance for illness and pain in our children, and if you're concerned now, then I suggest you act on it. You'll feel better if you do. I hope Peyton feels better soon.