Would like some insight... Sick 4 month old

[i*wish]

i feel like I'm going crazy with sleep deprivation...my barely 4month old baby baby has been sick for a week and I'm scared. I don't want to overreact (again...read below) but don't want to under react either. If anyone here can offer me some clear-headed insight I would be most appreciative.
Timeline:
Saturday 9/24 overnight...nasal congestion, happy baby, no fever...
This continued through Tuesday
Tuesday night woke at midnight very fussy, would not nurse, couldn't really go back to sleep, held all night, at 6am I thought he felt warm and checked his axillary temp and got 104... Based on this and the not nursing (big concern for me) and the irritability I decided to go to the ER. When we got there his rectal temp was 100.0 and he nursed. I felt so relieved and was ready to head home. He seemed improved in his mood. They scared the crap out of me and wanted to do all invasive tests...we rechecked his temp and it was 100.6. I was exhausted and scared and consented to these horrible tests...all neg. Blood and urine culture and lumbar puncture...
So that was all day wed
Thurs went to our ped...his temp was still there off and on up to about 102 axillary. He was mostly sleepy but not irritable. Doc checked him out and we thought virus.
Friday around 3pm he became intensely upset and could not be calmed...he would not nurse, he had a temp to 101. I gave him his first dose of Tylenol out of desperation to help him feel better. He felt better and was asleep for the next couple of hours.
Friday night he was up a lot and very very pained by something...literally screaming. We wound up giving 2 more doses as this was the only thing to give relief.
Saturday I took him in to be checked. Ears looked fine, he seemed better mood-wise and was mostly afebrile... Then last night it was much of the same as Friday night he barely slept. Temp this morning was 101.1. He might have a rumbling in his belly, stools seem normal except green/brown I stead of yellow.

I'm so sorry this is so long. I'm so worried about my baby. Right now he had seemed comfortable for about an hour and is now starting to scream again.

I'm so reluctant to take him back to the ER but my gut is saying something is wrong. Something is causing him pain.

This is my 3rd baby and I do tend to fret when they're sick but this is such a young baby and a long time and a strange course....
Thank you for any thoughts.

[Jenni1894]

Did they check his ears?   It almost sounds like an ear infection.  or teething?

My YDD gets them often and she is fussy at night, didn't like to nurse and would get a low temp.  

[prothyraia]

I know this is two days old now, but I just wanted to say that you shouldn't hesitate to call or take him in if you're worried.  I hope he's feeling better by now.

We usually stay home with high fevers in toddlers and older kids, but four months is very very young.

[1love4ever]

Certainly take him in if you feel you should.  I would like to add though that a fever does have a purpose, and if it is 101 and under it is best to try not to give them medicine although if they are miserable and crying a lot I still do, but I try to let the fever be there to help for at least a little while as long as it is not over 101.  I would also like to say that teething is often exaggerated or used as an explaination for a baby acting fussy or being sick, when in reality most babies pop teeth without their parents even noticing because they act no different.  There are exceptions of course, and sometimes the baby will be fussy for a little while and chewing on everything, but not screaming non-stop and feverish, etc.