I actually think your dr is correct that delaying solids beyond 6 months (not 4 months) does not reduce, and may increase allergies.
http://www.ncbi.nlm.nih.gov/pubmed/19685855
Breastfeed Rev. 2009 Jul;17(2):23-31.
Is 6 months still the best for exclusive breastfeeding and introduction of solids? A literature review with consideration to the risk of the development of allergies.
Anderson J, Malley K, Snell R.
"Health professionals advising mothers on the introduction of solid foods to infants need evidence-based guidelines. A literature review on this topic was undertaken to examine the current international recommendations of expert bodies and evidence-based research published since 2003. Particular reference in this review is made to the timing of introducing food allergens and the risk of development of allergy in the child. Recommendations in developed countries of reducing this risk by avoidance of allergenic foods until the child is of varying ages past 6 months have been challenged by recent population studies. Where the risk of allergy is a key consideration, currently-available research suggests that introducing solids at 4-6 months may result in the lowest allergy risk. When all aspects of health are taken into account, the recommended duration of exclusive breastfeeding and age of introduction of solids were confirmed to be 6 months, but no later."
From what I have read, the earlier recommendations for delay were based on speculation bc the evidence was not there. Evidence has started to accumulate in the past few years and so the guidelines are starting to change.
However, this does not seem to me like a good reason to start solids in a child who is not ready or interested. I really think that your child's behavior is the best indicator of his readiness, because your kid is not a statistic. Remember these studies are population averages.
If your kid is sitting well, has lost his tongue thrust, and is grabbing at food at 4 months, I say why not. If he's not doing these things, I would not push food at him because you think it is going to reduce his allergy risk by 15.7% or whatever.