I don't think that 40 is too old personally. However, with a 4 1/2 year old and a one year old, you may be stretched pretty thin if you get pregnant. You certainly need to consider the impact that pregnancy and the postpartum recovery, as well as the additional demands of another babe will have on you, your body, and your two children - as well as your dh! How much of a support network do you have locally?
I am 38 and almost 20 weeks pregnant with my second child. I will be 39 in August. I am having a tough pregnancy with being unbelievably exhausted all of the time and the marathon sickness is very hard to deal with as well. Fortunately my dd is 4 1/2 and is very able to cope on her own when momma can't play with her or when I need to nap, which happens almost every day! She is mature enough that I can trust her to play alone while I nap, but I doubt I would be willing to do that with a 1 year old!
The advice from a pp on waiting longer is nice advice, but it may not be feasible if you are really intent on having another child. It took me 2 1/2 years to get pregnant this time around, but it turns out that we were dealing with male factor issues that don't relate to my age. lol But you have to consider whether you would rather have another child now and deal with the difficulties as opposed to taking your chances on waiting a year.
Did you get pregnant easily with your two children? If so you may have better odds if you want to wait longer to ttc.
Good luck with your decision. It's a hard one to make.