Probably both. I've had a hard time keeping on weight BFing. I'm 5'3", and before veg diet was about 135 lbs. Now I'm 112 lbs, which is about 3 lbs less than I'd like, and I'm nursing my 12 mo also. Could be water loss, could be kind of a loss of toxin/back up type thing, especially if you had digestive issues on a conventional diet. The key to keeping on weight (at least maintaining it) for me has been high-fiber complex carbs. Eat spaghetti for lunch and oatmeal prepared with coconut milk before bed.
A veg diet is more easily processed by your body, foods are used more efficiently and more quickly, carbs and fats are less likely to be tucked away as glucose, and instead your body will start releasing glucose/water weight for energy - which you are using a lot of to nurse and take care of a toddler even if you're not "exercising" per se. I think you will find you feel much better in general without meat and dairy in your diet, but you may need to eat more/more often than you typically do. On a conventional diet, I usually ate 3 meals and a snack, on a veg/nursing diet, I eat more like 7 little meals a day.
Smoothies are very good for energy too and can help you keep on weight if you add the right stuff - coconut or almond milk, nut butters, fruits and greens.
Most people who switch to a veg diet lose most of their visceral (bad) fat, especially if they don't have much of it to lose. Sounds pretty typical to me, and I wouldn't worry too much about your underweight status unless you are also feeling a lack of energy, fatigue, overly warm, etc.