What do you mean by only putting up ferns? That's all it ever does, in a way. The shoots, that you are used to eating, normally grow up to be big tall ferns. I harvest mine for a few weeks and then stop and let the rest grow up into ferns to feed the plant. So it's the first growth in Spring that is eaten. If you harvest ALL of the "shoots" then the plants will get weaker. If all of the new growth is small, so it doesn't look like edible asparagus, then your plants are not established yet and you shouldn't harvest anything yet. because they are not strong enough. A good rule of thumb is that when most sprouts are pencil-sized or smaller, you need to let them grow. I'm guessing that despite their age your plants need compost, water, and weeding, and no harvesting for a couple of years so that they can be more vigorous.
I've had my ups and downs with the asparagus beetles and now I just try to make my plants as healthy as possible instead of fighting the insects. Healthier plants are less attractive to insects and they recover more easily from the damage too. Things like using wood ash to make the soil less acid can help asparagus. I feel for you. Those little larvae can be really hard on the plants, there are so many that they leave even big plants leafless.