my best guess would be a prolonged zinc deficiency... your body uses zinc for many things, among them synthesizing testosterone. a typical (non-supplemented) vegetarian diet probably has little to no zinc in it (since most people's dietary zinc comes from beef), but even if you're taking a multivitamin you're probably still low in zinc because some of the staples of a vegetarian diet (namely grains and beans) block zinc bioavailability. Firstly you body needs proteins to absorb zinc, and if you're low in protein you're probably low in zinc (and vise versa), secondly, you body cannot effectively digest protein and carbohydrates at the same time (and favors carbohydrate digestion if the conflict arises), and thirdly, grains and beans contain phytic acid, which chelates (grabs on to) zinc, magnesium, calcium and iron--preventing your body from absorbing them.
Also, low dietary protein causes your liver to release SHBG (sex hormone-binding globulin) an enzyme that, in men, detectives bioavailable testosterone (and actives estrogen, lol)--so that could be part of it.
only a guess though, probably ask a dietitian?
(a good, but not fool proof test is your fingernails. your body needs zinc to produce keratin, as well, and low zinc levels often show as brittle cloudy white spots on your finger/toe nails... if you have those you definitely have had low zinc for a long time, if not--you still might.)