I can summarize my problem with typical vertical antennas as follows:
- Ground losses
- Environment losses
- Multiband inefficiencies
First notice that not all parts of an antenna are equal. The contribution to radiation is in proportion to the current. For a ground plane antenna with the current next to ground (½ on the monopole and the rest equally distributed on the radials) the coupling to ground and an obstructed view of the horizon (buildings, trees, terrain variation) can be disastrous.
Even with a vertical dipole the loss is substantial, and only gets slightly lower when somewhat higher (-5.1 db when mounted at 10 meters). Remember that when you peruse those ads for all-band, no-radial verticals.
In the majority of cases a low dipole will outperform a ground-mounted vertical, and not just for DX. In the above example, although when up only λ/4 the radiation mostly goes straight up, at 26° elevation (where the ground plane peaks) it actually outperforms the ground plane by ~3 db. It also sees fewer low-angle obstructions.
The addition of (many!) more radials can reduce near-field ground losses but cannot address the environmental and far-field losses, which should not be underestimated. I have used verticals that work, but only when mounted at a height that is at least roof level, which at least addresses some of the environmental losses if not other problems.
Other verticals types of antenna that lift the current to higher points do perform better. These antenna include the quarter-wave (or half) sloper, and side-fed full-wave loops. Actually the last is not really a vertical, but is vertically polarized.
Making a vertical multiband is not easy. Unlike with dipoles you cannot make a "fan" vertical since only one or a few of the monopoles can be at or near vertical, so for the other bands the pattern is skewed. It also gets expensive if the structure is anchored to the ground since it will require construction from solid tubing; although element suspension from an overhead tree limb can be an alternative for some locations.
Commercial multiband verticals use a combination of one or more of traps, linear loading and matching networks to achieve their aim. There are unavoidable compromises that must be made. Traps add losses, as do matching networks. They also increase the antenna Q such that they are both difficult to adequately tune on all bands and have a small bandwidth. Both require additional tuning at the transmitter end, which adds its own losses. If horizontal space is at a premium and effectiveness if not a primary aim, these antennas can fit the bill. For my purposes they are unacceptable.
So, why do so many hams swear by verticals? In my own experience this would appear to come down to human nature. Whether it is a house, a car, a smart phone or an antenna, once we've invested time and money into it, and others see what we've chosen, there is a tendency to retroactively justify the choice, to convince not only others but also ourselves. The antenna might even seem to hear well, although this is usually due to attenuation of both signal and noise so that the SNR for all but the lowest-angle DX can be comparable to other antennas; however, they are not comparable at the other end of the path. For many, a common solution is an amplifier. I suppose it's a good thing that so many commercial vertical are rated for a kilowatt or more. Do that, and if you have yet to meet your neighbours you soon will.
If the low bands are your thing -- 80 and 160 meters -- there may be no alternative to a ground-mounted vertical. Get them out in an open field and put down lots of radials and/or ground screens. They can also be made multi-element for directivity and gain. Since I cannot achieve any of this in my current situation I plan to initially focus on 7 MHz and up for my new station's antennas. The lower bands will have to wait.