An antenna is a significant part of any two way radio system. No radio can outperform its antenna. Many times users have been able to improve the performance of a communications system merely by improving the antennas.
The AM radio waves are below 2 MHz; these signals follow the Earth's curvature because they are reflected off the atmosphere. Therefore AM radio signals in low-noise environments can be received by radios that are way below the horizon hundreds of miles away.
Whereas two-way radios usually fall in the frequency range of 150MHz to 900MHz; these frequencies travel in straight lines. Radio waves can be reflected, or bounce off surfaces so the straight line between radios, may not always be so straight. Anyway as a general rule these waves cannot travel over the horizon or behind solid obstacles. Because of this reason, you have to factor in antenna height as well for sending signals farther.
Antenna gain is measured in decibels, or dB. A 0 dB antenna (sometimes called a "unity" gain antenna), is an antenna with no gain. More gain on an antenna will, usually but not always, give you more range.
Generally, to get more gain you need a bigger antenna. A simple "quarter wave" (0 dB, or unity gain) antenna for VHF is about 18 inches long. A 3 dB antenna for the same frequency would be about 4 feet long, while a 5 dB antenna is maybe 8 feet. For UHF, a quarter wave is six inches, and a 5 dB about 2 feet. Lower the frequency, longer the antenna.
If you're using the 3 inch stubbies on a UHF portable, you might see a difference going to the six inch rubber duck. Theoretically the longer antenna is better.
Directional antennas achieve much of their gain by rejecting pick-up from certain areas, such as the sides and rear in the case of a log periodic (the typical shark fin is an example of a log periodic antenna). A higher level signal helps improve the signal to noise ratio by overcoming the noise induced in an antenna cable, but a signal that is too high can cause problems at the receiver. This is why antenna amplifiers typically have jumper settings to apply the proper amount of gain needed to compensate for loss in a cable of particular type and length.
There are other factors that affect the range of a two-way radio too such as weather, exact frequency used, and obstructions. The radio's power output has a factor too.