I haven't seen stats to that effect but I'll take you word for it for the moment. There are many other factors in polygamous societies besides height and other physical traits, with money and power being major ones. Not sure that the wealthy always represent the healthiest part of your gene pool. It makes the masses less able to find partners, similar to the one child rule and abortion of females in China, which in turn leads to other social problems.
In my opinion it's not a race of making the most children as fast as possible.