Abstract: Genetic differentiation of populations has been traditionally quantified by Wright's F-statistics, typically assuming mutation–migration–drift equilibrium. However, the equilibrium perspective can be unrealistic as many natural populations are likely not yet in equilibrium. Therefore, understanding the behaviors, robustness, and power of the differentiation indexes under non-equilibrium conditions has important implications. Here, we report an extensive examination of the properties of two major indexes G_ST and D under non-equilibrium conditions by theoretical deduction under the infinite allele model (IAM) and simulation under the stepwise mutation model (SMM). Several properties of G_ST and D valid under both SMM and IAM, which have not been recognized under the equilibrium perspective, were unveiled. First, if gene flow is very weak (e.g., m < 10⁻⁴), G_ST, like D, also takes a fairly long time to reach equilibrium if mutation rate is not very large. When G_ST (D) is in equilibrium depends on when H_S and H_T are both in equilibrium. Under IAM and complete isolation, this is determined by the product of µ and t: G_ST will not approach equilibrium as long as µt ≪ 1. Under SMM, 10⁻⁴ appears to be the rough threshold migration rate; when m < 10⁻⁴, G_ST approaches equilibrium much slower and later than H_S, whereas the opposite is true when m > 10⁻⁴. Second, contrary to the popular belief, µ ≪ m is neither an obligatory request for G_ST to be an effective differentiation measure nor a sufficient condition for using G_ST to estimate gene flow level (Nm), if G_ST is not yet approaching equilibrium. Third, under SMM (but not IAM) and complete isolation, when population size is large (e.g., ≥1000), mutation rate shows a great impact on G_ST but only a mild influence on D; hence D can be much less sensitive to mutation rate heterogeneity than G_ST in certain situations. Fourth, whatever the level of gene flow, drift plays a dominant role on G_ST, whereas gene flow appears to have a stronger influence on D when subpopulations exchange individuals (even only infrequently). Moreover, G_ST appears to have a larger power to detect recent population genetic events than D. Finally, it is the migration rate (m), not the absolute number of migrants (Nm), that determines the speed at which G_ST or D approaches equilibrium. These results suggest that G_ST and D bear unique revealing powers under non-equilibrium conditions.

Key words: F-statistics, G_ST and D, infinite allele model, nonequilibrium, population genetic differentiation, stepwise mutation model.