J Syst Evol

• Research Articles •    

Excessive and asymmetrical removal of heterozygous sites by maxSH biases downstream population genetic inference: Implications for hybridization between two primroses

Jie Zhang1,2,3*, Francisco Pina‐Martins4, Zu‐Shi Jin5, Yong‐Peng Cha1,3, Zu‐Yao Liu6, Jun‐Chu Peng1,3, Jian‐Li Zhao1,3*, and Qing‐Jun Li1,3*   

  1. 1 Laboratory of Ecology and Evolutionary Biology, School of Ecology and Environmental Sciences, Yunnan University, Kunming 650500, China
    2 Institute of International Rivers and Eco‐Security, Yunnan University, Kunming 650500, China
    3 Yunnan Key Laboratory of Plant Reproductive Adaptation and Evolutionary Biology, Yunnan University, Kunming 650500, China
    4 Computational Biology and Population Genomics Group, Departamento de Biologia Animal, Faculdade de Ciências, Centre for Ecology, Evolution and Environmental Changes, Universidade de Lisboa, 1649‐004 Lisbon, Portugal
    5 School of Food Science, Tibet Agricultural and Animal Husbandry University, Nyingchi 860000, China
    6 Division of Evolutionary Ecology, Institute of Ecology and Evolution, University of Bern, 3012 Bern, Switzerland
  • Received:2022-04-20 Accepted:2022-10-21 Online:2022-11-03


Techniques of reduced-representation sequencing (RRS) have revolutionized ecological and evolutionary genomics studies. Precise establishment of orthologs is a critical challenge for RRS, especially when a reference genome is absent. The proportion of shared heterozygous sites across samples is an alternative criterion for filtering paralogs. In the prevailing pipeline for variant calling of RRS data – PYRAD/IPYRAD, maxSH is an often overlooked parameter with implications to detecting and filtering paralogs according to shared heterozygosity. Using empirical genotyping by sequencing data of two primroses (Primula alpicola Stapf and Primula florindae Ward) and their putative hybrids, and extra data sets of Californian golden cup oaks, we explore the impact of maxSH on filtering paralogs and further downstream analyses. Our study sheds light on the simultaneous validity and risk of filtering paralogs using maxSH, and its significant effects on downstream analyses of outlier detection, population assignment, and demographic modeling, emphasizing the importance of attention to detail during bioinformatic processes. The mutual confirmation between results of population assignment and demographic modeling in this study suggested maxSH = 0.10 has a potentially excessive and asymmetrical effect on the removal of truly shared heterozygous sites as paralogs. These results indicate that hybridization origin hypotheses of putative hybrids represented by results with maxSH = 0.25 and 0.50 are more credible. In conclusion, we revealed the critical hazard of paralogs filtration according to sharing heterozygosity at first, so that we propose to use specific protocols, rather than maxSH, to filter potential paralogs for closely related lineages.

Key words: bioinformatics, demographic modeling, Heterozygous site, paralog, Primula