ISSN: 1545-5963

DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/TCBB.2013.123

Noah A. Rosenberg , Stanford University, Stanford

ABSTRACT

A coalescent history is an assignment of branches of a gene tree to branches of a species tree on which coalescences in the gene tree occur. The number of coalescent histories for a pair consisting of a labeled gene tree topology and a labeled species tree topology is important in gene tree probability computations, and more generally, in studying evolutionary possibilities for gene trees on species trees. Defining the \emph{$T_r$-caterpillar-like family} as a sequence of $n$-taxon trees constructed by replacing the $r$-taxon subtree of $n$-taxon caterpillars by a specific $r$-taxon labeled topology $T_r$, we examine the number of coalescent histories for caterpillar-like families with \tcr{matching} gene tree and species tree labeled topologies. For each $T_r$ with size $r\leq 8$, we compute the number of coalescent histories for $n$-taxon trees in the $T_r$-caterpillar-like family. Next, as $n\rightarrow \infty$, we find that the limiting ratio of the numbers of coalescent histories for the $T_r$ family and caterpillars themselves is correlated with the number of labeled histories for $T_r$. The results support a view that large numbers of coalescent histories occur when a tree has both a relatively balanced subtree and a high tree depth, contributing to deeper understanding of the combinatorics of gene trees and species trees.

INDEX TERMS

trees, phylogenetics, lineage sorting, coalescent

CITATION

Noah A. Rosenberg, "Coalescent Histories for Caterpillar-Like Families",

*IEEE/ACM Transactions on Computational Biology and Bioinformatics*, vol. , no. , pp. 0, 5555, doi:10.1109/TCBB.2013.123