Inference for [lambda]-coalescents

Abstract

One of the main problems in mathematical genetics is the inference of evolutionary parameters of a population (such as the mutation rate) based on the observed genetic types in a finite DNA sample. If the population model under consideration is in the domain of attraction of a classical Fleming-Viot process, then the standard means to describe the corresponding genealogy is Kingman's coalescent. For this process, powerful inference methods are well-established. An important feature of this class of models is, roughly speaking, that the number of offspring of each individual is small when compared to the total population size. Recently, more general population models have been studied, in particular in the domain of attraction of so-called generalised Lambda Fleming-Viot processes, as well as their (dual) genealogies, given by the so-called Lambda-coalescents. Moreover, Eldon & Wakeley (2006) have provided evidence that such more general coalescents, which allow m ultiple collisions, might actually be more adequate to describe real populations with extreme reproductive behaviour, in particular many marine species. In this paper, we extend methods of Ethier & Griffiths (1987) and Griffiths & Tavaré (1994) to obtain a likelihood based inference method for general Lambda-coalescents. In particular, we obtain a method to compute (approximate) likelihood surfaces for the observed type probabilities of a given sample. We argue that within the (vast) family of Lambda-coalescents, the parametrisable sub-family of Beta$(2-alpha,alpha)$-coalescents, where $alpha in (1,2]$, are of particular biological relevance. We apply our method in this case to simulated and real data (taken from Árnason (2004)). We conclude that for populations with extreme reproductive behaviour, the Kingman-coalescent as standard model might have to be replaced by more general coalescents, in particular by Beta$(2-alpha,alpha)$-coalescents.