Variation in memory and the hippocampus across populations from different climates: a common garden approach

Selection for enhanced cognitive traits is hypothesized to produce enhancements to brain structures that support those traits. Although numerous studies suggest that this pattern is robust, there are several mechanisms that may produce this association. First, cognitive traits and their neural underpinnings may be fixed as a result of differential selection on cognitive function within specific environments. Second, these relationships may be the product of the selection for plasticity, where differences are produced owing to an individual's experiences in the environment. Alternatively, the relationship may be a complex function of experience, genetics and/or epigenetic effects. Using a well-studied model species (black-capped chickadee, Poecile atricapillus), we have for the first time, to our knowledge, addressed these hypotheses. We found that differences in hippocampal (Hp) neuron number, neurogenesis and spatial memory previously observed in wild chickadees persisted in hand-raised birds from the same populations, even when birds were raised in an identical environment. These findings reject the hypothesis that variation in these traits is owing solely to differences in memory-based experiences in different environments. Moreover, neuron number and neurogenesis were strikingly similar between captive-raised and wild birds from the same populations, further supporting the genetic hypothesis. Hp volume, however, did not differ between the captive-raised populations, yet was very different in their wild counterparts, supporting the experience hypothesis. Our results indicate that the production of some Hp factors may be inherited and largely independent of environmental experiences in adult life, regardless of their magnitude, in animals under high selection pressure for memory, while traits such as volume may be more plastic and modified by the environment.     

Inertia: the discrepancy between individual and common good in dispersal and prospecting behaviour

The group selection debate of the 1960s made it clear that evolution does not necessarily increase population performance. Individuals can be selected to have traits that diminish a common good and make population persistence difficult. At the extreme, the discrepancy between levels of selection is predicted to make traits evolve towards values at which a population can no longer persist (evolutionary suicide). Dispersal and prospecting are prime examples of traits that have a strong influence on population persistence under environmental and demographic stochasticity. Theory predicts that an ‘optimal’ dispersal strategy from a population point of view can differ considerably from that produced by individual‐level selection. Because dispersal is frequently risky or otherwise costly, individuals are often predicted to disperse less than would be ideal for population performance (persistence or size). We define this discrepancy as ‘inertia’ and examine current knowledge of its occurrence and effects on population dynamics in nature. We argue that inertia is potentially widespread but that a framework is currently lacking for predicting precisely the extent to which it has a real influence on population persistence. The opposite of inertia, ‘hypermobility’ (more dispersal by individuals than would maximize population performance) remains a possibility: it is known that highest dispersal rates do not lead to best expected population performance, and examples of such high dispersal evolving exist at least in the theoretical literature. We also show, by considering prospecting behaviour, that similar issues arise in species with advanced cognitive and learning abilities. Individual prospecting strategies and the information acquired during dispersal are known to influence the decisions and therefore the fate of individuals and, as a corollary, populations. Again, the willingness of individuals to sample environments might evolve to levels that are not optimal for populations. This conflict can take intriguing forms. For example, better cognitive abilities of individuals may not always lead to better population‐level performance. Simulation studies have found that ‘blind’ dispersal can lead to better connected metapopulations than cognitively more advanced habitat choice rules: the latter can lead to too many individuals sticking to nearby safe habitat. The study of the mismatch between individual and population fitness should not be a mere intellectual exercise. Population managers typically need to take a population‐level view of performance, which may necessitate human intervention if it differs from what is selected for. We conclude that our knowledge of inertia and hypermobility would advance faster if theoretical studies—without much additional effort—quantified the population consequences of the evolving traits and compared this with hypothetical (not selectively favoured) dispersal rules, and if empirical studies were similarly conducted with the differing levels of selection in mind.     

Studying the evolutionary ecology of cognition in the wild: a review of practical and conceptual challenges

Cognition is defined as the processes by which animals collect, retain and use information from their environment to guide their behaviour. Thus cognition is essential in a wide range of behaviours, including foraging, avoiding predators and mating. Despite this pivotal role, the evolutionary processes shaping variation in cognitive performance among individuals in wild populations remain very poorly understood. Selection experiments in captivity suggest that cognitive traits can have substantial heritability and can undergo rapid evolution. However only a handful of studies have attempted to explore how cognition influences life‐history variation and fitness in the wild, and direct evidence for the action of natural or sexual selection on cognition is still lacking, reasons for which are diverse. Here we review the current literature with a view to: (i) highlighting the key practical and conceptual challenges faced by the field; (ii) describing how to define and measure cognitive traits in natural populations, and suggesting which species, populations and cognitive traits might be examined to greatest effect; emphasis is placed on selecting traits that are linked to functional behaviour; (iii) discussing how to deal with confounding factors such as personality and motivation in field as well as captive studies; (iv) describing how to measure and interpret relationships between cognitive performance, functional behaviour and fitness, offering some suggestions as to when and what kind of selection might be predicted; and (v) showing how an evolutionary ecological framework, more generally, along with innovative technologies has the potential to revolutionise the study of cognition in the wild. We conclude that the evolutionary ecology of cognition in wild populations is a rapidly expanding interdisciplinary field providing many opportunities for advancing the understanding of how cognitive abilities have evolved.     

Variation in hippocampal morphology along an environmental gradient: controlling for the effects of day length

Environmental conditions may create increased demands for memory, which in turn may affect specific brain regions responsible for memory function. This may occur either via phenotypic plasticity or selection for individuals with enhanced cognitive abilities. For food-caching animals, in particular, spatial memory appears to be important because it may have a direct effect on fitness via their ability to accurately retrieve food caches. Our previous studies have shown that caching animals living in more harsh environments (characterized by low temperatures, high snow cover and short day lengths) possess more neurons within a larger hippocampus (Hp), a part of the brain involved in spatial memory. However, the relative role of each of these environmental features in the relationship is unknown. Here, we dissociate the effects of one theoretically important factor (day length) within the environmental severity/Hp relationship by examining food-caching birds (black-capped chickadee, Poecile atricapillus) selected at locations along the same latitude, but with very different climatic regimes. There was a significant difference in Hp attributes among populations along the same latitude with very different climatic features. Birds from the climatically mild location had significantly smaller Hp volumes and fewer Hp neurons than birds from the more harsh populations, even though all populations experienced similar day lengths. These results suggest that variables such as temperature and snow cover seem to be important even without the compounding effect of reduced day length at higher latitudes and suggest that low temperature and snow cover alone may be sufficient to generate high demands for memory and the hippocampus. Our data further confirmed that the association between harsh environment and the hippocampus in food-caching animals is robust across a large geographical area and across years.     

Sex Ratio and Sex‐specific Latitudinal Variation in Floral Characteristics of Gynodioecious Kallstroemia grandiflora (Zygophyllaceae) in Mexico

In species with broad distribution, geographic variation could be associated to particular environmental factors that may affect plant reproduction. In dimorphic species, environmental factors may influence evolutionary processes governing sex ratio variation and floral traits evolution. Kallstroemia grandiflora is an annual insect‐pollinated gynodioecious herb with broad distribution from the Sonoran desert to the semiarid west coast of Mexico. In this study, we explored if female frequency increases in harsh environments and the associated changes in floral morphology in female and hermaphrodite plants. We selected 23 populations along the pacific coast of Mexico where we estimated female frequency and in eight of them we also measured flower size in hermaphrodite and female plants. The average decrease in precipitation was correlated with the increase in female frequency, and a reduction in the flower size of hermaphrodite plants. Our results support the idea that harsh environments may influence female frequency, but also flower size and possibly breeding system evolution.     

Contrasting patterns of sexually selected traits in Mediterranean and continental populations of European mouflon

The expression of sexually selected traits in highly dimorphic ungulates may be influenced by environmental quality. Variations in habitat conditions can impose different constraints on the allocation of energy resources to male life‐history traits, and possibly alter the female preferences for specific features. Here, we compared the horn growth patterns in male European mouflon Ovis aries musimon living in different habitats (Mediterranean vs. continental) but sharing a common genetic origin. We hypothesized that the expression of sexually selected traits such as horn development should be promoted in more favorable habitat conditions (i.e., Mediterranean). Using linear mixed models on data retrieved from individuals harvested under the same hunting regime, we found longer horns and greater individual variance in horn segment length in the Mediterranean population than in the continental one. Furthermore, Mediterranean rams showed no evidence of compensatory horn growth, as opposed to the continental rams. Unexpectedly, horn base circumference was greater in the continental habitat than in the Mediterranean one. The overall results suggest different patterns of investment in horns in the two populations, with seemingly stronger pressure and consequences of sexual selection on mouflon rams living in more favorable environments. Although the role of hunters' selectivity cannot be excluded a priori, our data suggest that the differences in the expression of sexually selected traits in our study populations may be influenced by environmental conditions. Because sexual selection can impose substantial fitness costs on individuals, further investigations on the trade‐offs between reproduction and survival would improve our understanding of the dynamics of mouflon populations living in different environmental conditions.     

Environmental variability and acoustic signals: a multi-level approach in songbirds

Among songbirds, growing evidence suggests that acoustic adaptation of song traits occurs in response to habitat features. Despite extensive study, most research supporting acoustic adaptation has only considered acoustic traits averaged for species or populations, overlooking intraindividual variation of song traits, which may facilitate effective communication in heterogeneous and variable environments. Fewer studies have explicitly incorporated sexual selection, which, if strong, may favour variation across environments. Here, we evaluate the prevalence of acoustic adaptation among 44 species of songbirds by determining how environmental variability and sexual selection intensity are associated with song variability (intraindividual and intraspecific) and short-term song complexity. We show that variability in precipitation can explain short-term song complexity among taxonomically diverse songbirds, and that precipitation seasonality and the intensity of sexual selection are related to intraindividual song variation. Our results link song complexity to environmental variability, something previously found for mockingbirds (Family Mimidae). Perhaps more importantly, our results illustrate that individual variation in song traits may be shaped by both environmental variability and strength of sexual selection.     

Consequences of genetic erosion on fitness and phenotypic plasticity in European tree frog populations (Hyla arborea)

The detrimental effects of genetic erosion on small isolated populations are widely recognized contrary to their interactions with environmental changes. The ability of genotypes to plastically respond to variability is probably essential for the persistence of these populations. Genetic erosion impact may be exacerbated if inbreeding affects plastic responses or if their maintenance were at higher phenotypic costs. To understand the interplay 'genetic erosion-fitness-phenotypic plasticity', we experimentally compared, in different environments, the larval performances and plastic responses to predation of European tree frogs (Hyla arborea) from isolated and connected populations. Tadpoles from isolated populations were less performant, but the traits affected were environmental dependant. Heterosis observed in crosses between isolated populations allowed attributing their low fitness to inbreeding. Phenotypic plasticity can be maintained in the face of genetic erosion as inducible defences in response to predator were identical in all populations. However, the higher survival and developmental costs for isolated populations in harsh conditions may lead to an additional fitness loss for isolated populations.     

What is comparable in comparative cognition?

To understand how complex, or 'advanced' various forms of cognition are, and to compare them between species for evolutionary studies, we need to understand the diversity of neural-computational mechanisms that may be involved, and to identify the genetic changes that are necessary to mediate changes in cognitive functions. The same overt cognitive capacity might be mediated by entirely different neural circuitries in different species, with a many-to-one mapping between behavioural routines, computations and their neural implementations. Comparative behavioural research needs to be complemented with a bottom-up approach in which neurobiological and molecular-genetic analyses allow pinpointing of underlying neural and genetic bases that constrain cognitive variation. Often, only very minor differences in circuitry might be needed to generate major shifts in cognitive functions and the possibility that cognitive traits arise by convergence or parallel evolution needs to be taken seriously. Hereditary variation in cognitive traits between individuals of a species might be extensive, and selection experiments on cognitive traits might be a useful avenue to explore how rapidly changes in cognitive abilities occur in the face of pertinent selection pressures.     

Ecophysiology of cognition: How do environmentally induced changes in physiology affect cognitive performance?

Cognitive performance is based on brain functions, which have energetic demands and are modulated by physiological parameters such as metabolic hormones. As both environmental demands and environmental energy availability change seasonally, we propose that cognitive performance in free‐living animals might also change seasonally due to phenotypic plasticity. This is part of an emerging research field, the ‘ecophysiology of cognition’: environmentally induced changes in physiological traits, such as blood glucose and hormone levels, are predicted to influence cognitive performance in free‐living animals. Energy availability for the brain might change, and as such cognition, with changing energetic demands (e.g. reproduction) and changes of energy availability in the environment (e.g. winter, drought). Individuals spending more energy than they can currently obtain from their environment (allostatic overload type I) are expected to trade off energy investment between cognition and other life‐sustaining processes or even reproduction. Environmental changes reducing energy availability might thus impair cognition. However, selection pressures such as predation risk, mate choice or social demands may act on the trade‐off between energy saving and cognition. We assume that different environmental conditions can lead to three different trade‐off outcomes: cognitive impairment, resilience or enhancement. Currently we cannot understand these trade‐offs, because we lack information about changes in cognitive performance due to seasonal changes in energy availability and both the resulting changes in homeostasis (for example, blood glucose levels) and the associated changes in the mechanisms of allostasis (for example, hormone levels). Additionally, so far we know little about the fitness consequences of individual variation in cognitive performance. General cognitive abilities, such as attention and associative learning, might be more important in determining fitness than complex and specialized cognitive abilities, and easier to use for comparative study in a large number of species. We propose to study seasonal changes in cognitive performance depending on energy availability in populations facing different predation risks, and the resulting fitness consequences.     

Social competition as a driver of phenotype–environment correlations: implications for ecology and evolution

While it is universally recognised that environmental factors can cause phenotypic trait variation via phenotypic plasticity, the extent to which causal processes operate in the reverse direction has received less consideration. In fact individuals are often active agents in determining the environments, and hence the selective regimes, they experience. There are several important mechanisms by which this can occur, including habitat selection and niche construction, that are expected to result in phenotype–environment correlations (i.e. non-random assortment of phenotypes across heterogeneous environments). Here we highlight an additional mechanism – intraspecific competition for preferred environments – that may be widespread, and has implications for phenotypic evolution that are currently underappreciated. Under this mechanism, variation among individuals in traits determining their competitive ability leads to phenotype–environment correlation; more competitive phenotypes are able to acquire better patches. Based on a concise review of the empirical evidence we argue that competition-induced phenotype–environment correlations are likely to be common in natural populations before highlighting the major implications of this for studies of natural selection and microevolution. We focus particularly on two central issues. First, competition-induced phenotype–environment correlation leads to the expectation that positive feedback loops will amplify phenotypic and fitness variation among competing individuals. As a result of being able to acquire a better environment, winners gain more resources and even better phenotypes – at the expense of losers. The distinction between individual quality and environmental quality that is commonly made by researchers in evolutionary ecology thus becomes untenable. Second, if differences among individuals in competitive ability are underpinned by heritable traits, competition results in both genotype–environment correlations and an expectation of indirect genetic effects (IGEs) on resource-dependent life-history traits. Theory tells us that these IGEs will act as (partial) constraints, reducing the amount of genetic variance available to facilitate evolutionary adaptation. Failure to recognise this will lead to systematic overestimation of the adaptive potential of populations. To understand the importance of these issues for ecological and evolutionary processes in natural populations we therefore need to identify and quantify competition-induced phenotype–environment correlations in our study systems. We conclude that both fundamental and applied research will benefit from an improved understanding of when and how social competition causes non-random distribution of phenotypes, and genotypes, across heterogeneous environments.     

Multivariate selection shapes environment-dependent variation in the clonal morphology of a red seaweed

Within-individual strategies of variation (e.g., phenotypic plasticity) are particularly relevant to modular organisms, in which ramets of the same genetic individual may encounter diverse environments imposing diverse patterns of selection. Hence, measuring selection in heterogeneous environments is essential to understanding whether environment-dependent phenotypic change enhances the fitness of modular individuals. In sublittoral marine habitats, competition for light and space among modular taxa generates extreme patchiness in resource availability. Little is known, however, of the potential for plasticity within individuals to arise from spatially-variable selection in such systems. We tested whether plasticity enhances genet-level fitness in Asparagopsis armata, a clonal seaweed in which correlated traits mediate morphological responses to variation in light. Using the capacity for rapid, clonal growth to measure fitness, we identified aspects of ramet morphology targeted by selection in two contrasting light environments and compared patterns of selection across environments. We found that directional selection on single traits, coupled with linear and nonlinear selection on multi-trait interactions, shape ramet morphology within environments and favor different phenotypes in each. Evidence of environment-dependent, multivariate selection on correlated traits is novel for any marine modular organism and demonstrates that seaweeds, such as A. armata, may potentially adapt to environmental heterogeneity via plasticity in clonal morphology.     

The effect of environmental harshness on neurogenesis: a large-scale comparison

Harsh environmental conditions may produce strong selection pressure on traits, such as memory, that may enhance fitness. Enhanced memory may be crucial for survival in animals that use memory to find food and, thus, particularly important in environments where food sources may be unpredictable. For example, animals that cache and later retrieve their food may exhibit enhanced spatial memory in harsh environments compared with those in mild environments. One way that selection may enhance memory is via the hippocampus, a brain region involved in spatial memory. In a previous study, we established a positive relationship between environmental severity and hippocampal morphology in food-caching black-capped chickadees (Poecile atricapillus). Here, we expanded upon this previous work to investigate the relationship between environmental harshness and neurogenesis, a process that may support hippocampal cytoarchitecture. We report a significant and positive relationship between the degree of environmental harshness across several populations over a large geographic area and (1) the total number of immature hippocampal neurons, (2) the number of immature neurons relative to the hippocampal volume, and (3) the number of immature neurons relative to the total number of hippocampal neurons. Our results suggest that hippocampal neurogenesis may play an important role in environments where increased reliance on memory for cache recovery is critical.     

A framework for the study of genetic variation in migratory behaviour

Evolutionary change results from selection acting on genetic variation. For migration to be successful, many different aspects of an animal’s physiology and behaviour need to function in a co-coordinated way. Changes in one migratory trait are therefore likely to be accompanied by changes in other migratory and life-history traits. At present, we have some knowledge of the pressures that operate at the various stages of migration, but we know very little about the extent of genetic variation in various aspects of the migratory syndrome. As a consequence, our ability to predict which species is capable of what kind of evolutionary change, and at which rate, is limited. Here, we review how our evolutionary understanding of migration may benefit from taking a quantitative-genetic approach and present a framework for studying the causes of phenotypic variation. We review past research, that has mainly studied single migratory traits in captive birds, and discuss how this work could be extended to study genetic variation in the wild and to account for genetic correlations and correlated selection. In the future, reaction-norm approaches may become very important, as they allow the study of genetic and environmental effects on phenotypic expression within a single framework, as well as of their interactions. We advocate making more use of repeated measurements on single individuals to study the causes of among-individual variation in the wild, as they are easier to obtain than data on relatives and can provide valuable information for identifying and selecting traits. This approach will be particularly informative if it involves systematic testing of individuals under different environmental conditions. We propose extending this research agenda by using optimality models to predict levels of variation and covariation among traits and constraints. This may help us to select traits in which we might expect genetic variation, and to identify the most informative environmental axes. We also recommend an expansion of the passerine model, as this model does not apply to birds, like geese, where cultural transmission of spatio-temporal information is an important determinant of migration patterns and their variation.     

Sexual selection and the evolution of male and female cognition: A test using experimental evolution in seed beetles*

The mating system is thought to be important in shaping animal intelligence and sexual selection has been depicted as a driver of cognitive evolution, either directly by promoting superior cognitive ability during mate competition, or indirectly via genic capture of sexually selected traits. However, it remains unclear if intensified sexual selection leads to general improvements in cognitive abilities. Here, we evaluated this hypothesis by applying experimental evolution in seed beetles. Replicate lines, maintained for 35 generations of either enforced monogamy (eliminating sexual selection) or polygamy, were challenged to locate and discriminate among mates (male assays) or host seeds (female assays) in a spatial chemosensory learning task. All lines displayed learning between trials. Moreover, polygamous males outperformed monogamous males, providing evidence that sexual selection can lead to the evolution of improved male cognition. However, there were no differences between regimes in rates of male learning, and polygamous females showed no improvement in host search and even signs of reduced learning. Hence, sexual selection increased performance in cognitively demanding mate search, but it did not lead to general increases in cognitive abilities. We discuss the possibility that sexually antagonistic selection is an important factor maintaining abundant genetic variation in cognitive traits.     

Effects of the demographic transition on the genetic variances and covariances of human life‐history traits

The recent demographic transitions to lower mortality and fertility rates in most human societies have led to changes and even quick reversals in phenotypic selection pressures. This can only result in evolutionary change if the affected traits are heritable, but changes in environmental conditions may also lead to subsequent changes in the genetic variance and covariance (the G matrix) of traits. It currently remains unclear if there have been concomitant changes in the G matrix of life‐history traits following the demographic transition. Using 300 years of genealogical data from Finland, we found that four key life‐history traits were heritable both before and after the demographic transition. The estimated heritabilities allow a quantifiable genetic response to selection during both time periods, thus facilitating continued evolutionary change. Further, the G matrices remained largely stable but revealed a trend for an increased additive genetic variance and thus evolutionary potential of the population after the transition. Our results demonstrate the validity of predictions of evolutionary change in human populations even after the recent dramatic environmental change, and facilitate predictions of how our biology interacts with changing environments, with implications for global public health and demography.     

Grow with the flow: a latitudinal cline in physiology is associated with more variable precipitation in Erythranthe cardinalis

Local adaptation is commonly observed in nature: organisms perform well in their natal environment, but poorly outside it. Correlations between traits and latitude, or latitudinal clines, are among the most common pieces of evidence for local adaptation, but identifying the traits under selection and the selective agents is challenging. Here, we investigated a latitudinal cline in growth and photosynthesis across 16 populations of the perennial herb Erythranthe cardinalis (Phrymaceae). Using machine learning methods, we identify interannual variation in precipitation as a likely selective agent: southern populations from more variable environments had higher photosynthetic rates and grew faster. We hypothesize that selection may favour a more annualized life history – grow now rather than save for next year – in environments where severe droughts occur more often. Thus, our study provides insight into how species may adapt if Mediterranean climates become more variable due to climate change.     

Brain size predicts behavioural plasticity in guppies (Poecilia reticulata): An experiment

Understanding how animal personality (consistent between‐individual behavioural differences) arises has become a central topic in behavioural sciences. This endeavour is complicated by the fact that not only the mean behaviour of individuals (behavioural type) but also the strength of their reaction to environmental change (behavioural plasticity) varies consistently. Personality and cognitive abilities are linked, and we suggest that behavioural plasticity could also be explained by differences in brain size (a proxy for cognitive abilities), since accurate decisions are likely essential to make behavioural plasticity beneficial. We test this idea in guppies (Poecilia reticulata), artificially selected for large and small brain size, which show clear cognitive differences between selection lines. To test whether those lines differed in behavioural plasticity, we reared them in groups in structurally enriched environments and then placed adults individually into empty tanks, where we presented them daily with visual predator cues and monitored their behaviour for 20 days with video‐aided motion tracking. We found that individuals differed consistently in activity and risk‐taking, as well as in behavioural plasticity. In activity, only the large‐brained lines demonstrated habituation (increased activity) to the new environment, whereas in risk‐taking, we found sensitization (decreased risk‐taking) in both brain size lines. We conclude that brain size, potentially via increasing cognitive abilities, may increase behavioural plasticity, which in turn can improve habituation to novel environments. However, the effects seem to be behaviour‐specific. Our results suggest that brain size likely explains some of the variation in behavioural plasticity found at the intraspecific level.     

Study of Natural Genetic Variation in Early Fitness Traits Reveals Decoupling Between Larval and Pupal Developmental Time in <Emphasis Type="Italic">Drosophila melanogaster</Emphasis>

Characterizing the relationships between genotype and phenotype for developmental adaptive traits is essential to understand the evolutionary dynamics underlying biodiversity. In holometabolous insects, the time to reach the reproductive stage and pupation site preference are two such traits. Here we characterize aspects of the genetic architecture for Developmental Time (decomposed in Larval and Pupal components) and Pupation Height using lines derived from three natural populations of Drosophila melanogaster raised at two temperatures. For all traits, phenotypic differences and variation in plasticity between populations suggest adaptation to the original thermal regimes. However, high variability within populations shows that selection does not exhaust genetic variance for these traits. This could be partly explained by local adaptation, environmental heterogeneity and modifications in the genetic architecture of traits according to environment and ontogenetic stage. Indeed, our results show that the genetic factors affecting Developmental Time and Pupation Height are temperature-specific. Varying relationships between Larval and Pupal Developmental Time between and within populations also suggest stage-specific modifications of genetic architecture for this trait. This flexibility would allow for a somewhat independent evolution of adaptive traits at different environments and life stages, favoring the maintenance of genetic variability and thus sustaining the traits’ evolvabilities.     

Brain size affects responsiveness in mating behaviour to variation in predation pressure and sex ratio

Despite ongoing advances in sexual selection theory, the evolution of mating decisions remains enigmatic. Cognitive processes often require simultaneous processing of multiple sources of information from environmental and social cues. However, little experimental data exist on how cognitive ability affects such fitness‐associated aspects of behaviour. Using advanced tracking techniques, we studied mating behaviours of guppies artificially selected for divergence in relative brain size, with known differences in cognitive ability, when predation threat and sex ratio was varied. In females, we found a general increase in copulation behaviour in when the sex ratio was female biased, but only large‐brained females responded with greater willingness to copulate under a low predation threat. In males, we found that small‐brained individuals courted more intensively and displayed more aggressive behaviours than large‐brained individuals. However, there were no differences in female response to males with different brain size. These results provide further evidence of a role for female brain size in optimal decision‐making in a mating context. In addition, our results indicate that brain size may affect mating display skill in male guppies. We suggest that it is important to consider the association between brain size, cognitive ability and sexual behaviour when studying how morphological and behavioural traits evolve in wild populations.