Background: The evolution of exaggerated vocal signals in anuran species is an important topic. Males and females have both evolved the ability to discriminate communication sounds. However, the nature of sexual dimorphism in cognition and sensory discrimination and in the evolution and limitation of sexual signal exaggeration remain relatively unexplored.

Methods: In the present study, we used male calls of varied complexity in the serrate-legged small treefrog, Kurixalus odontotarsus, as probes to investigate how both sexes respond to variations in call complexity and how sex differences in signal discrimination play a role in the evolution of sexual signal exaggeration. The compound calls of male K. odontotarsus consist of a series of one or more harmonic notes (A notes) which may be followed by one or more short broadband notes (B notes).

Results: Male playback experiments and female phonotaxis tests showed that increasing the number of A notes in stimulus calls elicits increased numbers of response calls by males and increases the attractiveness of the stimulus calls to females. The addition of B notes, however, reduces male calling responses. Moreover, call stimuli which contain only B notes suppress spontaneous male calling responses. Phonotaxis experiments show that females prefer calls with greater numbers of A notes and calls containing both A notes and B notes, but do not prefer calls with only B notes.

Discussion: Male-male competition and female choice appear to have played different roles in the evolution and limitation of signal complexity in K. odontotarsus. These results provide new insights into how exaggerated compound signals evolve and how signal complexity may be limited in anurans.