The idea of an evolutionarily stable strategy (or ESS) was introduced by John Maynard Smith (with impetus from George Price) in an essay on "Game Theory and the Evolution of Fighting". It was also used by Robert Axelrod in his work described in The Evolution of Cooperation, in the context of strategies used to compete at the iterated prisoner's dilemma.
An ESS depends on the idea of invasion, where a population of strategy-X players is visited by a strategy-Y player. The new player is said to invade if, following strategy Y, he scores better than the average strategy-X player. Assuming players are able to choose and switch strategies, this would induce the indigenous population to start switching to strategy Y. In many cases there are diminishing returns for the later adopters, and what follows is an equilibrium ratio of strategy-X players to strategy-Y players.
A strategy X is evolutionarily stable if there is no strategy Y that can invade it. That is, anybody bringing a new strategy into a population of strategy-X players will fare no better on average than the X players are already doing. (See the closely-related Nash equilibrium.)
The recent, controversial science of sociobiology attempts to explain animal and human behavior and social structures, largely in terms of evolutionarily stable strategies. For example, in one well-known 1995 paper by Linda Mealey, Sociopathy (chronic antisocial/criminal behavior) is explained as a combination of two such strategies.
