From a classical viewpoint, sex-specific behavior and physiological functions as well as the brain structures of mammals such as rats and mice, have been thought to be influenced by perinatal sex steroids secreted by the gonads. Sex steroids have also been thought to affect the differentiation of the sex-typical behavior of a few members of the avian order Galliformes, including the Japanese quail and chickens, during their development in ovo. However, recent mammalian studies that focused on the artificial shuffling or knockout of the sex-determining gene, Sry, have revealed that sex chromosomal effects may be associated with particular types of sex-linked differences such as aggression levels, social interaction, and autoimmune diseases, independently of sex steroid-mediated effects. In addition, studies on naturally occurring, rare phenomena such as gynandromorphic birds and experimentally constructed chimeras in which the composition of sex chromosomes in the brain differs from that in the other parts of the body, indicated that sex chromosomes play certain direct roles in the sex-specific differentiation of the gonads and the brain. In this article, we review the relative contributions of sex steroids and sex chromosomes in the determination of brain functions related to sexual behavior and reproductive physiology in mammals and birds.