They say it's to avoid birth defects and other genetic abnormalities, but in fact such things are not apparent for several generations.
More likely it is a matter of exogamy: bringing a new person into the community brings new ideas, new fashions, new technology -- vital for cultural growth.
Plus the daily dynamics of sharing a household with someone while you're growing up makes them too familiar to be erotic.
In the U.S., marriage or sexual relations are forbidden between biological siblings, parents and children, and aunts/uncles/nieces/nephews.
First cousins (your uncle or aunt's child) are usually forbidden, and sometimes first cousins once removed (your parents' cousins) and second cousins (your cousins' children).
Most states also prohibit adopted parents, children, and siblings, even though genetics are not an issue.
The rules have gotten more strict during the last century. In the Victorian Era, it was not uncommon to marry your uncle or your cousin.
I don't know if erotic activity between brothers is more common than between brother and sister, but it certainly is less stigmatized.
A lot of guys characterize their same-sex activity as "play" or "fooling around," not real sex, especially in adolescence, so they have no qualms against doing things with their brother that they would never think of with their sister.
When young guys call you Daddy, they're referring to the social distance of your age difference, not imagining sex with their real father.
Father-son erotic activity is much less common, and more stigmatized, than brother-brother, probably because the parental dynamic is so different. Nurturing and protecting you is not erotic. Why would you be attracted to someone who changed your diapers when you were a baby?
I've never heard of a father and son having a sexual relationship in real life. Alan hooked up with a father and son at the same time, but they never actually interacted with each other.