I don't know if you're completely wrong, but you're not really understanding the situation or its complexity. So you're certainly at least a bit wrong!
Your take is that men are assigned to be fa'afafine and that this is close to casteism. There are factual problems and less factual complexities to unpack here.
On the pure factual level, children are made fa'afafine if the family sees a fa'afafine in their behavior as a child. This isn't pure self-selection but it isn't entirely arbitrary either. Today, many fa'afafine actually self-select, even with resistance from their parents. Surveyed fa'afafine think positively of their childhood gender behavior whether or not parents supported it, suggesting that the primary driver of being fa'afafine may be internal, not external. If the child chooses to be fa'fafafine, how can it be a caste system? The evidence seems to suggest that fa'afafine know they are fa'afafine from very early childhood.
On the more complex political level, we face a number of questions:
-To what extent are we able to judge the moral rightness or wrongness of another society's gender systems? We hold egalitarian norms as a universal ideal, but our own societies still practice sharply gendered caste-style upbringing.
-Aren't all gender roles externally mandated? To what extent are gender systems a product of feedback between individually motivated behavior and social reinforcement of that behavior?
-You're not wrong to be worried about the idea of assigning a child to a gender role, but is it possible to avoid doing this? We already provide strong assignment to boys and girls. How are the fa'afafine different?
The fa'fafine are absolutely a different take on gender. They map to no similar concept in the West. We have no pervasive third gender; we do have gay men, a sexual orientation often accompanied by different gendered behavior, but gay men are generally involved in liaisons with other gay men, whereas fa'afafine rarely if ever liase with other fia'afafine, almost exclusively partnering with 'straight' (to appropriate the Western term) Samoan men.
You seem to have interpreted the fa'fafine as a proposed example of a 'better' system, some more socially just or equal setup. But that's not why I think they're interesting; I'm not trying to evaluate their moral rectitude or the egalitarianism of Samoan genders. The remarkable thing about the fa'afafine, and other third or liminal genders around the world, is the parallax they provide in demonstrating why our gender system isn't necessarily the normative one, and why gender is more fluid and diverse than we commonly conceive.
Although the evidence is weak, the fa'afafine and the fertility of their female relatives also provides interesting support for the sexual antagonism/'Johnny Depp' hypothesis.