You're absolutely right: the concept of "Latin American" is a North American invention. The rest were originally called "iberoamericano" in the two mother countries, since Portugal and Spain make up the Iberian peninsula.
The iberoamericanos can be further divided in "hispanoamericanos" and "lusoamericanos" according to their language. "Luso" means Portuguese.
So it seems to me that you only consider hispanoamericanos to be "real" iberoamericanos, and that, of course, makes no sense. Today's iberoamericanos see "latino" as a synonym (and a shorter term) for iberoamericano. Brazilians too.
I think you see "latino" as a race rather than an ethnicity: millions of people from Argentina and Uruguay to Colombia and Venezuela are very white people of Spanish, Portuguese, German and Italian descent. They would never be shown on your favorite "latin" sites, yet they are just as latin as the millions of blonde-haired, blue-eyed Brazilians with German, Jewish, Polish and Russian surnames, or the millions of blackblackblack 100% African-looking men in the Americas.
Or to say it otherwise: Jeff Palmer is from Argentina, the Rockland brothers have Guatemalan parents. Latino enough to you?