Why do people argue that Brazil isn't in latin America?

I think there is a bit of confusion here. Brazil is one of the few countries in Latin America that the official language isn't Spanish, but I have never heard anyone argue that it isn't in Latin America. Portuguese and French (there are some French speaking countries here as well) both are just as Latin as Spanish.