You have not noticed that there are various shades of most races?

Caucasian/White can be almost paper white to a deep tan color (normal).

African Americans can be almost white (in the old days, they could "pass" for white, and some did) through various shades of brown to almost black.

Asians can be a yellowish-white tone to a dark brown.

Middle Easterners can be from a light tan to an almost African-American dark brown.

And all other races have various shades, too.

Most of the world's people can be of any mixed race, so any color - Well not green or blue tones that I know of. That allows for numerous shades of colors, no matter what race they are or "go by".

Race and color are different things, and there is no hard and fast rule.

For example, ethnographically speaking, Asian Indians are Caucasians, although their skin color can be as dark or darker than many people of African ancestry.

The principle focus of the phrase, though, most likely relates to the Hispanic community in the United States. In the 1960's, you would see people make reference to "black, white and brown" as a means of expanding the reach of their remarks beyond the Black-White division.

It actually doesn't work as a synonym for Hispanic, though, since Hispanics can be of any race.

One example is Caucasian or white as your color. In most surveys they will then ask if you are Hispanic. That is another race but the same color. You can be Jewish which is a religion and a race but you are the same color as other Europeans.