How do I justify white people existing in my fantasy setting

Considering that white people are a rarity in fiction and that black people are the default I feel that the presence of white people need to be justified otherwise my reader might feel its strange and make them uncomfortable.

​

I mean of course white people can be made to come from another continent but then I would need a explanation on why there here mixing with the regular people.

​

I feel any story that doesn't explain why white people exist is lazy. While any story that doesn't explain black people is fine because there's no reason to.