No, it really isn't; the Bantus only came down from central Africa in the late 18th century, the European settlers have been in the Cape since 1652 and in the interior of southern Africa since the 1690s. Any way you look at it Whites pre-date the Black presence in the area now known as 'South Africa'.
Actually, technically, my ancestors (and yours)were in Africa just as long as any Black man's, at least if you believe in the out of Africa theory. Therefore we can exclude them from Europe but they can't exclude us from Africa, we have a history there independent of any conquests in the last 500 years. And as for Europe being the White man's continent, perhaps it is but we are not aboriginal Europeans any more than White Africans are aboriginal Africans (although they are indigenous), because the Basques were in Europe before us Indo-Europeans.
What about Northern Berbers and Northern Egyptians? Africa isn't the continent of the blacks as Asia isn't the continent of the Mongols or hindus... What are white Russians doing in Siberia?