There was always some sort of desire for land, and hunters and explorers etc had always headed inland. Large numbers of French Hugonout 'refugees' arrived in the Cape to escape religious persecution in their homeland, and many of these names (du Toit, Cronje, de Villiers etc) became 'Boer' names. There is a town called 'Franschoek' in the Cape - meaning 'French Corner'.
The real drive to leave the Cape, however, did not come until after the British occupation, and the resultant ending of slavery. This was seen as liberal nonsense by many of the more unpleasant Boers and they decided to head off, rather than live under British rule.
Even under British rule, the Cape never attracted anything approaching the sort of immigration levels which the USA or Australia did - maybe 5% as much if that. Cape Town was a tiny back water of a place.

Yes - some did. There were various groups of such people - known by names like 'Griquas' or (my favourite) 'the Bastards'. These groups settled in what is today the Northern Cape and also into Namibia. They either dominated or interbred with the Hottentots who were there already.

It is a very confused and fluid period and there was very little written record of some of the events.
There is still a large 'Cape Coloured' community all over the Cape Provinces, most of whom will have Afrikaans as a first language, though this might slowly change as English is increasingly adopted - who knows. While many of these Coloureds are staunch Christians, a sizable proportion are Muslim. This is due to the large numbers of Malay immigrants / indentured servants who entered the mix over the centuries.
The Cape Coloureds (that's what they call themselves - it's not a 'racist' term) were in a very difficult position during the Apartheid era, and a similarly awkward situation now under black rule. A Coloured friend of mine summed it up by saying: 'We weren't white enough for the Apartheid Government, and we're not Black enough for the ANC'.
Interestingly, there is a Cape Town-based Muslim terrorist group called 'PAGAD' (People Against Gangsterism And Drugs) which carried out several anti-Western atrocities a few years ago, the most famous of which being the bombing of the 'Planet Hollywood' restaurant at the Cape Town Waterfront.

What I am trying to say is that it is a very confused and complex situation and not easy to explain / pigeon hole the people of South Africa into groups very easily.