If you wash the oils from your model, then you won't find any flaking when sanding, as long it is done properly and with patience.
However, when you say vigorous sanding, is where I do not understand. Primer has to be applied in very thin layers (to preserve details) so there is no need for...