If you wash the oils from your model, then you won't find any flaking when sanding, as long it is done properly and with patience.

However, when you say vigorous sanding, is where I do not understand. Primer has to be applied in very thin layers (to preserve details) so there is no need for such strenuous force to be applied by the sanding paper, because it will be easy to remove. By sanding vigorously, then you may find flaking not only on Vallejo, but pretty much any other primer. If I may ask, why would you need to sand primer vigorously? The only I can think of, is if you apply a thick layer of primer, which will hide the details, the brand won't matter, it is guaranteed that it will peel off, because of the thickness, that is the only scenario that I could think of vigorous sanding.