If they are aggressive to some plastics, it just means better adhesion—usually.
It can also cause them to leech pigment out of the plastic, depending on the kit manufacturer. This is not normally a problem, but can be if you try to kill two birds with one stone by using white primer to, for example, paint a white tank interior or as the base colour for a UN vehicle.
These photos are of an Asuka Sherman tank into which I grafted a driver's position from an Academy M36 tank destroyer, as well as scratchbuilt a bunch of stuff. Here are the same bits unpainted:
The white paint is Vallejo primer from an aerosol can. As you can see, the dark green parts of the Academy interior and the Italeri box next to the instrument panel are distinctly pale green, but the medium green parts (which are Asuka) ended up just as white as the white plastic card. This is not because the paint didn't cover well enough, it's because the paint drew pigment out of the plastic. I first noticed this happening 20+ years ago with white car primer on a Tamiya M113 that I had put single-link tracks in black plastic on. After spraying, the whole hull was pale green but the tracks were white, so the primer
did cover as it was supposed to. But my first thought was that it hadn't, so I added a second coat, which didn't really solve the problem, just made it a bit less. IIRC I had to put four or five coats on before the vehicle was the UN white I was after.
On the Sherman, I ignored the pale green because the whole interior was supposed to be dirty and rusted, so it didn't show under all the other painting:
However, for a good, white colour you would want to overspray the primer with normal white paint.