Hmmmm.

What kind of matte varnish?

It almost sounds/looks like the varnish is drying a bit before hitting the surface. If this is the case, it's the little 'pebbles' of dry varnish that is giving it this frosted look.

There are a few things that it could be, one of them being temp/humidity another being too high an air pressure and even what and how you thinned it(if that applies).