Quitting Dairy Has These Effects on Your Skin The worst skin irritant among dairy is skim milk due to the whey protein, hormones and sugars. In fact, skim milk has more sugar in it than whole milk. Avnee Shah, Dermatologist Sarika Snell, Dermatologist Of course, you'll need to give it some time before you see any results. Sarika Snell, Dermatologist The idea of giving up milk or cheese might seem like an impossible task to some people, but fortunately there are plenty of vegan options to try along your dairy-detox journey. If you don't know where to start, try substituting almond or soy milk for regular milk and give vegan cheese a try!