Remove the "" in the set and you are fine... Don't know why, yet. But that's the answer
set CATALINA_OPTS=-Djava.awt.headless=true -Dfile.encoding=UTF-8 -server -Xms64m -Xmx512m -XX:PermSize=128m -XX:MaxPermSize=512m -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled
-- edit
Btw, I think you want the UseConcMarkSweepGC as well, otherwise the other 2 CMS parameters won't do a thing (and one of the 2 is only needed, depening on the version of the JVM)
See What does JVM flag CMSClassUnloadingEnabled actually do?
-- edit 2
This question was also asked for tomcat6 here: https://serverfault.com/questions/64894/setting-catalina-opts-for-tomcat6-on-windows-doesnt-work
Too bad the blogger doesn't explain why you don't need to use quotes in windows.
http://javahowto.blogspot.nl/2006/06/6-common-errors-in-setting-java-heap.html
But I think it's because with the quotes it's seen as one big argument. The only thing that I don't understand is why the jvisualvm pretends you set it correctly. But I guess the display of the arguments doesn't actually look at how it's set in the JVM, but only formats the arguments nicely.