This article from AMD will help you find how much performance degredation will this cause, they've even made some tweaks to reduce the performance hit caused by remote debugging on a Hotspot JVM, and I quote:
"We will also show how we applied these changes to improve the performance of the Hotspot JVM in the OpenJDK when it is started with debugging enabled but the debugger has not yet attached."
1.4.0 introduced "Full Speed Debugging" : In the previous version of HotSpot (prior to 1.4.0), when debugging was enabled, the program executed using only the interpreter. From 1.4.0 , the full performance advantage of HotSpot Technology was available to programs running with debugging enabled. The improved performance allows long running programs to be more easily debugged.
After 1.4.1 there are not performance related enhancement mentioned by oracle in JPDA area. Last one was done in JDK 1.4.0 for Java hotspot client VM and for Java Hotspot Server VM in JDK 1.4.1.
So it is safe to say that no performance improvement could be expected in debugging area from 1.4.1 to the latest version. Otherwise it would have been mentioned by oracle docs.