odd issue with 2003 Server R2 x64 and remote desktop-mouse clicks delayed by 5-10 sec

This problem only seems to be when i'm in the remote session window..
i have no other problems with other servers on my network..

The task manager shows very little cpu utilization, very low network
utilization..

This is an almost bare installation too.

Basically.. if i click with the mouse.. it will sortof "pause" for
about 5-10 seconds before moving on.. if i press a key on the keyboard
it forces things along.

I dont see this behavior when logged in directly..

Anyone ever run into this?

Thanks