RDC XP at work to W2K3 at home (slow refresh time)

I am trying to connect to my W2K3 (all updates on all systems) system at my
house over cable connection and am doing so from work.
I am noticing that the refresh rate on my connection can be horrible at
times! It seems strange that when I minimize or refresh it gets bad.
I am able to connect to my system quickly from the office, but when I try to
do something it lags. I look at the processing power it is at 268K or so. I
have 512MB RAM on a P4 2.4GHz hyperthreading system on the W2K3SP1 server. I
am able to connect over my linksys 54G wireless router that is physically
connected via a network connection to a 100MB switch that allows me to go
from the 1st floor to the 3rd floor with a CAT5e cable. I have lowered the
RDC speed to LAN10 to 28.8 connection and it still does not make a difference
for refresh. The server does not see me to be CHUGGING as I am running a
simple web site application and it will not take up much processor which
might be the problem. No one is using the network or so it seems, just
myself connected.
Not sure what is causing the problem. Any ideas?