Networking Slows Down after working hours

We have a server using Win 2003 and the rest of the workstatiosn are Win XP.
I understand that the more PC acceses the server, the slower the network
becomes. However, whenever it's after 6pm, when most of the PCs are logged
off, I find it extremely slow in accessing the server even when i'm the only
one left in the office. When i restart my PC, everything becomes normal
again. If this normal? Is there any settings i need to set on the server?
Is there any patching that i need to do on my server? This is becoming a
problem for everyone working beyond 6pm!!!


Skit