TIME_WAIT in NetStat

If I understand correctly, the TIME_WAIT Status when you do a NETSTAT on the
server indicates that the connection has been closed, and it is waiting to
clear. Is that right? If so, we have a lot of SQL Server connections that
are in TIME_WAIT status mode. We may have well over 100 or so of these, and
they do not clear out very fast. My understanding (and what I've seen from
the settings) is that this is because of connection pooling. It will keep
the connection there for 120 seconds, just in case it needs to be used
somewhere else, but these connections are staying open way beyond 120
seconds. Why would they stick around like that, and how do I get them to
clear out sooner? Could they potentially be causing problems by not
clearing out?

I don't know if this is a SQL Server issue, or a networking issue. I've
tried the SQL Server forum, and haven't got any answers as of yet, but
figured I'd post here too to see if anyone had any suggestions.

Thanks,
Jesse