how to transfer small files?

Hi all, I have to checkout 100000 files from an internal repository
and the operation is very slow. Both server and client nics are
gigabit full duplex, no errors detected by any side.

I believe the problem is the tcp overhead created by the tcp protocol
for each small file.

Could you suggest me another protocol or a solution to checkout a huge
quantity of small files?
Maybe I could force to use udp but it's not a solution because it
doesn't grant the integrity..


Regards


Marco Strullato

Marco skrev:
> Hi all, I have to checkout 100000 files from an internal repository
> and the operation is very slow. Both server and client nics are
> gigabit full duplex, no errors detected by any side.
>
> I believe the problem is the tcp overhead created by the tcp protocol
> for each small file.
>
> Could you suggest me another protocol or a solution to checkout a huge
> quantity of small files?
> Maybe I could force to use udp but it's not a solution because it
> doesn't grant the integrity..
>
>
> Regards
>
>
> Marco Strullato

It is most likely the tcp overhead causing you the slow transfer as you
say. Cant you zip the files first into one large archive and then
transfer it?

Cheers,
Tobias Skytte

Unfortunately the operation is a checkout so I can not compress
files...


> It is most likely the tcp overhead causing you the slow transfer as you
> say. Cant you zip the files first into one large archive and then
> transfer it?


Marco Strullato

On Tue, 7 Oct 2008 06:48:54 -0700 (PDT), I waved a wand and this
message magically appears in front of Marco:

> Unfortunately the operation is a checkout so I can not compress
> files...

If you're using cvs, there's the -z9 option to compress files.
--
http://www.munted.org.uk

Fearsome grindings.

On Tue, 07 Oct 2008 20:01:38 -0400, Maxwell Lol <(E-Mail Removed)> wrote:

>Alex Buell <(E-Mail Removed)> writes:
>
>> On Tue, 7 Oct 2008 06:48:54 -0700 (PDT), I waved a wand and this
>> message magically appears in front of Marco:
>>
>>> Unfortunately the operation is a checkout so I can not compress
>>> files...
>>
>> If you're using cvs, there's the -z9 option to compress files.
>
>Does that compress individual files? If so, I'm not sure that will
>help that much. You still have the overhead of opening the
>connection.
>
>Marco - sometimes you have to start the process, and walk away from the
>computer. Come back the next morning......

That's too easy, want it now! )

Grant.
--
http://bugsplatter.id.au/

Marco skrev:
> Unfortunately the operation is a checkout so I can not compress
> files...
>
>
>> It is most likely the tcp overhead causing you the slow transfer as you
>> say. Cant you zip the files first into one large archive and then
>> transfer it?
>
>
> Marco Strullato

Ok, is it Subversion or something? Maybe checkout the files locally
first then zip them (the entire directory) and transfer and unzip. In
svn there are commandline tools to do that, so you can automate it with
a script.

Regards,
Tobias