Best speed for file transfers?

Hi all,

I need to transfer lots of large (4-5G)files
from several computers to one over a
network. I have access to 1G switches
and such so what would be the best
way to accomplish this, ftp, cp, samba,
nfs, zfs etc....? There is a Windows
machine in the mix but it only has a
few files in the mix of files I need to
transfer. The others are all Linux. Is
one brand/technology of hardware
any better than any other?

thanks,
charles.....

On Thu, 13 Sep 2007 14:08:15 +0000, ***** charles wrote:

> I need to transfer lots of large (4-5G)files from several computers to
> one over a network. I have access to 1G switches and such so what
> would be the best way to accomplish this, ftp, cp, samba, nfs, zfs
> etc....? [...]

Any of the above will work as long as the receiving filesystem can handle
files larger than 2GB. You will be limited by the slowest link in the
network between your new host and each system. I usually use c-kermit on
both ends of Unix connections to transfer lots of files. It can be used
over telnet or ssh to do this. If it matters to you c-kermit is not OSS.
The version for Micro$oft systems is not "free".

http://www.columbia.edu/kermit/ck80.html
http://www.columbia.edu/kermit/k95.html

If you use it and like it you might want to purchase the manual:

http://www.columbia.edu/kermit/manuals.html

Use this URL to read the C-Kermit 7.0/8.0 license:

ftp://kermit.columbia.edu/kermit/f/COPYING.TXT

Cross-post to: comp.protocols.kermit.misc

Gene (e-mail: gene \a\t eracc \d\o\t com)
--
Mandriva Linux release 2007.1 (Official) for i586
Got Rute? http://www.anrdoezrs.net/email-25465...sbn=0130333514
ERA Computers & Consulting - http://www.eracc.com/
Preloaded PCs - eComStation, Linux, FreeBSD, OpenServer & UnixWare

On Sep 13, 10:08 pm, "***** charles" <m...@mine.com> wrote:
> Hi all,
>
> I need to transfer lots of large (4-5G)files
> from several computers to one over a
> network. I have access to 1G switches
> and such so what would be the best
> way to accomplish this, ftp, cp, samba,
> nfs, zfs etc....? There is a Windows
> machine in the mix but it only has a
> few files in the mix of files I need to
> transfer. The others are all Linux. Is
> one brand/technology of hardware
> any better than any other?

It depends on what you mean by "best". But if by "best" you mean
"fastest" then ftp is indeed fastest since ftp doesn't really have a
"protocol" on the file transfer channel. Ftp negotiates the transfer
on the control channel and then simply opens another socket and blasts
the file directly into that socket - no headers or tails or crc or
checksum added to the traffic so no additional overhead beyond TCP/IP.

On Sep 14, 10:46 am, "slebet...@yahoo.com" <slebet...@gmail.com>
wrote:
> On Sep 13, 10:08 pm, "***** charles" <m...@mine.com> wrote:
>
> > Hi all,
>
> > I need to transfer lots of large (4-5G)files
> > from several computers to one over a
> > network. I have access to 1G switches
> > and such so what would be the best
> > way to accomplish this, ftp, cp, samba,
> > nfs, zfs etc....? There is a Windows
> > machine in the mix but it only has a
> > few files in the mix of files I need to
> > transfer. The others are all Linux. Is
> > one brand/technology of hardware
> > any better than any other?
>
> It depends on what you mean by "best". But if by "best" you mean
> "fastest" then ftp is indeed fastest since ftp doesn't really have a
> "protocol" on the file transfer channel. Ftp negotiates the transfer
> on the control channel and then simply opens another socket and blasts
> the file directly into that socket - no headers or tails or crc or
> checksum added to the traffic so no additional overhead beyond TCP/IP.

Just out of interest, how does this compare with rsync using data
compression - obviously it depends on the files being transferred
(whether they're already compressed and so on) - just wondered how
they compare as a rule of thumb.

"(E-Mail Removed)" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed) oups.com...
> On Sep 13, 10:08 pm, "***** charles" <m...@mine.com> wrote:
> > Hi all,
> >
> > I need to transfer lots of large (4-5G)files
> > from several computers to one over a
> > network. I have access to 1G switches
> > and such so what would be the best
> > way to accomplish this, ftp, cp, samba,
> > nfs, zfs etc....? There is a Windows
> > machine in the mix but it only has a
> > few files in the mix of files I need to
> > transfer. The others are all Linux. Is
> > one brand/technology of hardware
> > any better than any other?
>
> It depends on what you mean by "best". But if by "best" you mean
> "fastest" then ftp is indeed fastest since ftp doesn't really have a
> "protocol" on the file transfer channel. Ftp negotiates the transfer
> on the control channel and then simply opens another socket and blasts
> the file directly into that socket - no headers or tails or crc or
> checksum added to the traffic so no additional overhead beyond TCP/IP.

Are there any good gui based ftp programs out there for linux/windows?

thanks,
charles.....

Hello,

Andy Ruddock wrote:

> On Sep 14, 10:46 am, "slebet...@yahoo.com" <slebet...@gmail.com>
> wrote:

>> It depends on what you mean by "best". But if by "best" you mean
>> "fastest" then ftp is indeed fastest since ftp doesn't really have a
>> "protocol" on the file transfer channel. Ftp negotiates the transfer
>> on the control channel and then simply opens another socket and
>> blasts the file directly into that socket - no headers or tails or
>> crc or checksum added to the traffic so no additional overhead beyond
>> TCP/IP.
>
> Just out of interest, how does this compare with rsync using data
> compression - obviously it depends on the files being transferred
> (whether they're already compressed and so on) - just wondered how
> they compare as a rule of thumb.

rsync will check existence of files on both sides first, then dates,
then parts of files, whether they have changed. It does so pretty
efficiently. If you know everything will have to be transmitted, it
will probably not give any advantage to use rsync, but it won't lose
too much either, so it can be a safe default even in this case. rsync
uses about the same commandline semantics as cp, which might be easier
to use than some ftp script.

Bernd Strieder

On 2007-09-14, ***** charles <(E-Mail Removed)> wrote:
>> It depends on what you mean by "best". But if by "best" you mean
>> "fastest" then ftp is indeed fastest since ftp doesn't really have a
>> "protocol" on the file transfer channel. Ftp negotiates the transfer
>> on the control channel and then simply opens another socket and blasts
>> the file directly into that socket - no headers or tails or crc or
>> checksum added to the traffic so no additional overhead beyond TCP/IP.
>
> Are there any good gui based ftp programs out there for linux/windows?

For Linux, use mc (Midnight Commander). It's text, but visual.
For Windows, get FileZilla.


--
There is an art, it says, or rather, a knack to flying.
The knack lies in learning how to throw yourself at the ground and miss.
Douglas Adams

Rikishi 42 wrote:
> For Windows, get FileZilla.

For Linux too, now.

--
As we enjoy great advantages from inventions of others, we should be
glad of an opportunity to serve others by any invention of ours;
and this we should do freely and generously.
--Benjamin Franklin

On 2007-09-14, johnny bobby bee <useraddshine-(E-Mail Removed)> wrote:

>> For Windows, get FileZilla.
>
> For Linux too, now.

Haven't looked for that, for a while.
Thanks for the info.


--
There is an art, it says, or rather, a knack to flying.
The knack lies in learning how to throw yourself at the ground and miss.
Douglas Adams

On 2007-09-13, ERACC <(E-Mail Removed)> wrote:
: On Thu, 13 Sep 2007 14:08:15 +0000, ***** charles wrote:
:
:> I need to transfer lots of large (4-5G)files from several computers to
:> one over a network. I have access to 1G switches and such so what
:> would be the best way to accomplish this, ftp, cp, samba, nfs, zfs
:> etc....? [...]
:
: Any of the above will work as long as the receiving filesystem can handle
: files larger than 2GB. You will be limited by the slowest link in the
: network between your new host and each system. I usually use c-kermit on
: both ends of Unix connections to transfer lots of files. It can be used
: over telnet or ssh to do this. If it matters to you c-kermit is not OSS.
: The version for Micro$oft systems is not "free".
:
: http://www.columbia.edu/kermit/ck80.html
: http://www.columbia.edu/kermit/k95.html
:
: If you use it and like it you might want to purchase the manual:
:
: http://www.columbia.edu/kermit/manuals.html
:
: Use this URL to read the C-Kermit 7.0/8.0 license:
:
: ftp://kermit.columbia.edu/kermit/f/COPYING.TXT
:
: Cross-post to: comp.protocols.kermit.misc
:
Note that support for large files (> 2GB) in the current Kermit
releases is limited to pure 64-bit platforms like Tru64 Unix.

Preview versions of the next C-Kermit release are available here:

http://www.columbia.edu/kermit/ckdaily.html

This version supports transfer of large files, both with Kermit
protocol and FTP, on many platforms, listed here:

http://www.columbia.edu/kermit/ckc212.html

As to speed, you can measure it yourself but modern Kermit is
a lot faster than many people imagine. It has been adapted to
today's relatively transparent and error-free transports (TCP/IP,
error-correcting modems, big buffers, etc) to go at speeds close
to the connection speed itself; sometimes a bit faster due to
a simple built-in compression method.

- Frank