can I rsync on UDP?

its a lame question, but let me ask you Gurus,

I have task of downloading files sized little over 1GB using rsync. It
is taking lot of time because of my lousy network speed. Just to test
I want to try rsync over UDP if possible, I am sure someone should
have done this trial, Can you share ideas about how to do it.

Thanks

Jagadeesh schrieb:
> I have task of downloading files sized little over 1GB using rsync. It
> is taking lot of time because of my lousy network speed. Just to test
> I want to try rsync over UDP if possible, I am sure someone should
> have done this trial, Can you share ideas about how to do it.

Most important question: why do you have lousy network speed? Or what do
you consider to be "lousy speed"?

There are protocols that you want to use over TCP because you can use
jumbo frames (e.g. NFS). Albeit TCPs overhead for small packets has to
be considered, overall it allows for much more throughput when you (are
able to) use the right parameters. Simplified: file size over 1GB
screams for TCP.

I could be wrong. But I think you got hooked up at the wrong side of the
problem.

Regards,
Benjamin

P.S. do you really have to use rsync? Only access via a rsync server?
There are cases were rsync really s*cks!

Joe Pfeiffer <(E-Mail Removed)> writes:

>Jagadeesh <(E-Mail Removed)> writes:

>> its a lame question, but let me ask you Gurus,
>>
>> I have task of downloading files sized little over 1GB using rsync. It
>> is taking lot of time because of my lousy network speed. Just to test
>> I want to try rsync over UDP if possible, I am sure someone should
>> have done this trial, Can you share ideas about how to do it.

>Of course, 1GB files will take a long time with pretty darn good
>network speed (I for one would regard 5Mbit/sec as pretty good. 1GB
>at 5Mb/sec = 2000 seconds = half an hour). Why do you suspect network
>speed that would be improved by using UDP? Slow is slow, after
>all... FWIW, if there's a UDP-based protocol that rsync can use, I
>don't know of it.

Well, I would say that it depends on the connection. If he has Gb network
between the machines, rsync over ssh should give about 30MB/s. with the
encryption probably being the bottleneck.
If it is over 100Mb link, 10MB/s is achievable.
If it is a long distance transfer then an hour would not be out of place.

Why he thinks UDP will change his "lousy network speed" I have no idea.

Unruh wrote:
> Joe Pfeiffer <(E-Mail Removed)> writes:
>
>> Jagadeesh <(E-Mail Removed)> writes:
>
>>> its a lame question, but let me ask you Gurus,
>>>
>>> I have task of downloading files sized little over 1GB using rsync. It
>>> is taking lot of time because of my lousy network speed. Just to test
>>> I want to try rsync over UDP if possible, I am sure someone should
>>> have done this trial, Can you share ideas about how to do it.
>
>> Of course, 1GB files will take a long time with pretty darn good
>> network speed (I for one would regard 5Mbit/sec as pretty good. 1GB
>> at 5Mb/sec = 2000 seconds = half an hour). Why do you suspect network
>> speed that would be improved by using UDP? Slow is slow, after
>> all... FWIW, if there's a UDP-based protocol that rsync can use, I
>> don't know of it.
>
> Well, I would say that it depends on the connection. If he has Gb network
> between the machines, rsync over ssh should give about 30MB/s. with the
> encryption probably being the bottleneck.

FWIW, I've found that rsync over ssh over a 1 Gbit link has a bottleneck
at the platter, not the cpu. Of course, that's with
electrically-connected drives (SAS, SATA).

It would be interesting to do the same experiment with a good
fiber-attached SAN (I'm thinking HP StorageWorks, e.g.), but I can only
dream about what *that* answer might be, since I haven't got the
hardware to actually do the experiment.

> If it is over 100Mb link, 10MB/s is achievable.
> If it is a long distance transfer then an hour would not be out of place.
>
> Why he thinks UDP will change his "lousy network speed" I have no idea.

Allen Kistler <(E-Mail Removed)> writes:

>Unruh wrote:
>> Joe Pfeiffer <(E-Mail Removed)> writes:
>>
>>> Jagadeesh <(E-Mail Removed)> writes:
>>
>>>> its a lame question, but let me ask you Gurus,
>>>>
>>>> I have task of downloading files sized little over 1GB using rsync. It
>>>> is taking lot of time because of my lousy network speed. Just to test
>>>> I want to try rsync over UDP if possible, I am sure someone should
>>>> have done this trial, Can you share ideas about how to do it.
>>
>>> Of course, 1GB files will take a long time with pretty darn good
>>> network speed (I for one would regard 5Mbit/sec as pretty good. 1GB
>>> at 5Mb/sec = 2000 seconds = half an hour). Why do you suspect network
>>> speed that would be improved by using UDP? Slow is slow, after
>>> all... FWIW, if there's a UDP-based protocol that rsync can use, I
>>> don't know of it.
>>
>> Well, I would say that it depends on the connection. If he has Gb network
>> between the machines, rsync over ssh should give about 30MB/s. with the
>> encryption probably being the bottleneck.

>FWIW, I've found that rsync over ssh over a 1 Gbit link has a bottleneck
>at the platter, not the cpu. Of course, that's with
>electrically-connected drives (SAS, SATA).

It depends on the computer. I ran tests with a SATA drive on a 2GH machine
and Gb ethernet, and it was ssh that was the bottleneck. With a faster
machine
it is possible that disk access would be the bottleneck. ( If I ran rsync
straight-- ie rsync->rsync daemon which does not go through ssh) I got
about 50-80% higher throughput.


>It would be interesting to do the same experiment with a good
>fiber-attached SAN (I'm thinking HP StorageWorks, e.g.), but I can only
>dream about what *that* answer might be, since I haven't got the
>hardware to actually do the experiment.

>> If it is over 100Mb link, 10MB/s is achievable.
>> If it is a long distance transfer then an hour would not be out of place.
>>
>> Why he thinks UDP will change his "lousy network speed" I have no idea.

Maxwell Lol <(E-Mail Removed)> wrote:
> *ALL* networks are unreliable. That's why TCP was developed - to
> provide reliable transfer of data over a network.

Might it be more accurate to say that all IP networks are unreliable?

rick jones
--
The glass is neither half-empty nor half-full. The glass has a leak.
The real question is "Can it be patched?"
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Maxwell Lol <(E-Mail Removed)> writes:

>AZ Nomad <(E-Mail Removed)> writes:

>> On Wed, 22 Oct 2008 07:43:08 -0400, Maxwell Lol <(E-Mail Removed)> wrote:
>>>Jagadeesh <(E-Mail Removed)> writes:
>>
>>>> I have task of downloading files sized little over 1GB using rsync. It
>>>> is taking lot of time because of my lousy network speed. Just to test
>>>> I want to try rsync over UDP if possible, I am sure someone should
>>>> have done this trial, Can you share ideas about how to do it.
>>
>>>So you prefer a faster unreliable backup over a slower reliable one?
>> rsync has error checking. It works perfectly over unreliable networks.

>*ALL* networks are unreliable. That's why TCP was developed - to
>provide reliable transfer of data over a network.


>If rsync works over UDP, what is the command option? I don't see one.
>Rsync runs over rsh/ssh, which uses TCP.

>If there is a remote shell that is reliable and uses UDP as a
>transport, that could be specified. But I don't know of any.


>So please explain how rsync works over UDP......

rsync has two modes-- one is over ssh (or rsh.) in that mode the local
rsync makes an ssh connection to the remote machine and starts up a version
of rsync on that remote machine to connect the two together.
In the other mode, when the local rsync contacts the remote machines on
port 873, the remote (x)inted starts up a version of rsync and intiates the
transfer. That can be either tcp or udp.(Not sure how it decides which to
use). That mode is NOT encrypted ( as rsync over ssh is) and can be about
50% faster depending on your CPU , network,...

It means that the remote machine MUST set up rsync to run as an (x)inetd
daemon. In the first case all that is needed is that the user be able to
log on with ssh, and run rsync. In the second the remote sysadmin must set
up rsync to run as a daemon. This is obviously somewhat more risky since it
potentially opens up the system to remote attacks. That can be mitigated
with hosts.allow and /etc/rsync.conf

Maxwell Lol <(E-Mail Removed)> writes:

>Unruh <unruh-(E-Mail Removed)> writes:

>> rsync has two modes-- one is over ssh (or rsh.) in that mode the local
>> rsync makes an ssh connection to the remote machine and starts up a version
>> of rsync on that remote machine to connect the two together.
>> In the other mode, when the local rsync contacts the remote machines on
>> port 873, the remote (x)inted starts up a version of rsync and intiates the
>> transfer. That can be either tcp or udp.(Not sure how it decides which to
>> use). That mode is NOT encrypted ( as rsync over ssh is) and can be about
>> 50% faster depending on your CPU , network,...

>Thanks. This is new to me. But I didn't see anything about rsync
>using UDP in the man page. And does it use it to initiate a TCP
>connection? Or is EVERYTHING done with UDP?

No idea. All I know is that port 873 is defined as rsync for both TCP and
UDP. I have no idea if and how it uses it.


>And are you use the speedup is due to UDP, or is the speedup duee to
>using RSH versus SSH (no encyprion vs. encryption)

It does not use rsh in the daemon mode. It connects directly to the daemon
running on the other machine.



>>
>> It means that the remote machine MUST set up rsync to run as an (x)inetd
>> daemon. In the first case all that is needed is that the user be able to
>> log on with ssh, and run rsync. In the second the remote sysadmin must set
>> up rsync to run as a daemon. This is obviously somewhat more risky since it
>> potentially opens up the system to remote attacks. That can be mitigated
>> with hosts.allow and /etc/rsync.conf

>I still don't see any config option that says to use UDP versus TCP.

>I'm not on a system where I can monitor a rsync session, but I still doubt
>file transfers can be done using UDP. They could be INITIATED using UDP.

>I suppose you can change/etc/initd.conf so it reads

> rsync stream udp nowait root /usr/bin/rsync rsyncd --daemon

Most modern systems use xinetd, and there is an rsync file in
/etc/xinetd.d/ which you have to enable (disable=no)



>and try it out. Try it both ways and see if there is any difference.
>If UDP is not supported, it ought to generate an error.

>Anyone willling to try?

Maxwell Lol <(E-Mail Removed)> wrote:
> If rsync works over UDP, what is the command option? I don't see one.

You could try something like this, I suppose:

RSYNC_RSH='ssh -f %h nc -u -l 60873; nc -u %h 60873'
rsync -avHP local/files remotehost:/remote/path

Chris

Maxwell Lol <(E-Mail Removed)> wrote:
> *ALL* networks are unreliable [...]

Rick Jones <(E-Mail Removed)> wrote:
> Might it be more accurate to say that all IP networks are unreliable?

Consider two computers linked by {whatever}. I can't see a situation where
you couldn't (theoretically) break the linkage described by {whatever}
by some means.

Chris