UDP packets dropping

Hi,
Its a very simple problem which anyone who has worked on any UDP
application might have faced.
UDP by itself is unreliable. If packets are lost, no acknowledgements
are sent.

I have a UDP application which is receiving packets on a standard port.
It receives packets asynchronously. At any time any packet can arrive.
The problem is that sometimes a number of packets arrive and our UDP
socket buffer ( being of finite size ) is not able
to handle them and simply drops the packet.
The UDP socket buffer cannot be increased beyond a finite size(
dynamically it cannot be increased, it is fixed for an OS) .

So a number of packets simply drop with out any intimation to the
receiver/sender.

I need to have checkpoint ( some kind of intimation ) that the packets
are being dropped.

I have implemented polling with poll parameter as POLLIN or
POLLRDNORM.( I have tried both parameters)
Both implementation works that is gives me a poll status implying
socket buffer is full.

In my test set up I can control the rate at which packets are
generated.
When the packets are sent at a very high rate and my socket buffer is
not able to handle them, polling status tells me that
buffer is full.

when the rate of sending packets is slow, I get an intimation that
buffer space is available.

But in some cases ( when the rate of sending packets is slow ), I get a
poll status that buffer is full ( which is wrong ). Even though
these are corner cases but it made me think.( and caused worry) .

I have following queries..

a) Do we have any mechanism/ way in which we can find that udp socket
buffer is full and packets are being dropped.
b) Can polling be actually be used. My corner cases made me think that.
Is my approach totally wrong.
c) If not Polling then what ?
d) If polling can be used, which other parameters I can use to avoid
these corner case.

e) Is my implementation of polling wrong, Is it possible that I am
misunderstanding the polling status to tell whether
buffer is full.

if you have some idea about this issue please let me know.
If you need some more input from my end let me know.
You can mail me at (E-Mail Removed)

"Vishal" <(E-Mail Removed)> writes:

> I need to have checkpoint ( some kind of intimation ) that the packets
> are being dropped.

Why not just use TCP that provides reliable transport?

How does your application handle if packets arrive out of order? If it
doesn't it's yet another reason to look at TCP.

> I have implemented polling with poll parameter as POLLIN or
> POLLRDNORM.( I have tried both parameters)
> Both implementation works that is gives me a poll status implying
> socket buffer is full.

What if your buffer is almost full? E.g. there is room for a small
packet, but not for a bigger one?

> a) Do we have any mechanism/ way in which we can find that udp socket
> buffer is full and packets are being dropped.

What makes you think that packets are only being dropped when your
buffer is full? What will happen if packets are dropped elsewhere in
the network?

> b) Can polling be actually be used. My corner cases made me think that.
> Is my approach totally wrong.

I believe it is.

> c) If not Polling then what ?

TCP or some other transport protocol that provides reliable transport
if that is really what you need?

--
Jacob

Vishal <(E-Mail Removed)> wrote:
> I have implemented polling with poll parameter as POLLIN or
> POLLRDNORM.( I have tried both parameters)
> Both implementation works that is gives me a poll status implying
> socket buffer is full.

> In my test set up I can control the rate at which packets are
> generated.
> When the packets are sent at a very high rate and my socket buffer is
> not able to handle them, polling status tells me that
> buffer is full.

> when the rate of sending packets is slow, I get an intimation that
> buffer space is available.

> But in some cases ( when the rate of sending packets is slow ), I get a
> poll status that buffer is full ( which is wrong ). Even though
> these are corner cases but it made me think.( and caused worry) .

unless there is a watermark invovled, poll POLLIN/POLLRDNORM only
means there is _some_ data in the buffer. It does not mean the socket
buffer is full.

> I have following queries..

> a) Do we have any mechanism/ way in which we can find that udp socket
> buffer is full and packets are being dropped.

Platform-specific

> b) Can polling be actually be used. My corner cases made me think that.
> Is my approach totally wrong.

It is, polling via poll can tell you nothing about dropped packets

> c) If not Polling then what ?

Platform-specific

> d) If polling can be used, which other parameters I can use to avoid
> these corner case.

none, save perhaps being able to set watermarks on the socket, which
isn't a portable thing.

> e) Is my implementation of polling wrong, Is it possible that I am
> misunderstanding the polling status to tell whether buffer is full.

Yes, you are misunderstanding what poll returns.

I think others have pointed-out that it isn't just a full socket
buffer which can lead to your UDP datagrams being lost. The only way
to know that UDP datagrams have been lost is to have your own
application-level acknowledgement mechanism.

rick jones
--
Process shall set you free from the need for rational thought.
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Rick Jones wrote:
> Vishal <(E-Mail Removed)> wrote:
> > I have implemented polling with poll parameter as POLLIN or
> > POLLRDNORM.( I have tried both parameters)
> > Both implementation works that is gives me a poll status implying
> > socket buffer is full.
>
> > In my test set up I can control the rate at which packets are
> > generated.
> > When the packets are sent at a very high rate and my socket buffer is
> > not able to handle them, polling status tells me that
> > buffer is full.
>
> > when the rate of sending packets is slow, I get an intimation that
> > buffer space is available.
>
> > But in some cases ( when the rate of sending packets is slow ), I get a
> > poll status that buffer is full ( which is wrong ). Even though
> > these are corner cases but it made me think.( and caused worry) .
>
> unless there is a watermark invovled, poll POLLIN/POLLRDNORM only
> means there is _some_ data in the buffer. It does not mean the socket
> buffer is full.
>
> > I have following queries..
>
> > a) Do we have any mechanism/ way in which we can find that udp socket
> > buffer is full and packets are being dropped.
>
> Platform-specific

When you say platform specific how does it matter whether it it Unix or
windows? Or do you mean to different processors( thats unlikely)
>
> > b) Can polling be actually be used. My corner cases made me think that.
> > Is my approach totally wrong.
>
> It is, polling via poll can tell you nothing about dropped packets
>
> > c) If not Polling then what ?
>
> Platform-specific
>
> > d) If polling can be used, which other parameters I can use to avoid
> > these corner case.
>
> none, save perhaps being able to set watermarks on the socket, which
> isn't a portable thing.
>

What are watermarks ? And how to use them ? Can you give insights .
May if we can something for one OS( unix or Windows ) that will be
great help ..


> > e) Is my implementation of polling wrong, Is it possible that I am
> > misunderstanding the polling status to tell whether buffer is full.
>
> Yes, you are misunderstanding what poll returns.
>
> I think others have pointed-out that it isn't just a full socket
> buffer which can lead to your UDP datagrams being lost. The only way
> to know that UDP datagrams have been lost is to have your own
> application-level acknowledgement mechanism.
>
> rick jones
> --
> Process shall set you free from the need for rational thought.
> these opinions are mine, all mine; HP might not want them anyway...
> feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

On Fri, 10 Nov 2006 14:26:25 +0100, Jacob Bunk Nielsen wrote:

> "Vishal" <(E-Mail Removed)> writes:
>
>> I need to have checkpoint ( some kind of intimation ) that the packets
>> are being dropped.
>
> Why not just use TCP that provides reliable transport?

That seems the most obvious solution. The op wants reliable
transport from UDP, which is the wrong approach.

Dan

Hi Jacob,
As i had mentioned, we are receving at a standard port( UDP port) . The
software has to receive on UDP port as it is based on a standard. So
TCP is totally ruled out in my case. as we have to stuck to a specific
standard.

Regards,
Vishal
Jacob Bunk Nielsen wrote:
> "Vishal" <(E-Mail Removed)> writes:
>
> > I need to have checkpoint ( some kind of intimation ) that the packets
> > are being dropped.
>
> Why not just use TCP that provides reliable transport?
>
> How does your application handle if packets arrive out of order? If it
> doesn't it's yet another reason to look at TCP.
>
> > I have implemented polling with poll parameter as POLLIN or
> > POLLRDNORM.( I have tried both parameters)
> > Both implementation works that is gives me a poll status implying
> > socket buffer is full.
>
> What if your buffer is almost full? E.g. there is room for a small
> packet, but not for a bigger one?
>
> > a) Do we have any mechanism/ way in which we can find that udp socket
> > buffer is full and packets are being dropped.
>
> What makes you think that packets are only being dropped when your
> buffer is full? What will happen if packets are dropped elsewhere in
> the network?
>
> > b) Can polling be actually be used. My corner cases made me think that.
> > Is my approach totally wrong.
>
> I believe it is.
>
> > c) If not Polling then what ?
>
> TCP or some other transport protocol that provides reliable transport
> if that is really what you need?
>
> --
> Jacob

"guddu" <(E-Mail Removed)> writes:

> As i had mentioned, we are receving at a standard port( UDP port) . The
> software has to receive on UDP port as it is based on a standard. So
> TCP is totally ruled out in my case. as we have to stuck to a specific
> standard.

Doesn't the standard describe some way for handling dropped packets?

If you tell us which standard you are working with someone here might
know something about implementations of this specific standard.

--
Jacob

>
> Doesn't the standard describe some way for handling dropped packets?
>
> If you tell us which standard you are working with someone here might
> know something about implementations of this specific standard.
>
> --
> Jacob

Hi Jacob,
I have been working on a SNMP application which is based on UDP.

regards,
Vishal

"guddu" <(E-Mail Removed)> writes:

> I have been working on a SNMP application which is based on UDP.

I just browsed RFC 1157 (which describes SNMPv1). It looks like it's a
design choice to keep SNMP as simple as possible and therefor reliable
transport of SNMP-messages has been deselected.

So basically you have no way of knowing if SNMP-packets are lost (in
the network or in a buffer or whatever). It's by design.

I'm not sure that SNMPv3 can't run over TCP. At least this is what my
/etc/services says:

$ grep -i snmp /etc/services
snmp 161/tcp # Simple Net Mgmt Protocol
snmp 161/udp # Simple Net Mgmt Protocol
snmp-trap 162/tcp snmptrap # Traps for SNMP
snmp-trap 162/udp snmptrap # Traps for SNMP
smux 199/tcp # SNMP Unix Multiplexer

.... and also what RFC 1157 hints at in section 3.2.4.

--
Jacob

Jacob Bunk Nielsen <(E-Mail Removed)> writes:
>
> I'm not sure that SNMPv3 can't run over TCP. At least this is what my
> /etc/services says:
>
> $ grep -i snmp /etc/services
> snmp 161/tcp # Simple Net Mgmt Protocol
> snmp 161/udp # Simple Net Mgmt Protocol
> snmp-trap 162/tcp snmptrap # Traps for SNMP
> snmp-trap 162/udp snmptrap # Traps for SNMP
> smux 199/tcp # SNMP Unix Multiplexer
>
> ... and also what RFC 1157 hints at in section 3.2.4.

The contents of /etc/services are no help at all for this -- as it
says at the top of the file,

# Note that it is presently the policy of IANA to assign a single well-known
# port number for both TCP and UDP; hence, officially ports have two entries
# even if the protocol doesn't support UDP operations.
(it doesn't say it, but it's also true of protocols that don't support
TCP operations).

I haven't read the RFP, but if the network devices are sending UDP
packets, no amount of telling them they could send TCP packets as well
is going to help them. And since they aren't likely to support
requests to retransmit, it sounds like the OP is simply going to have
to write an application that is robust in the presence of dropped
packets.
--
Joseph J. Pfeiffer, Jr., Ph.D. Phone -- (505) 646-1605
Department of Computer Science FAX -- (505) 646-1002
New Mexico State University http://www.cs.nmsu.edu/~pfeiffer