Kernel panic due to NF_IP_LOCAL_OUT handler calling itself again

Discussion in 'Linux Networking' started by Morfean, Mar 1, 2005.

  1. Morfean

    Morfean Guest

    I am writing an implementation of a source routing protocol as a
    loadable module. I am using netfilter, I am not using the IP SSR
    option, I am using kernel 2.6.5, without smp and preemption support. My
    design is based on the DSR protocol. I have a header after the IP
    header, describing the source route and the route error. If 's' is the
    src, 'd' is the dst, and x1,x2 ... are the hops, the source route is :
    x1-x2...d. However unlike SSR I don't change the dst field of the ip
    header, which is set to d. Also every src routed packet carries with it
    an ack request for the next hop, to which the next hop is supposed to
    reply to.
    The jist of my code is as follows:

    For all source routed packets:
    ackReply to previous hop
    nxtHop = getNextHop(packet);
    drop packet;
    return NF_ACCEPT.

    In local_out_handler:
    create a packet with the source route. route it to the first hop,
    using ip_route_output_key(flowi,skb). I then do an awaitAck(firstHop).

    I intend to use this protocol in wireless networks. And hence I have
    implemented an ack based design, where every node is responsible for
    ensuring that the packet makes it to the next hop. For this the
    function awaitAck does the following:

    add an ackRequest header.
    add a timer(a pointer to it) in the skb->cb.
    put this skb to a queue.

    When this timer fires:
    if rtxCount for the skb less than a MAX_RXMT, retransmit, else send a
    route error on the reverse route to the source.

    This scheme is working fine when there are no errors. I am testing it
    across an ethernet lan by pinging. However when there are link
    failures, weird things happen.
    The topology I am using is A - - - B, where B is some fictional node
    not connected to A, but in the same subnet (the routing table so
    configured to put such packets to eth0). In the local out handler, I do
    a ip_route_output to B, which succeeds. I call the output function
    okfn, given to me by netfilter directly and return NF_STOLEN. The next
    packet to come to me in local_out is an icmp dest unreach. It is
    destined to me, so I accept it. The next packet is again an icmp dest
    unreach, and after that somehow my local_out_handler is called again,
    while the first call of it hasn't finished. (My kernel is not smp and
    not preemptible). At times this happens over and over. My kernel then
    panics, either due to a stack overflow, or some bad eip value, or
    something else (with eip value not decoded, and nowhere in the
    /proc/kallsyms). I detect this double calling of my local_out_handler
    by using an atomic_t variable. The same effect is seen if I return an
    NF_DROP or NF_ACCEPT on the skb in the reentrant call. I have also used
    spinlocks but the kernel always crashes.

    After some poking around the kernel code, I found that if an arp entry
    to a node is not present, an entry is created in the arptable with
    status set to pending. All packets waiting for this arp request to be
    resolved are queued up. When the arp request ultimately fails(times
    out), the packets in this queue are freed and an icmp destination
    unreachable error is sent back to the source of these packets.

    But I are still unable to figure out as to why EIP is getting
    corrupted(or the reason for stack overflow).

    I am quite helpless, I have posted on several mailing lists(linux-net,
    kernelnewbies, linux-kernel), but haven't received any replies, Also I
    couldn't find any decent material on the web related to my problem.
    I would be obliged if someone could please help me with this. What am I
    doing wrong, where can I find information on this?
    My code is completely based on Alex Song's DSR implementation for linux
    2.4, available online at

    Vinay Reddy
    Morfean, Mar 1, 2005
    1. Advertisements

  2. Morfean

    Morfean Guest

    There is an update.

    The protocol I am writing is a source routing based protocol called
    Meghadoot similar to the wireless networks DSR, as written by Alex Song
    ( I am using netfilter, I am using linux 2.6.5
    (supplied along with FC2), without the SMP and PREEMPTIBLE options.
    I have a seperate header called MeghHeader, which carries the source
    route and it comes b/w the iph and the tcph. I am using an isolated
    machine called A from which I send out ping packets. Here is what is
    In my local_out_handler, I cruft the source routing header, route the
    packet using ip_route_output (which causes an arp request to be send).
    And then call the output routine passed to me by netfilter (okfn).
    As the arp request obviously will fail, and hence
    skb->dst->neighbour->output points to neigh_resolve_output, which sends
    the arp request, and queues the skb in the neigh->arp_queue, the
    neigh->state is NUD_INCOMPLETE.
    When neigh_timer_handle fires, it walks thru the arp_queue and sends an
    icmp dest unreach for each of the queued skb's thru the
    ipv4_link_failure function.
    Please note that my local_out_handler is serialized by having a
    spinlock at its entry.
    Hence my debugging output looks something like this:
    Entering local out with icmp packet type 8, code 0 : The normal ping
    packet, in pings context.
    ... Some more of the above
    Entering local out with icmp packet type 3, code 1 : The icmp dest
    unreach, given off in the TIMER_SOFTIRQ context, when
    neigh_timer_handler is fired.
    ...And then some more of the two...

    Now this is something, that has totally freaked me out ....

    I sometimes get an icmp skb in my local out handler with some invalid
    type say like 234 etc, As a hack in my local_out_handler, I return
    NF_DROP ('cos I noticed that icmp_send, assumes that the type is
    correct before indexing into the icmp table). As soon as my
    local_out_handler exits with this screwed up skb,(Again, my
    local_out_handler is not reentrant, it is locked by a spinlock) I get
    either a bad eip or a null eip somewhere in the neigh_timer_handler.
    The stack dump looks like this. (The full debugging output, including
    the oops message is available online at
    I am trying to ping the machine, whose route is hardcoded into
    <------------ My debugging output begins ---------------->
    Meghadoot: ****************** Entering local out *******************
    function addr 224c483, dst output 224a842<2>
    Meghadoot: okfn points to :Address: 224c483, name: dst_output, offset:
    0,size: 1c, module: Kernel<2>,
    Meghadoot: skb->dst->output points to : Address: 224a842, name:
    ip_output, offset: 0,size: 5c, module: Kernel<2>
    Meghadoot: local out handler, dumping call trace
    [<22926360>] local_out_handler+0x4e/0x296 [megh_linux]
    [<0224c483>] dst_output+0x0/0x1c
    [<0224a842>] ip_output+0x0/0x5c
    [<0223c787>] nf_iterate+0x40/0x89
    [<0224c483>] dst_output+0x0/0x1c
    [<0223ca91>] nf_hook_slow+0x90/0x101
    [<0224c483>] dst_output+0x0/0x1c
    [<0224c0dc>] ip_push_pending_frames+0x2cf/0x37c
    [<0224c483>] dst_output+0x0/0x1c
    [<0226686f>] icmp_send+0x35e/0x397
    [<02107352>] do_IRQ+0x134/0x169
    [<02239dc9>] neigh_timer_handler+0x0/0x112
    [<0211de03>] run_timer_softirq+0x10b/0x12a
    [<0211af6d>] __do_softirq+0x35/0x73
    [<021078f5>] do_softirq+0x46/0x4d
    [<0210737b>] do_IRQ+0x15d/0x169
    [<0210403b>] default_idle+0x23/0x26
    [<0210408c>] cpu_idle+0x1f/0x34
    [<02318612>] start_kernel+0x174/0x176

    Meghadoot: In context: in_softirq:1024,in_irq:0,in_interrupt:1024<2>
    Meghadoot: local out called in context of swapper<2>
    Meghadoot: finished dumping stack ....<2>
    Meghadoot: Entered local Out handler,printing skb info<2>
    Meghadoot: Local out handler , icmp type is 0, code is 0<2>s:
    d: and protocol is 1<2> (Valid icmph->type, I return NF_DROP
    if icmph->type > NR_ICMP_TYPES )
    Meghadoot: Got Locally destined packet (ie iph->daddr = MYIPADDR )<2>,
    returned NF_ACCEPT.
    Meghadoot: ****************** Exiting local out
    *******************<1>Unable to handle kernel NULL pointer dereference
    at virtual address 00000000
    printing eip:
    *pde = 00000000
    Oops: 0000 [#1]
    CPU: 0
    EIP: 0060:[<00000000>] Not tainted
    EFLAGS: 00010206 (2.6.5-1.358)
    EIP is at 0x0
    eax: 00000000 ebx: 00000000 ecx: 0000b404 edx: 00000007
    esi: 00000000 edi: 00000000 ebp: 00000000 esp: 02344fb0
    ds: 007b es: 007b ss: 0068
    Process swapper (pid: 0, threadinfo=02344000 task=022c5aa0)
    Stack: 00020000 00000000 00594400 1c8e4900 00000056 02239dc9 1c8e495c
    02344fd0 02344fd0 0234007b 00000001 02369428 0000000a 02316000
    02317f94 00000046 00000000 021078f5
    Call Trace:
    [<02239dc9>] neigh_timer_handler+0x0/0x112
    [<0211de03>] run_timer_softirq+0x10b/0x12a
    [<0211af6d>] __do_softirq+0x35/0x73
    [<021078f5>] do_softirq+0x46/0x4d
    [<0210737b>] do_IRQ+0x15d/0x169
    [<0210403b>] default_idle+0x23/0x26
    [<0210408c>] cpu_idle+0x1f/0x34
    [<02318612>] start_kernel+0x174/0x176

    Code: Bad EIP value.
    <0>Kernel panic: Fatal exception in interrupt
    In interrupt handler - not syncing

    <---------------------- End of debugging dump --------------------->

    I will be really greateful, if any one of you could help me out with
    this. Please.

    Expecting your reply eagerly.
    Morfean, Mar 1, 2005
    1. Advertisements

  3. Morfean

    jimxu Guest

    When you receive the icmp dest unreach message, do you drop it or inform the
    and can you reset the atomic_t variable?
    try to stop local_out_handler when receive icmp dest unreach message by set
    a flag.
    jimxu, Mar 3, 2005
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.