Problem seeing correct originating IP in Ubuntu linux

Hi all,

I have my own linux Ubuntu 6.10 server running a variety of services.
Since I last upgraded my kernel to version 2.6.17-12-generic, I have the
strangest problem. A little background on my very simple network
configuration:

<LAN>
|
[10.0.0.1 - WRT54G running DD-WRT with DHCP and dnsmasq] - <internet>
|
[10.0.0.3 - server]
|
[10.0.0.5 - xbox]
|
[10.0.0.25 - laptop]

Now, I often log into my server with SSH from the laptop, and normally I
would see the login coming from 10.0.0.25 like this (I think this is
what it looked like, at least I am sure I could see the source IP in
some form, but I cannot reproduce it anymore):

user@server:~$ netstat -tn
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address
State
ESTABLISHED
tcp6 0 0 ::ffff:10.0.0.3:22 ::ffff:10.0.0.25:2276
user@server:~$

...but after this kernel update (I think this is what changed it), I have:

user@server:~$ netstat -tn
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address
State
ESTABLISHED
tcp6 0 0 ::ffff:10.0.0.3:22 ::ffff:10.0.0.1:2276
user@server:~$

So it looks like my SSH connection is coming from the router, while it
is really coming from my laptop. This strange behavior is also present
in all log-files showing information about originating IP (syslog for
instance).

One funny thing is that if the xbox connects via samba to the server,
everything looks ok:

user@server:~$ netstat -tn
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address
State
tcp 0 0 10.0.0.3:445 10.0.0.5:1040
ESTABLISHED

So to me this looks like a problem displaying ipv6 ip's? Or is it me not
having understood ipv6 at all?

Ifconfig of eth0 on the server gives:

eth0 Link encap:Ethernet HWaddr 00:40:63:E6:5B:8B
inet addr:10.0.0.3 Bcast:10.0.0.255 Mask:255.255.255.0

...so netmask is correct, and it is the same on the windows laptop. So
traffic really shouldn't be routed. And if I ping the laptop:

root@server:~# ping 10.0.0.25
PING 10.0.0.25 (10.0.0.25) 56(84) bytes of data.
64 bytes from 10.0.0.25: icmp_seq=1 ttl=128 time=2.79 ms

Last but not least - if I run a "arp -a" everything seems correct:

root@server:# arp -a
wrt54g.mydomain.dk (10.0.0.1) at 00:18:39:C0:1A:30 [ether] on eth0
laptop.mydomain.dk (10.0.0.25) at 00:0E:35E:8F:49 [ether] on eth0
xbox.mydomain.dk (10.0.0.5) at 00:12:5A:20:C5:C5 [ether] on eth0
root@server:#

I am not sure whether this is something to be worried about (I guess it
isn't), but I would like the correct information in my logfiles and in
my netstat. Any clues?

Best regards,
Henrik

Hello,

Henrik Uhrenfeldt a écrit :
[...]
> ..but after this kernel update (I think this is what changed it), I have:
>
> user@server:~$ netstat -tn
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State
> ESTABLISHED
> tcp6 0 0 ::ffff:10.0.0.3:22 ::ffff:10.0.0.1:2276
> user@server:~$
>
> So it looks like my SSH connection is coming from the router, while it
> is really coming from my laptop. This strange behavior is also present
> in all log-files showing information about originating IP (syslog for
> instance).

How do you connect to the server ? By name, by address ?
It looks like your laptop is connecting to the server through the router
and some source NAT is involved. Could it be that the ssh client uses
the router's external address and the router has a port forwarding NAT
rule redirecting port 22 to the server address ? For this to work, the
router has to replace the original client source address with its own.

[...]
> So to me this looks like a problem displaying ipv6 ip's?

No. The tcp6 and ::ffff: IPv6 prefix are caused by sshd receiving the
IPv4 connections on an IPv6 socket, so IPv4 addresses are seen as
IPv4-mapped IPv6 addresses. I didn't like that so I set up my box so
that sshd listens on separate IPv4 and IPv6 sockets.

Hi,

I inlined some answers below..

Pascal Hambourg wrote:
> Hello,
>
> Henrik Uhrenfeldt a écrit :
> [...]
>> ..but after this kernel update (I think this is what changed it), I have:
>>
>> user@server:~$ netstat -tn
>> Active Internet connections (w/o servers)
>> Proto Recv-Q Send-Q Local Address Foreign Address State
>> ESTABLISHED
>> tcp6 0 0 ::ffff:10.0.0.3:22 ::ffff:10.0.0.1:2276
>> user@server:~$
>>
>> So it looks like my SSH connection is coming from the router, while it
>> is really coming from my laptop. This strange behavior is also present
>> in all log-files showing information about originating IP (syslog for
>> instance).
>
> How do you connect to the server ? By name, by address ?
> It looks like your laptop is connecting to the server through the router
> and some source NAT is involved. Could it be that the ssh client uses
> the router's external address and the router has a port forwarding NAT
> rule redirecting port 22 to the server address ? For this to work, the
> router has to replace the original client source address with its own.
>

I connect directly to IP 10.0.0.3 from the laptop, and since netmask is
255.255.255.0 on the laptop, I really think it should do a direct
connection. But yes - it does look like it connects through the router;
this is what puzzles me.

> [...]
>> So to me this looks like a problem displaying ipv6 ip's?
>
> No. The tcp6 and ::ffff: IPv6 prefix are caused by sshd receiving the
> IPv4 connections on an IPv6 socket, so IPv4 addresses are seen as
> IPv4-mapped IPv6 addresses. I didn't like that so I set up my box so
> that sshd listens on separate IPv4 and IPv6 sockets.

Ah, so this is why some connections look like IPv6 and others don't. I
was beginning to think I was running an IPv6 network :-)
Where did you change this - in sshd config?

- Henrik

Henrik Uhrenfeldt a écrit :
>
> I connect directly to IP 10.0.0.3 from the laptop, and since netmask is
> 255.255.255.0 on the laptop, I really think it should do a direct
> connection. But yes - it does look like it connects through the router;
> this is what puzzles me.

tcpdump or (t)ethereal/wireshark/tshark on the client and server may
help tracking the actual packets path.

>> No. The tcp6 and ::ffff: IPv6 prefix are caused by sshd receiving the
>> IPv4 connections on an IPv6 socket, so IPv4 addresses are seen as
>> IPv4-mapped IPv6 addresses. I didn't like that so I set up my box so
>> that sshd listens on separate IPv4 and IPv6 sockets.
>
> Ah, so this is why some connections look like IPv6 and others don't. I
> was beginning to think I was running an IPv6 network :-)
> Where did you change this - in sshd config?

If you have an IPv4-only network, add or uncomment the following
directive in /etc/ssh/sshd_config, so sshd opens an IPv4 socket :

ListenAddress 0.0.0.0

And delete or comment any ListenAddress directive with address "::" (any
IPv6 address) so sshd does not open an IPv6 socket. Then restart sshd.

If you have and IPv4+IPv6 network, you must set the kernel sysctl
/proc/sys/net/ipv6/bindv6only to 1 to prevent the use of IPv4
communications with IPv6 sockets. Add or uncomment the following two
directives in sshd_config, so sshd opens IPv4 and IPv6 sockets :

ListenAddress ::
ListenAddress 0.0.0.0

Then restard sshd. Note that setting bindv6only to 1 may have issues
with other dual stack services, because IPv6 sockets won't accept IPv4
connections any more.

Hi again,

The plot thickens.

I just sniffed my server eth0 with wireshark. I see broadcast netbios
name traffic from my laptop MAC address using IP 10.0.0.25, and then I
see a direct connection to my server from the same MAC, but with ip
10.0.0.1 as source... Does my laptop spoof it's IP? And if so - why?

Come to think of it, my router actually bridges the wireless part (on
which the laptop is) together with the physical LAN ethernet. But I
would think that MAC's would still be traceable in the packet header. I
could understand if it had been NAT translated, but it shouldn't when
bridged, right?

- Henrik

Henrik Uhrenfeldt wrote:
> Hi,
>
> I inlined some answers below..
>
> Pascal Hambourg wrote:
>> Hello,
>>
>> Henrik Uhrenfeldt a écrit :
>> [...]
>>> ..but after this kernel update (I think this is what changed it), I
>>> have:
>>>
>>> user@server:~$ netstat -tn
>>> Active Internet connections (w/o servers)
>>> Proto Recv-Q Send-Q Local Address Foreign Address State
>>> ESTABLISHED
>>> tcp6 0 0 ::ffff:10.0.0.3:22 ::ffff:10.0.0.1:2276
>>> user@server:~$
>>>
>>> So it looks like my SSH connection is coming from the router, while
>>> it is really coming from my laptop. This strange behavior is also
>>> present in all log-files showing information about originating IP
>>> (syslog for instance).
>>
>> How do you connect to the server ? By name, by address ?
>> It looks like your laptop is connecting to the server through the
>> router and some source NAT is involved. Could it be that the ssh
>> client uses the router's external address and the router has a port
>> forwarding NAT rule redirecting port 22 to the server address ? For
>> this to work, the router has to replace the original client source
>> address with its own.
>>
>
> I connect directly to IP 10.0.0.3 from the laptop, and since netmask is
> 255.255.255.0 on the laptop, I really think it should do a direct
> connection. But yes - it does look like it connects through the router;
> this is what puzzles me.
>
>> [...]
>>> So to me this looks like a problem displaying ipv6 ip's?
>>
>> No. The tcp6 and ::ffff: IPv6 prefix are caused by sshd receiving the
>> IPv4 connections on an IPv6 socket, so IPv4 addresses are seen as
>> IPv4-mapped IPv6 addresses. I didn't like that so I set up my box so
>> that sshd listens on separate IPv4 and IPv6 sockets.
>
> Ah, so this is why some connections look like IPv6 and others don't. I
> was beginning to think I was running an IPv6 network :-)
> Where did you change this - in sshd config?
>
> - Henrik

Henrik Uhrenfeldt a écrit :
>
> The plot thickens.
>
> I just sniffed my server eth0 with wireshark. I see broadcast netbios
> name traffic from my laptop MAC address using IP 10.0.0.25, and then I
> see a direct connection to my server from the same MAC, but with ip
> 10.0.0.1 as source... Does my laptop spoof it's IP? And if so - why?

Rather unlikely. If so, it wouldn't receive the replies and the SSH
session could not be established.

> Come to think of it, my router actually bridges the wireless part (on
> which the laptop is) together with the physical LAN ethernet. But I
> would think that MAC's would still be traceable in the packet header. I
> could understand if it had been NAT translated, but it shouldn't when
> bridged, right?

Bridged IPv4 packets traverse iptables chains - including NAT - in 2.6
kernels compiled with bridge-nf support (CONFIG_BRIDGE_NETFILTER=y) if
/proc/sys/net/bridge/bridge-nf-call-iptables=1 (this is the default).
However the 'physdev' iptables match has an option
'--physdev-is-bridged' which matches only bridged packets, not routed
ones. If /proc/sys/net/bridge/bridge-nf-call-iptables is present on your
router and set to 1, try to set it to 0.
However 2.4 kernels do not support bridge-nf.

Pascal Hambourg a écrit :
>
> However 2.4 kernels do not support bridge-nf.

Unless patched with the ebtables-brnf patch from
<http://ebtables.sourceforge.net/>.

Pascal Hambourg wrote:
> Henrik Uhrenfeldt a écrit :
>>
>> I connect directly to IP 10.0.0.3 from the laptop, and since netmask
>> is 255.255.255.0 on the laptop, I really think it should do a direct
>> connection. But yes - it does look like it connects through the
>> router; this is what puzzles me.
>
> tcpdump or (t)ethereal/wireshark/tshark on the client and server may
> help tracking the actual packets path.
>
>>> No. The tcp6 and ::ffff: IPv6 prefix are caused by sshd receiving the
>>> IPv4 connections on an IPv6 socket, so IPv4 addresses are seen as
>>> IPv4-mapped IPv6 addresses. I didn't like that so I set up my box so
>>> that sshd listens on separate IPv4 and IPv6 sockets.
>>
>> Ah, so this is why some connections look like IPv6 and others don't. I
>> was beginning to think I was running an IPv6 network :-)
>> Where did you change this - in sshd config?
>
> If you have an IPv4-only network, add or uncomment the following
> directive in /etc/ssh/sshd_config, so sshd opens an IPv4 socket :
>
> ListenAddress 0.0.0.0
>
> And delete or comment any ListenAddress directive with address "::" (any
> IPv6 address) so sshd does not open an IPv6 socket. Then restart sshd.
>
> If you have and IPv4+IPv6 network, you must set the kernel sysctl
> /proc/sys/net/ipv6/bindv6only to 1 to prevent the use of IPv4
> communications with IPv6 sockets. Add or uncomment the following two
> directives in sshd_config, so sshd opens IPv4 and IPv6 sockets :
>
> ListenAddress ::
> ListenAddress 0.0.0.0
>
> Then restard sshd. Note that setting bindv6only to 1 may have issues
> with other dual stack services, because IPv6 sockets won't accept IPv4
> connections any more.

Pascal,

Thanks for your help. I started suspected that something fishy was up
with the ethernet bridge in the router, and I ended up installing a
newer firmware on it, and with a fresh router firmware and default
settings, things started to work again.

I'll try your SSH tip - thanks for the info.

- Henrik

Pascal Hambourg wrote:
> Henrik Uhrenfeldt a écrit :
>>
>> I connect directly to IP 10.0.0.3 from the laptop, and since netmask
>> is 255.255.255.0 on the laptop, I really think it should do a direct
>> connection. But yes - it does look like it connects through the
>> router; this is what puzzles me.
>
> tcpdump or (t)ethereal/wireshark/tshark on the client and server may
> help tracking the actual packets path.
>
>>> No. The tcp6 and ::ffff: IPv6 prefix are caused by sshd receiving the
>>> IPv4 connections on an IPv6 socket, so IPv4 addresses are seen as
>>> IPv4-mapped IPv6 addresses. I didn't like that so I set up my box so
>>> that sshd listens on separate IPv4 and IPv6 sockets.
>>
>> Ah, so this is why some connections look like IPv6 and others don't. I
>> was beginning to think I was running an IPv6 network :-)
>> Where did you change this - in sshd config?
>
> If you have an IPv4-only network, add or uncomment the following
> directive in /etc/ssh/sshd_config, so sshd opens an IPv4 socket :
>
> ListenAddress 0.0.0.0
>
> And delete or comment any ListenAddress directive with address "::" (any
> IPv6 address) so sshd does not open an IPv6 socket. Then restart sshd.
>
> If you have and IPv4+IPv6 network, you must set the kernel sysctl
> /proc/sys/net/ipv6/bindv6only to 1 to prevent the use of IPv4
> communications with IPv6 sockets. Add or uncomment the following two
> directives in sshd_config, so sshd opens IPv4 and IPv6 sockets :
>
> ListenAddress ::
> ListenAddress 0.0.0.0
>
> Then restard sshd. Note that setting bindv6only to 1 may have issues
> with other dual stack services, because IPv6 sockets won't accept IPv4
> connections any more.

Pascal,

Thanks for your help. I started suspected that something fishy was up
with the ethernet bridge in the router, and I ended up installing a
newer firmware on it, and with a fresh router firmware and default
settings, things started to work again.

I'll try your SSH tip - thanks for the info.

- Henrik