NFS from Linux PXE client to AIX 5.3 server?

Hello World,

I'm trying to network-boot Linux clients off an AIX 5.3 server, and I'm
up to the part where it needs to mount its root file system. This
doesn't work - the mount just hangs complaining about the NFS server
not responding. I did a tcpdump on the server (tcpdump host client) and
spotted a few package reassembly errors. I reduced nfso
nfs_max_write_size and nfs_max_read_size to 1024 on the server, and
then I could mount the volume, and read files in it, though I could not
list directories - I/O errors.

I then tried specifying the rsize and wsize like this in
/tftpboot/pxelinux.cnf/default:
append initrd=initrd.img-2.6.15-26-686 root=/dev/nfs \
nfsroot=172.16.1.1:/data/scratch/nfsroot,rsize=32768,wsize=32768

(Line continuation is not actually there in my file, just for clarity
here).

This doesn't work :-/ the machine tries to mount a directory with the
parameters in. Is there any way that a PXE client will use anything but
another Linux NFS server?

Cheers,
Menno Willemse

Menno Willemse schrieb:
> Hello World,
>
> I'm trying to network-boot Linux clients off an AIX 5.3 server, and I'm
> up to the part where it needs to mount its root file system. This
> doesn't work - the mount just hangs complaining about the NFS server
> not responding. I did a tcpdump on the server (tcpdump host client) and
> spotted a few package reassembly errors. I reduced nfso
> nfs_max_write_size and nfs_max_read_size to 1024 on the server, and
> then I could mount the volume, and read files in it, though I could not
> list directories - I/O errors.
>
> I then tried specifying the rsize and wsize like this in
> /tftpboot/pxelinux.cnf/default:
> append initrd=initrd.img-2.6.15-26-686 root=/dev/nfs \
> nfsroot=172.16.1.1:/data/scratch/nfsroot,rsize=32768,wsize=32768
>
> (Line continuation is not actually there in my file, just for clarity
> here).
>
> This doesn't work :-/ the machine tries to mount a directory with the
> parameters in. Is there any way that a PXE client will use anything but
> another Linux NFS server?
>
> Cheers,
> Menno Willemse
>

Unfortunately I have no idea, why the server is not responding. Have you
started the portmapper and nfslock on the client machine? Here's how I
made it a few years ago:

May solution needed absolutely no harddisk and not boot media (except
the network adapter, of course) on the client side:

- The network card PXE environment got an IP address from the dhcp
server, and the information what to boot next.
- I didn't boot a kernel image, but a grub bootloader image.
- After the startup of grub, grub got it's configurationfile via tftp
from the fileserver (can also be configured on the dhcp server)
- Grub then appeared with a nice menu, letting me choose, whether to
start the local windows installation or the linux network installation.
- Grub downloads the (unmodified) linux kernel and it's initrd via tftp
and boots it.
- The initrd has to be build manually: It has to contain the necessary
binaries and libraries, in order to mount all directories via nfs and a
small script called "linuxrc" in the intrd's root directory. This script
is automatically executed after the kernel boot process and it can
handle all the mounting stuff and make a chroot (it's actually a
pivot_root) to the mountet directory.
- The system can now boot as if it was installed locally.

Neither the kernel, nor anything in the distribution had to be
adapted/compiled. Only grub had to be compiled manually, as it needs
network/ttftp support and the necessary network adpater support compiled in.

greets Boris

Boris Glawe wrote:
> Menno Willemse schrieb:
> > Hello World,
> >
> > I'm trying to network-boot Linux clients off an AIX 5.3 server, and I'm
> > up to the part where it needs to mount its root file system. This
> > doesn't work - the mount just hangs complaining about the NFS server
> > not responding. I did a tcpdump on the server (tcpdump host client) and
> > spotted a few package reassembly errors. I reduced nfso
> > nfs_max_write_size and nfs_max_read_size to 1024 on the server, and
> > then I could mount the volume, and read files in it, though I could not
> > list directories - I/O errors.
> >
> > I then tried specifying the rsize and wsize like this in
> > /tftpboot/pxelinux.cnf/default:
> > append initrd=initrd.img-2.6.15-26-686 root=/dev/nfs \
> > nfsroot=172.16.1.1:/data/scratch/nfsroot,rsize=32768,wsize=32768
> >
> > (Line continuation is not actually there in my file, just for clarity
> > here).
> >
> > This doesn't work :-/ the machine tries to mount a directory with the
> > parameters in. Is there any way that a PXE client will use anything but
> > another Linux NFS server?
> >
> > Cheers,
> > Menno Willemse
> >
>
> Unfortunately I have no idea, why the server is not responding. Have you
> started the portmapper and nfslock on the client machine? Here's how I
> made it a few years ago:

Nope, haven't. What I'm using is the initrd that you can build on
Ubuntu. I went into /etc/mkinitramfs/initramfs.conf, changed BOOT to
nfs, changed MODULES to netboot, and ran mkinitramfs. And that is what
I'm feeding to pxeboot/the kernel. When I fall through to the
"busybox", I cannot mount it by hand either. If I put the NFS-root on a
Linux NFS server instead, it does work, but I want to build one big
boot/install server for both Linux and AIX.

> May solution needed absolutely no harddisk and not boot media (except
> the network adapter, of course) on the client side:
>
> - The network card PXE environment got an IP address from the dhcp
> server, and the information what to boot next.

Yep. Got that as well. DHCP server is the same AIX box, actually.

> - I didn't boot a kernel image, but a grub bootloader image.

Ah. I do boot a Linux kernel, but I don't want the menu. Will put up
with a menu if I have to, though.

> - After the startup of grub, grub got it's configurationfile via tftp
> from the fileserver (can also be configured on the dhcp server)
> - Grub then appeared with a nice menu, letting me choose, whether to
> start the local windows installation or the linux network installation.

Now what kind of a choice is that? ;-)

> - Grub downloads the (unmodified) linux kernel and it's initrd via tftp
> and boots it.
> - The initrd has to be build manually: It has to contain the necessary
> binaries and libraries, in order to mount all directories via nfs and a
> small script called "linuxrc" in the intrd's root directory. This script
> is automatically executed after the kernel boot process and it can
> handle all the mounting stuff and make a chroot (it's actually a
> pivot_root) to the mountet directory.

Hmm. If I look at the initrd directory, I see a /init, which is a link
to /usr/share/initramfs-tools/init. That calls a "mountroot" function,
which is defined both in "/scripts/local" and "/scripts/nfs". Depending
on where you're booting from, one or the other gets sourced. A bit more
digging shows that I'll need to add any AIX NFS voodoo to a script in
/etc/mkinitramfs/scripts/nfs-promount.

> - The system can now boot as if it was installed locally.
>
> Neither the kernel, nor anything in the distribution had to be
> adapted/compiled. Only grub had to be compiled manually, as it needs
> network/ttftp support and the necessary network adpater support compiled in.
>

Well, as soon as I find out how to mount my NFSRoot, at least now I
know where to put the required commands. I can't find any portmapper in
the initrd though. Maybe that's built into the kernel.

> greets Boris

Thanks, Boris. Got a few new avenues to explore.

Cheers,
Menno

Menno Willemse wrote:

Normaly NFS problem on AIX can be caused by:
a) Naming resolution - You need the by dhcp provided hostname also in
your /etc/hosts
b) Client tries to use NFS v4 . So you might set the desired version in
your /etc/exportsf file like

/export/suse/sles10/ -vers=3

and remove security setting on NFS exports to keep things simple.

hth
Hajo

BTW:
I our environment we have a dhcp/nfs/http/tftp Suse installation server
on AIX 5.3 TL5 but i never testet a root NFS mount.

Hi Hajo,

Hajo Ehlers wrote:
> Menno Willemse wrote:
>
> Normaly NFS problem on AIX can be caused by:
> a) Naming resolution - You need the by dhcp provided hostname also in
> your /etc/hosts

Got that.

> b) Client tries to use NFS v4 . So you might set the desired version in
> your /etc/exportsf file like
>
> /export/suse/sles10/ -vers=3

Will try that. This is clearly some sort of protocol problem with the
kernel-only version of nfs-mount, as I can mount the directory OK once
the system is up.

> and remove security setting on NFS exports to keep things simple.
>
> hth
> Hajo
>
> BTW:
> I our environment we have a dhcp/nfs/http/tftp Suse installation server
> on AIX 5.3 TL5 but i never testet a root NFS mount.

The next problem I can see myself solving: Fix the major/minor numbers
in /dev >_<

Cheers,
Menno

Hello World,

Further enlightenment has happened.

To get a shell on the diskless client, I make a small change to the
nfsroot= line in my pxeroot.cfg file, which results in a mount error
and drops me into the busybox. I have modified the initrd image to
start the portmapper as well, which lets me mount my NFSroot on /root
by hand.

Now when I do a manual chroot into this directory, the "NFS Server not
Responding" message pops up (and I get no prompt).

So my current theory is that this is also what happens when I try do do
everything automatically.

Something in my AIX-hosted NFS root is amiss. I have recreated the
NFSROOT by running debootstrap from a Linux box onto the nfsroot file
system. This preserves the minor/major device numbers that Linux
expects, which is a great advantage. To archive this directory, I add
the -d flag to tar so devices get saved correctly[1]. My Linux-hosted
NFS server uses a copy taken from this tar file and *that* works,
though I get warnings about lockd, but that I can fix later.

Keep on hacking...

Cheers,
Menno


[1] Really, IBM - is there ever a reason NOT to save special files
correctly?!

Hello World,

Even though responding to oneself has a high navel contemplation
quotient... Thought I'd keep this up. In my ongoing attempts to host
diskless clients on AIX, I have downloaded and installed the LTSP
resources for diskless clients. Then, I copied the NFS-root, kernel and
initrd over to the AIX server and tried to have it mounted at boot.

As it turns out, this failed with the same "NFS Server Not Responding"
messages i've come to know and loathe. Now Linux is well capable of
mounting and using NFS resources on an AIX host (set the
nfs_use_reserved_ports nfso parameter to 1 on the server or it won't
work), but this is on an installed system that has the nfs-common
packages installed. I think that the nfs-common packages implement a
more capable NFS client than the one we get in the PXE/Kernel
environment. I think this is also what caused my FAI installations to
fail.

I don't want to keep hosting these diskless clients on Linux, because
what I'm trying to build is an AIX server that can host the lot:
Install AIX, install Linux *and* host diskless Linux clients. This NFS
problem is stupid and I want to get on with the more interesting jobs
of scripting Linux installs and providing a working environment.

So I put it to the Oracle: Is it possible to massage an AIX NFS server
so that even the Neanderthal kernel NFS client can mount directories
off it, or conversely, to upgrade the NFS client in the Linux kernel so
that it is able to mount directories off AIX?

Cheers,
Menno Willemse

Hello World,

Got it! The key to the solution was here:

http://wiki.ltsp.org/twiki/bin/view/...not_responding
https://launchpad.net/ubuntu/breezy/...ols/+bug/19196

To summarise: I had created an NFSroot on an AIX 5.3 server, exported
as follows:

==[/etc/exports extract]==========================
/data/linux/dapper/nfsroot -public,rw,root=term001
==========
For the Linux hackers here: this is equivalent to turning root squash
off for term001. The problem was that the client (a Wyse Winterm) was
using UDP to access the NFS server. UDP don't do no receipt
confirmation, so whenever the terminal gave a request, the response got
lost somewhere. (Possibly because it came too quickly for the WinTerm
to catch it). In addition, when I tried to mount the root filesystem
off the AIX host by hand, I got messages about the portmapper not
running, which was correct5, because this was basically the initial RAM
disk, which does not include a portmapper by default. Adding the
portmapper did not help either.

What did help, was to add a parameter to
/tftpboot/pxelinux.cfg/[HEX_IP_ADDRESS], like so:

==[/tftpboot/pxelinux.cfg/ubuntu_diskless_clients]==============
TIMEOUT 50
default auto
label auto
kernel ubuntu-diskless/vmlinuz-2.6.15-26-686
append initrd=ubuntu-diskless/initrd.img-2.6.15-26-686 \
root=/dev/nfs \
nfsroot=172.16.1.5:/data/linux/dapper/nfsroot \
NFSOPTS="-o nolock,rw,proto=tcp"
==========

Note that the \
line continuations don't actually appear in my file, and are here only
to avoid wrapping. The pxelinux file reader doesn't do line
continuation. Don't knock it, it can only be 512KB in size so
something's got to give.

Note also that this works only on Ubuntu. Look up the correct way of
specifying this parameter for your distro in the document at the URL
above.

The NFSOPTS="-o nolock,rw,proto=tcp" parameter is the important one.
(Though rw should probably be ro in the final version of this
environment). It makes the klibc nfsmount program use the TCP protocol
rather than UDP. TCP *does* do flow control, so no packets are lost. It
also removes the need for a portmapper. The nolock parameter removes
the need for a lockd on the client, at the cost of... having no locking
ability. Remember this when you start booting several clients off the
same NFSroot.

So far so good. I now have a character based Linux-running thin client.
The next step is to GUIfy this environment using Xorg/XFree86.
Hopefully this seeding of the archives will help someone in the future.

Cheers,
Menno Willemse.