multifunction production servers

G'day folks
I must stress from the outset, my networking and linux knowledge is all
from books, and my real-world experience amounts to 3months in total in
my new job. I should also note that nothing will change, this is all out
of my own curiosity.

The servers we use are each fully independent.
They each hold over 1000 customer websites and each run their own ftp,
apache, qmail, mysql and dns (primary and secondary as eth0 aliases).

It all sounds very cheeky but I can assure you, it works well!

I want to find out if my concerns are even warranted over these servers
being "jacks-of-all-trades". - Damn - a leading question if ever there
was one!
My concerns are just gut-reaction-by-analogy:
If every cafe employee has to cook, serve and wash-up, a lot of time is
wasted in changing clothes,washing hands etc. .. but if each has their
own job, more customers get fed.

Is there any problem with the machine's function changing continuously?
Is there a noticeable overhead in making memory room for a new set of
libraries to service one mysql request and then swapping them out again
to be a dns server and then swapping again to scan for a virus in an
incoming email etc. ?

These machines are rapidly being upgraded from 512Mb to 1Gb memory, they
are on a 2.2 kernel with no DMA.

I'm really interested in the facts and figures - but anecdotes are fun;-)

--
Andy Richardson

Never criticize a man 'til you've walked a mile in his shoes.
After that, you can say what you like..
'cos you're a mile away and you've got his shoes.

At Sat, 09 Dec 2006 12:52:41 +0000 Andy <andy-(E-Mail Removed)> wrote:

>
> G'day folks
> I must stress from the outset, my networking and linux knowledge is all
> from books, and my real-world experience amounts to 3months in total in
> my new job. I should also note that nothing will change, this is all out
> of my own curiosity.
>
> The servers we use are each fully independent.
> They each hold over 1000 customer websites and each run their own ftp,
> apache, qmail, mysql and dns (primary and secondary as eth0 aliases).
>
> It all sounds very cheeky but I can assure you, it works well!
>
> I want to find out if my concerns are even warranted over these servers
> being "jacks-of-all-trades". - Damn - a leading question if ever there
> was one!
> My concerns are just gut-reaction-by-analogy:
> If every cafe employee has to cook, serve and wash-up, a lot of time is
> wasted in changing clothes,washing hands etc. .. but if each has their
> own job, more customers get fed.
>
> Is there any problem with the machine's function changing continuously?
> Is there a noticeable overhead in making memory room for a new set of
> libraries to service one mysql request and then swapping them out again
> to be a dns server and then swapping again to scan for a virus in an
> incoming email etc. ?

No. Unlike your cafe analogy, the different services (functions) use
different resources. Also none are 'continious'. The computer is like
a person with multiple sets of hands, etc.: one pair up to it elbows in
dishwater, another pair working at the stove, and a third pair (with its
own pair of feet) out in the dining area serving food.

Note that under linux, shared code really is shared. And common file
accesses end up in memory-resident cache. Linux does not 'swap' at all
unless the machines are severerly memory starved.

>
> These machines are rapidly being upgraded from 512Mb to 1Gb memory, they

512Mb (on a *server*) is a long way from memory starved. 1Gb is a huge
amount of RAM for what you are doing. Apache, bind, and sendmail
will do thier thing quite happily in 256meg (all together) They can . MySql
*might* use more if the databases are large or complex:

sauron.deepsoft.com% ps aux | grep httpd
root 3427 0.0 2.8 18352 7356 ? Ss Dec01 0:17 /usr/sbin/httpd
apache 8791 0.0 1.5 18512 3908 ? S Dec03 0:00 /usr/sbin/httpd
apache 8792 0.0 1.5 18488 3888 ? S Dec03 0:00 /usr/sbin/httpd
apache 8793 0.0 1.4 18512 3828 ? S Dec03 0:00 /usr/sbin/httpd
apache 8794 0.0 1.5 18492 3876 ? S Dec03 0:00 /usr/sbin/httpd
apache 8795 0.0 1.5 18488 3880 ? S Dec03 0:00 /usr/sbin/httpd
apache 8796 0.0 1.5 18492 3876 ? S Dec03 0:00 /usr/sbin/httpd
apache 8797 0.0 1.5 18492 3872 ? S Dec03 0:00 /usr/sbin/httpd
apache 8798 0.0 1.5 18492 3880 ? S Dec03 0:00 /usr/sbin/httpd
sauron.deepsoft.com% echo '9*18.5'|bc
166.5
sauron.deepsoft.com% echo '(8*4)+7.3' | bc
39.3

All Apache (2.0.52): 166.5M virt, 40M resident

sauron.deepsoft.com% ps aux | grep sendmail
root 3275 0.0 0.8 7236 2064 ? Ss Dec01 0:01 sendmail: accepting connections
smmsp 3283 0.0 0.6 7680 1596 ? Ss Dec01 0:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue

Sendmail: 15meg virt, 3.5meg resident

sauron.deepsoft.com% ps aux | grep named
named 3132 0.0 1.2 38296 3084 ? Ssl Dec01 1:34 /usr/sbin/named -u named -t /var/named/chroot

Named (bind): 40meg virt, 3meg resident

Totals: 221.5 Virt, 46.5Meg resident

(This is on a *desktop* machine that is also a server on a LAN, running
CentOS 4.4 on a K6-500 with 256Meg of RAM.)

This machine *also* runs *both* MySql (when I start it) AND PostgreSQL:

sauron.deepsoft.com% ps aux | grep -i sql
postgres 3238 0.0 0.7 20012 2020 ? S Dec01 0:03 /usr/bin/postmaster -p 5432 -D /var/lib/pgsql/data -i -h 127.0.0.1
root 4080 0.6 0.4 5280 1084 pts/1 S 18:22 0:00 /bin/sh /usr/bin/mysqld_safe --defaults-file=/etc/my.cnf --pid-file=/var/run/mysqld/mysqld.pid
mysql 4113 5.9 6.9 125456 17892 pts/1 Sl 18:22 0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-locking --socket=/var/lib/mysql/mysql.sock

(MySql is far more of a memory hog than postgresSQL!)

Note that even will all of this running the machine *could* have
*everything* memory resident, if it wanted to, with only 256meg of RAM.
(It does not need to most of the time.) If it had 512Meg of RAM it
could have everything memory resident AND have a large cache of file
data in memory as well. The machine could be doing things without
hardly *every* touching the disk drives at all most of the time. The
only real bottleneck would be network bandwidth.

Apache, Sendmail, and Bind use very little CPU time -- all are pretty
much I/O bound (Disk -> memory -> network). The database backends will
use most of the 'spare' CPU cycles (processing SQL statements), but will
also be somewhat dominated by I/O constraints (Disk -> memory -> network).

> are on a 2.2 kernel with no DMA.

No DMA?!? What sort of disk drives? Don't tell me plain IDE in PIO mode
(dumb for a server). You really ought to be using at least SCSI disks
for servers (all modern (read PCI) SCSI controllers use DMA transfers).
Or SATA on newer machines.

>
> I'm really interested in the facts and figures - but anecdotes are fun;-)
>

--
Robert Heller -- 978-544-6933
Deepwoods Software -- Linux Installation and Administration
http://www.deepsoft.com/ -- Web Hosting, with CGI and Database
(E-Mail Removed) -- Contract Programming: C/C++, Tcl/Tk

In comp.os.linux.misc Robert Heller <(E-Mail Removed)>:
> At Sat, 09 Dec 2006 12:52:41 +0000 Andy <andy-(E-Mail Removed)> wrote:

[ running various services on the same machine at once? ]

>> to be a dns server and then swapping again to scan for a virus in an
>> incoming email etc. ?

> No. Unlike your cafe analogy, the different services (functions) use
> different resources. Also none are 'continious'. The computer is like
> a person with multiple sets of hands, etc.: one pair up to it elbows in
> dishwater, another pair working at the stove, and a third pair (with its
> own pair of feet) out in the dining area serving food.

> Note that under linux, shared code really is shared. And common file
> accesses end up in memory-resident cache. Linux does not 'swap' at all
> unless the machines are severerly memory starved.


>> These machines are rapidly being upgraded from 512Mb to 1Gb memory, they

> 512Mb (on a *server*) is a long way from memory starved. 1Gb is a huge
> amount of RAM for what you are doing. Apache, bind, and sendmail
> will do thier thing quite happily in 256meg (all together) They can . MySql
> *might* use more if the databases are large or complex:

I'd be careful with that and monitor workload, context
switches/etc pretty closely.

He was talking about virus scanning which can use quite some RAM,
he might in addition want to run SA, which is even more RAM
hungry and can make 1GB not that much if the box is busy fighting
todays spam loads. You might not be able to run more much more
then 20-30 x spamd at the same time with one gig RAM.

Databases tend to be extremely picky about their memory, though it
depends of course on the workload. I's suggest to monitor with
using at least sar for some time to get a picture what is really
going on and where if any possible shortcomings are. Or if you
can easily put more services on a box that is idling anyway?

[..]
--
Michael Heiming (X-PGP-Sig > GPG-Key ID: EDD27B94)
mail: echo (E-Mail Removed) | perl -pe 'y/a-z/n-za-m/'
#bofh excuse 133: It's not plugged in.

Thanks Robert and Michael for such thorough responses.

> No. Unlike your cafe analogy, the different services (functions) use
> different resources. Also none are 'continious'. The computer is
> like a person with multiple sets of hands, etc.: one pair up to it
> elbows in dishwater, another pair working at the stove, and a third
> pair (with its own pair of feet) out in the dining area serving food.
>

Thanks for continuing with the analogy ...

> sauron.deepsoft.com% echo '9*18.5'|bc
> 166.5
> sauron.deepsoft.com% echo '(8*4)+7.3' | bc
> 39.3
>
> All Apache (2.0.52): 166.5M virt, 40M resident

....and for the calculations I can get on and do for our servers.

I have just checked on twenty of the servers at 2o'clock on a Sunday
afternoon and according to pstree they vary from between 60 and 150
apache processes. (Although I guess many may be in the tcp wait state.
Actually I'd better look at the one with 150apache processes, I know
exactly which customer it is and what they're doing.

> 512Mb (on a *server*) is a long way from memory starved. 1Gb is
> a huge amount of RAM for what you are doing. Apache, bind, and
> sendmail will do thier thing quite happily in 256meg (all together)
> They can . MySql *might* use more if the databases are large or
> complex:

...and cat /proc/meminfo
says that none of them are using less than 20Mb of the swap partition.
I do get a little warning bubble on our status page when a sizeable
amount of swap is used.

> He was talking about virus scanning which can use quite some RAM,
> he might in addition want to run SA, which is even more RAM
> hungry and can make 1GB not that much if the box is busy fighting
> todays spam loads. You might not be able to run more much more
> then 20-30 x spamd at the same time with one gig RAM.

Yep everything grinds to a halt when a few clamscan processes start running.
We use qmail, and one customer on server gets 20,000 emails per hour.
All of them "pop_user_does_not_exist_will_deliver_to_postmaste r" and
then postmaster is set to bounce and we have to process them all the way
back out again. (of course to yet more non-existant addresses so they
sit in the outgoing queue for a week)

I have just discovered magic-smtpd and am trying to get its
valid_user_checking options working so at least we can block them at the
smtp(incoming) stage.

> MySql *might* use more if the databases are large or complex:

> Databases tend to be extremely picky about their memory, though it
> depends of course on the workload. I's suggest to monitor with
> using at least sar for some time to get a picture what is really
> going on and where if any possible shortcomings are. Or if you
> can easily put more services on a box that is idling anyway?

You are both right again.
Recently one server's load was up between 20 and 160 all day every day
- owch!
Customers complaining of php mysql timeouts all over the place.

My colleague guessed it was time to try using another server to deal
with its MySQL requests.

Now neither server's load gets above 1.5
That's what got me thinking about the multifunction issue... but it
seems that giving MySQL its own server was a good choice. I can't take
credit for any of it, I was just stunned at the effect. Double the
number of servers and we can service up to ten times more sites.
(assuming I can bare the thought of the loads going that high again...
which I can't)

>> are on a 2.2 kernel with no DMA.
>
> No DMA?!? What sort of disk drives? Don't tell me plain IDE in
> PIO mode (dumb for a server).

Yes, standard off-the-shelf IDE Western Digital or Maxtor drives
on 40way IDE cables thought the ones going into use now have 80core.
Maybe just a faster spindle version of what I used in my 286machine.

> You really ought to be using at least SCSI disks for servers
> (all modern (read PCI) SCSI controllers use DMA transfers). Or
> SATA on newer machines.

We are moving to CentOS4.4 from redhat7.3 but still no sata or scsi. I
guess the thinking is part money and part if-it-works-keep-it.

I'm sure if there were an animal rights movement for servers, we'd be
banned from ever keeping a server again, they are sure working hard.


Incidentally, after learning lots of how-it-should-be-done from lots of
books and exams, its interesting to find out how it's done in the real
world. I just hope I'm not picking up bad habits.

Andy


--
Andy Richardson

Never criticize a man 'til you've walked a mile in his shoes.
After that, you can say what you like..
'cos you're a mile away and you've got his shoes.

In comp.os.linux.misc Andy <andy-(E-Mail Removed)>:
[ stuff ]

> Yep everything grinds to a halt when a few clamscan processes start running.
> We use qmail, and one customer on server gets 20,000 emails per hour.
> All of them "pop_user_does_not_exist_will_deliver_to_postmaste r" and
> then postmaster is set to bounce and we have to process them all the way
> back out again. (of course to yet more non-existant addresses so they
> sit in the outgoing queue for a week)

> I have just discovered magic-smtpd and am trying to get its
> valid_user_checking options working so at least we can block them at the
> smtp(incoming) stage.

Please do so, sending bounces is a rather bad idea, spam sender
addresses are faked to about 100% so you are just hitting even
more innocent people with your bounces.

> >> are on a 2.2 kernel with no DMA.
[..]
> We are moving to CentOS4.4 from redhat7.3 but still no sata or scsi. I
> guess the thinking is part money and part if-it-works-keep-it.

RH 7.3 used kernel 2.4. I'd suggest to look into exim + sa-exim
both come with RHEL/CentOS 4 and offer sophisticated anti spam.

Good luck

[..]

--
Michael Heiming (X-PGP-Sig > GPG-Key ID: EDD27B94)
mail: echo (E-Mail Removed) | perl -pe 'y/a-z/n-za-m/'
#bofh excuse 389: /dev/clue was linked to /dev/null

Michael Heiming wrote:

>> I have just discovered magic-smtpd and am trying to get its
>> valid_user_checking options working so at least we can block them at the
>> smtp(incoming) stage.
>
> Please do so, sending bounces is a rather bad idea, spam sender
> addresses are faked to about 100% so you are just hitting even
> more innocent people with your bounces.

will do.. having a permissions problem and I'm trying to
avoid setuid root as much as possible even though the rest
of the server seems rife with them

>>>> are on a 2.2 kernel with no DMA.
> [..]
>> We are moving to CentOS4.4 from redhat7.3 but still no sata or scsi. I
>> guess the thinking is part money and part if-it-works-keep-it.
>
> RH 7.3 used kernel 2.4. I'd suggest to look into exim + sa-exim
> both come with RHEL/CentOS 4 and offer sophisticated anti spam.

Hmm, I guess the 7.3CD floating round the office was just
for playing purposes... uname -r definitely returns 2.2.25

> Good luck
Thanks again for your help

--
Andy Richardson

Never criticize a man 'til you've walked a mile in his shoes.
After that, you can say what you like..
'cos you're a mile away and you've got his shoes.