Bulletproofing a Linux server

I am building a server and I want to make sure that it is
"bulletproof", which, to me, means that it can last a long time and
survive adversity without too much attention and would have a great
uptime.

It will be hosted in a datacenter, and I do not want to visit it too
often. My current server is 3+ years old, and I saw it exactly once
during this time, when I added extra memory to it.

The server is still around, but its drives are not mirrored and in
fact paired with a big partition that spans them both. (dumb idea) So
I am in a precarious position. Plus it is too slow.

The new server, I want to last 5 years comfortably.

To that end, I have or plan on the following features.

1. Three pairs of mirrored drives managed by a 3ware controller (disk
reliability)

2. Active CPU Cooling (with fans)

3. Use cpufreqd to reduce CPU speed if overheats anyway despite active
cooling

4. Use smartmontools to monitor hard drives

5. Get a little too much on the performance side: 15k RPM drives for
things that matter, and 16 GB of RAM. (RAM being so cheap today, this
is a no brainer, extra 8 GB of registered 667 MHZ memory cost just
$287)

6. Get an IPMI card, so that even if the computer crashes, goes down,
cannot boot etc, I can recover remotely.

Is there something that I missed? Any good thoughts?

i

Ignoramus24141 wrote:
> I am building a server and I want to make sure that it is
> "bulletproof", which, to me, means that it can last a long time and
> survive adversity without too much attention and would have a great
> uptime.
>
> It will be hosted in a datacenter, and I do not want to visit it too
> often. My current server is 3+ years old, and I saw it exactly once
> during this time, when I added extra memory to it.
>
> The server is still around, but its drives are not mirrored and in
> fact paired with a big partition that spans them both. (dumb idea) So
> I am in a precarious position. Plus it is too slow.
>
> The new server, I want to last 5 years comfortably.
>
> To that end, I have or plan on the following features.
>
> 1. Three pairs of mirrored drives managed by a 3ware controller (disk
> reliability)
>
> 2. Active CPU Cooling (with fans)
>
> 3. Use cpufreqd to reduce CPU speed if overheats anyway despite active
> cooling
>
> 4. Use smartmontools to monitor hard drives
>
> 5. Get a little too much on the performance side: 15k RPM drives for
> things that matter, and 16 GB of RAM. (RAM being so cheap today, this
> is a no brainer, extra 8 GB of registered 667 MHZ memory cost just
> $287)
>
> 6. Get an IPMI card, so that even if the computer crashes, goes down,
> cannot boot etc, I can recover remotely.
>
> Is there something that I missed? Any good thoughts?
>
> i

Backup, especially snapshot capable backup, and robust status monitoring with
thought about when and what to report come to mind.

On 2008-02-25, Nico Kadel-Garcia <(E-Mail Removed)> wrote:
> Ignoramus24141 wrote:
>> I am building a server and I want to make sure that it is
>> "bulletproof", which, to me, means that it can last a long time and
>> survive adversity without too much attention and would have a great
>> uptime.
>>
>> It will be hosted in a datacenter, and I do not want to visit it too
>> often. My current server is 3+ years old, and I saw it exactly once
>> during this time, when I added extra memory to it.
>>
>> The server is still around, but its drives are not mirrored and in
>> fact paired with a big partition that spans them both. (dumb idea) So
>> I am in a precarious position. Plus it is too slow.
>>
>> The new server, I want to last 5 years comfortably.
>>
>> To that end, I have or plan on the following features.
>>
>> 1. Three pairs of mirrored drives managed by a 3ware controller (disk
>> reliability)
>>
>> 2. Active CPU Cooling (with fans)
>>
>> 3. Use cpufreqd to reduce CPU speed if overheats anyway despite active
>> cooling
>>
>> 4. Use smartmontools to monitor hard drives
>>
>> 5. Get a little too much on the performance side: 15k RPM drives for
>> things that matter, and 16 GB of RAM. (RAM being so cheap today, this
>> is a no brainer, extra 8 GB of registered 667 MHZ memory cost just
>> $287)
>>
>> 6. Get an IPMI card, so that even if the computer crashes, goes down,
>> cannot boot etc, I can recover remotely.
>>
>> Is there something that I missed? Any good thoughts?
>>
>> i
>
> Backup, especially snapshot capable backup, and robust status monitoring with
> thought about when and what to report come to mind.

I agree on backups. My source code and most OS config files that I
modify, lives in CVS. So it is backed up.

I bought a mirrored 500 GB USB drive to use for file based backups.

I did some scripting on status monitoring, but not enough. What I have
does, however, detect when my apache is down or not responsive (due to
bugs in my mod_perl scripts) and restarts it.

i

In comp.os.linux.misc Ignoramus24141 <(E-Mail Removed)> wrote:
> I am building a server and I want to make sure that it is
> "bulletproof", which, to me, means that it can last a long time and
> survive adversity without too much attention and would have a great
> uptime.
>
> It will be hosted in a datacenter, and I do not want to visit it too
> often. My current server is 3+ years old, and I saw it exactly once
> during this time, when I added extra memory to it.
>
> The server is still around, but its drives are not mirrored and in
> fact paired with a big partition that spans them both. (dumb idea) So
> I am in a precarious position. Plus it is too slow.
>
> The new server, I want to last 5 years comfortably.
>
> To that end, I have or plan on the following features.
>
> 1. Three pairs of mirrored drives managed by a 3ware controller (disk
> reliability)
>
> 2. Active CPU Cooling (with fans)
>
> 3. Use cpufreqd to reduce CPU speed if overheats anyway despite active
> cooling
>
> 4. Use smartmontools to monitor hard drives
>
> 5. Get a little too much on the performance side: 15k RPM drives for
> things that matter, and 16 GB of RAM. (RAM being so cheap today, this
> is a no brainer, extra 8 GB of registered 667 MHZ memory cost just
> $287)
>
> 6. Get an IPMI card, so that even if the computer crashes, goes down,
> cannot boot etc, I can recover remotely.
>
> Is there something that I missed? Any good thoughts?
>

three important items:
(1) rendundant ( prefereably hot-swap ) power supplies
(2) cheapo lowend RAM is NOT ECC - plus the more RAM you have
the more likely to have hardware problem. Unless you need
16GB RAM, scale it back to what you need
(3) 15kRPM drives are _not_ the choice for reliability

more is not necessarily better

Stan

--
Stan Bischof ("stan" at the below domain)
www.worldbadminton.com

On Mon, 25 Feb 2008 11:13:24 -0600, Ignoramus24141 wrote:
> I am building a server and I want to make sure that it is
> "bulletproof", which, to me, means that it can last a long time and
> survive adversity without too much attention and would have a great
> uptime.


maybe add an Intrusion Detection System

http://sourceforge.net/projects/aide
http://la-samhna.de/samhain/s_documentation.html

On 2008-02-25, (E-Mail Removed) <(E-Mail Removed)> wrote:
> In comp.os.linux.misc Ignoramus24141 <(E-Mail Removed)> wrote:
>> I am building a server and I want to make sure that it is
>> "bulletproof", which, to me, means that it can last a long time and
>> survive adversity without too much attention and would have a great
>> uptime.
>>
>> It will be hosted in a datacenter, and I do not want to visit it too
>> often. My current server is 3+ years old, and I saw it exactly once
>> during this time, when I added extra memory to it.
>>
>> The server is still around, but its drives are not mirrored and in
>> fact paired with a big partition that spans them both. (dumb idea) So
>> I am in a precarious position. Plus it is too slow.
>>
>> The new server, I want to last 5 years comfortably.
>>
>> To that end, I have or plan on the following features.
>>
>> 1. Three pairs of mirrored drives managed by a 3ware controller (disk
>> reliability)
>>
>> 2. Active CPU Cooling (with fans)
>>
>> 3. Use cpufreqd to reduce CPU speed if overheats anyway despite active
>> cooling
>>
>> 4. Use smartmontools to monitor hard drives
>>
>> 5. Get a little too much on the performance side: 15k RPM drives for
>> things that matter, and 16 GB of RAM. (RAM being so cheap today, this
>> is a no brainer, extra 8 GB of registered 667 MHZ memory cost just
>> $287)
>>
>> 6. Get an IPMI card, so that even if the computer crashes, goes down,
>> cannot boot etc, I can recover remotely.
>>
>> Is there something that I missed? Any good thoughts?
>>
>
> three important items:
> (1) rendundant ( prefereably hot-swap ) power supplies

My server has this.

> (2) cheapo lowend RAM is NOT ECC - plus the more RAM you have
> the more likely to have hardware problem. Unless you need
> 16GB RAM, scale it back to what you need

My extra RAM that I bought this morning is this:

http://www.newegg.com/Product/Produc...82E16820134328

Kingston 2GB 240-Pin DDR2 SDRAM ECC Registered DDR2 667 (PC2 5300)
Server Memory - Retail

The 8 GB of memory that was shipped to me with the server, is of the
same kind exactly (667 MHz DDR2 Registered ECC).

> (3) 15kRPM drives are _not_ the choice for reliability
>
> more is not necessarily better

I figure, that with mirroring, which I have, reliability of hard
drives is not as big of an issue.

The reason why I have three pairs is this. The first pair would host
root and swap. The second pair would host the websites that I want to
be fast regardless of what other things are being done. The third pair
would hold "everything else", all my other sites and projects under
/home and one database which is big, but is OK to be slow.

I do not want to cause reindexing, or other slow accesses, to the
aforementioned database or doing other stuff under /home, like "find",
to affect the websites and databases designated as "fast".

The first pair is two 74 GB 15k RPM SAS drives, the second pair is two
146GB 15k RPM SAS drives, and the third pair is a pair of 500 GB SATA
drives.

i

Ignoramus24141 <(E-Mail Removed)> burped up warm pablum in
news4SdnY3WqNSpal_anZ2dnUVZ_g-(E-Mail Removed):

> I am building a server and I want to make sure that it is
> "bulletproof", which, to me, means that it can last a long time and
> survive adversity without too much attention and would have a great
> uptime.

> It will be hosted in a datacenter, and I do not want to visit it too
> often. My current server is 3+ years old, and I saw it exactly once
> during this time, when I added extra memory to it.

You will also need "bulletproof" software to handle all that hardware. I recommend Windows 98
for its superb hardware support and crash-free reliability.

--
Tris "Barking Mad" Orendorff
[ Anyone naming their child should spend a few minutes checking rhyming slang and dodgy
sounding names. Brad and Angelina failed to do this when naming their kid Shiloh Pitt. At some
point, someone at school is going to spoonerise her name.
Craig Stark ]

On 2008-02-26, Tris Orendorff <(E-Mail Removed)> wrote:
> Ignoramus24141 <(E-Mail Removed)> burped up warm pablum in
> news4SdnY3WqNSpal_anZ2dnUVZ_g-(E-Mail Removed):
>
>> I am building a server and I want to make sure that it is
>> "bulletproof", which, to me, means that it can last a long time and
>> survive adversity without too much attention and would have a great
>> uptime.
>
>> It will be hosted in a datacenter, and I do not want to visit it too
>> often. My current server is 3+ years old, and I saw it exactly once
>> during this time, when I added extra memory to it.
>
> You will also need "bulletproof" software to handle all that hardware. I recommend Windows 98
> for its superb hardware support and crash-free reliability.
>

Why did not you suggest Windows ME?

i

Ignoramus24141 <ignoramus24...@NOSPAM.24141.invalid> wrote:
>
> Is there something that I missed? Any good thoughts?

There is strength in numbers - Consider a mirror site updated
by your favorite mirroring software. For your purposes it could
be a daily sync cold mirror and work just fine. If it really hist
the fan a DNS change can point to the mirror while you work
on the primary.

On 2008-02-27, Doug Freyburger <(E-Mail Removed)> wrote:
> Ignoramus24141 <ignoramus24...@NOSPAM.24141.invalid> wrote:
>>
>> Is there something that I missed? Any good thoughts?
>
> There is strength in numbers - Consider a mirror site updated
> by your favorite mirroring software. For your purposes it could
> be a daily sync cold mirror and work just fine. If it really hist
> the fan a DNS change can point to the mirror while you work
> on the primary.

Doug, thanks. In fact, I do have a backup server at home, not 100%
ready to go at any moment, but I would say 99%. The STUMP stuff is
also saved daily. Still, the issue of having a backup server at home
is that even though I am on a very good cable plan (10/2 MBPS), it is
barely enough to handle my bandwidth. So, while backups are important,
I want the primary server to not go down.

i