«

Hi,

I am using a multithreaded application, which hangs on RedHat 8.0
Linux.
The application is basically a server application which creates a
thread for processing each request. Each thread opens a socket, pumps
some data and would die.

If I send lesser number of requests, the application works correctly.
But, when a large number of requests are sent simultaneously, the
threads would hang.

This looks like a thread synchronization issue. But it is not
happening in RH 7.3

The kernel version is:
Red Hat Linux release 8.0 (Psyche)
Kernel 2.4.18-14 on an i686

And, glibc used is:
glibc-utils-2.3.2-4.80.8
glibc-debug-static-2.3.2-4.80.8
glibc-kernheaders-2.4-7.20
glibc-devel-2.3.2-4.80.8
glibc-2.3.2-4.80.8

Has anybody faced this kind of problem earlier?

Whether I need to install any patch for this version?

TIA,
Ramki

Ramki wrote:
> I am using a multithreaded application, which hangs on RedHat 8.0
> Linux.
> The application is basically a server application which creates a
> thread for processing each request. Each thread opens a socket, pumps
> some data and would die.
> If I send lesser number of requests, the application works correctly.
> But, when a large number of requests are sent simultaneously, the
> threads would hang.
> This looks like a thread synchronization issue. But it is not
> happening in RH 7.3.
> Has anyone faced this problem earlier?

I suspect an application programming error. When a server is given a large
number of requests and creates a new thread for each one (the "flaming
arrows" approach), any operating-system can deliver very poor performance
to the threads, especially if any of them wind up waiting on the same
resource. Yet this is an issue, not with the OS nor its version, but with
the design of the application. The threads are not "hanging," I think
you'll find, but they're all stoppered-up.

The problem needs to be addressed by limiting the number of threads that
exist, and by making them "not die" after each request is done. Rather, a
certain number of "workers" stand around waiting for a new request to come
in. They dequeue the request, process it, and then wait for another
request. If a flood of requests comes in, the /queue/ builds up but the
threads can process the queue with the same predictable throughput as
before.

Notice that I dismiss the comment, "it is not happening in RH 7.3." In my
prediction, you have simply been lucky.

> Notice that I dismiss the comment, "it is not happening in RH 7.3." In my
> prediction, you have simply been lucky.

Most likely it's on a different box. I've been writing multithreaded apps
for 10 years now and have seen different CPUs/memory/moboards lock apps up
and others never have a problem. It's all a timing issue which is changed
from one system to the next.

I agree with your assessment, it's going to be in the design. And your
queue approach is the right approach. Typically you create ( CPU_COUNT *
2 ) threads and let them handle the queue.

/dev/null wrote:
> I agree with your assessment, it's going to be in the design. And your
> queue approach is the right approach. Typically you create ( CPU_COUNT *
> 2 ) threads and let them handle the queue.

It can be _quite_ astonishing how much difference a control on the
multiprogramming level can make.

My first job was at a University, in a new IBM shop. We had engineering
students running simulation packages. When one user ran it, it completed
in about one minute. Two users, about 1.5 minutes each. Eight users...
HIT-THE-WALL. It could take three hours to finish any one! The IBM rep
was drawing up new sales documents. :-?

Instead, I put in a batch-system, and forced all of the simulation runs to
be submitted to batch. The batch system would allow no more than 2 copies
of the simulation to run concurrently, guaranteeing a predictable
completion rate of about 2 jobs a minute no matter how large the queue
became. (And since no one submits jobs "all at once," the typical queue
was actually no more than one job deep, most of the time.) The IBM rep
didn't get the upgrade-sale. We continued to use the same hardware for
several years.

The principle here is the same: look at a fast-food restauraunt. When you
arrive, the manager doesn't add water to an "instant food preparer" box,
assign the instantly-reconstituted employee to fix your meal, and then
shoot him or her, now does he? No, your order is efficiently subdivided
among several workers ("threads"), who process their part of your order and
then wait for the next ticket to pop up on their screen. There are queues
everywhere but orders get processed efficiently whether the waiting room is
full or empty. "All I ever needed to know about multiprogramming I learned
at McDonald's."

["Followup-To:" header set to comp.os.linux.setup.]
On 10 Nov 2004 00:41:01 -0800, Ramki
<(E-Mail Removed)> wrote:
>
> Has anybody faced this kind of problem earlier?
>
> Whether I need to install any patch for this version?
>
Try a 2.6 kernel, or at least a more recent 2.4 (I think they are up to
2.4.28 or 29), if you can get rid of whatever closed source software is
holding you back.


--
"How can 59,054,087 people be so dumb?" And that's the headline in
Britain, our closest ally. But here in America, as long as you-know-who
is president, we don't give a $%&# what the rest of the world thinks.

Ramki wrote:
> Hi,
> I am using a multithreaded application, which hangs on RedHat 8.0
> Linux.
> The application is basically a server application which creates a
> thread for processing each request. Each thread opens a socket, pumps
> some data and would die.
>
> If I send lesser number of requests, the application works correctly.
> But, when a large number of requests are sent simultaneously, the
> threads would hang.
>
> This looks like a thread synchronization issue. But it is not
> happening in RH 7.3

Basically, Ramki, "you were simply lucky" that the application didn't hang
on you in 7.3, and I suspect that if you had dumped excessive load on it
frequently enough, it would have.

The approach of creating a new thread for each request .. what I call the
"flaming arrow approach" .. is inherently flawed because it has no means to
control an overload. Sooner or later you'll find yourself with several
hundred threads, all slogging it out with one another, none of them getting
anywhere. Poor Linux (or Windows, or OS/X, or IBM MVS) doesn't stand a
chance against such a design.

A far, far better approach is to create a limited and controllable number of
threads, which do _not_ die after finishing a request. What happens
instead is that each incoming request is placed into a queue. The worker
threads sleep until they can remove a request from the queue, process it,
then go to sleep again.

A server designed like this will always provide a consistent and reliable
throughput of requests, and will never overload itself. Requests may pile
up in the queue momentarily, but you can always predict that "X" requests
per second will be processed, period. (If you wish, you can program the
"gatekeeper" to refuse incoming requests if the queue is found to contain
too-many unprocessed entries, e.g. as a defense against an intentional
DOS-flood.)