Do named pipes have disk I/O ?? [Unix Programming]

Prev: Learn about proxy sites and how to use them to open blocked sites unlimited downloads from RapidShare and megaupload and increase the speed of the Internet with new sites for free
Next: Why can't I catch a SIGBUS on my Mac (when strlen(NULL) is used)?

From: Peter Olcott on 6 Apr 2010 16:41

"Scott Lurndal" <scott(a)slp53.sl.home> wrote in message
news:MTMun.92150$DU3.68111(a)news.usenetserver.com...
> "Peter Olcott" <NoSpam(a)OCR4Screen.com> writes:
>>
>>"Scott Lurndal" <scott(a)slp53.sl.home> wrote in message
>>news:1FKun.92148$DU3.33231(a)news.usenetserver.com...
>>> "Peter Olcott" <NoSpam(a)OCR4Screen.com> writes:
>>>>
>>>>"Scott Lurndal" <scott(a)slp53.sl.home> wrote in message
>>>>news:D1Kun.92145$DU3.83045(a)news.usenetserver.com...
>>>>> "Peter Olcott" <NoSpam(a)OCR4Screen.com> writes:
>>>>>>
>>>>>>"Ian Collins" <ian-news(a)hotmail.com> wrote in message
>>>>>>news:81vfqpFa9rU3(a)mid.individual.net...
>>>>>>> On 04/ 6/10 09:27 AM, Peter Olcott wrote:
>>>>>>>
>>>>>>> I know you are using an awful client, but please fix
>>>>>>> your
>>>>>>> quoting!
>>>>>>>
>>>>>>>> I was going to do the thread priority thing very
>>>>>>>> simply.
>>>>>>>> Only one thread can run at a time, and the purpose
>>>>>>>> of
>>>>>>>> the
>>>>>>>> multiple threads was to make fast context switching
>>>>>>>> using
>>>>>>>> thread local data.
>>>>>
>>>>> maintain a single, priority ordered, queue of pending
>>>>> work
>>>>> items. Create one logical thread per physical thread
>>>>> or
>>>>> core
>>>>> and have each grab the next element from the top of
>>>>> the
>>>>> queue.
>>>>>
>>>>> New requests are inserted in priority order into the
>>>>> queue.
>>>>>
>>>>> Note this can result in starvation for lower priority
>>>>> requests,
>>>>> if higher priority requests occur with sufficient
>>>>> frequency,
>>>>> but it avoids the extremely silly idea of using thread
>>>>> priority.
>>>>>
>>>>> scott
>>>>
>>>>That won't work because this design could start on the
>>>>3.5
>>>>minute job, and not even be aware of the 50 ms job until
>>>>it
>>>>completes. I think that I will have a simple FIFO queue
>>>>at
>>>>first. Later on I will have at least two FIFO queues
>>>>where
>>>>one preempts the other. There won't be very many 3.5
>>>>minute
>>>>jobs, and all of the other jobs will be fast enough to
>>>>start
>>>>with.
>>>
>>> Given that all modern processors have at least two
>>> hardware
>>> threads/cores and for an application like this, you'll
>>> probably
>>> want a 4, 6 or 12 core processor (Shanghai, Istanbul or
>>> Magney-cours).
>>>
>>> With 12 hardware threads, you'd need 12 3.5 minute jobs
>>> all
>>> happening at the same time to starve the 50ms job. In
>>> any
>>> case, have two queues then, one for short jobs and one
>>> for
>>> long
>>> jobs and partition the hardware threads appropriately
>>> amongst
>>> the queues.
>>>
>>> scott
>>
>>I am only focusing on the design of the preemptive
>>scheduling now. Also the initial hardware will only have
>>a
>>single core with hyperthreading.
>>
>
> You are making this far more complicated than necessary.
> Have fun.
>
> scott

So then do you have a simple way to implement preemptive
scheduling?
I always aim for the simplest possible way to meet the
specs.
Initially, I will also simplify the specs, but, I want to
plan for the future, too.

From: Rainer Weikusat on 6 Apr 2010 16:45

"Peter Olcott" <NoSpam(a)OCR4Screen.com> writes:
> "Scott Lurndal" <scott(a)slp53.sl.home>: wrote in message
>> "Peter Olcott" <NoSpam(a)OCR4Screen.com> writes:

[...]

>>>I am only focusing on the design of the preemptive scheduling now.
>>>Also the initial hardware will only have a single core with
>>>hyperthreading.
>>>
>>
>> You are making this far more complicated than necessary.
>> Have fun.
>>
>> scott
>
> So then do you have a simple way to implement preemptive
> scheduling?

If you are convinced that this is the solution to your problem, the
kernel already provides exactly that. You just have to use the
existing facilities.

From: Peter Olcott on 6 Apr 2010 17:11

"Rainer Weikusat" <rweikusat(a)mssgmbh.com> wrote in message
news:87d3ycnwcl.fsf(a)fever.mssgmbh.com...
> "Peter Olcott" <NoSpam(a)OCR4Screen.com> writes:
>> "Scott Lurndal" <scott(a)slp53.sl.home>: wrote in message
>>> "Peter Olcott" <NoSpam(a)OCR4Screen.com> writes:
>
> [...]
>
>>>>I am only focusing on the design of the preemptive
>>>>scheduling now.
>>>>Also the initial hardware will only have a single core
>>>>with
>>>>hyperthreading.
>>>>
>>>
>>> You are making this far more complicated than necessary.
>>> Have fun.
>>>
>>> scott
>>
>> So then do you have a simple way to implement preemptive
>> scheduling?
>
> If you are convinced that this is the solution to your
> problem, the
> kernel already provides exactly that. You just have to use
> the
> existing facilities.

So the OS will put a thread to sleep, saving its
multi-megabyte state upon a process triggered event?

I would save the state with a simple change of the integer
subscript into an array of each threads data. Could the OS
be told that this is what is needed?

From: David Schwartz on 6 Apr 2010 19:54

On Apr 6, 5:36 am, "Peter Olcott" <NoS...(a)OCR4Screen.com> wrote:

> --Suppose the memory allocator lock is held by a
> lower-priority task. If

> There is no need for a memory allocator lock because memory
> will not be shared across threads.

Memory is shared across threads no matter what you do. Multi-threaded
programs have locks around their memory allocators. There's nothing
you can do about it.

Any function you call may wind up acquiring a lock somewhere in its
path -- if the thread is pre-empted during the execution of that
function, no other thread that needs that lock can make forward
progress until all threads at a higher priority than the pre-empted
thread are no longer ready-to-run.

Trust me, priorities are *not* the solution to your problem. Coding
your threads not to do work you don't want done is.

DS

From: Peter Olcott on 6 Apr 2010 21:06

Although I do understand the details a little better it is
still easy enough.

If you think that priorities are not the solution to my
problem then how can I make sure that certain HTTP requests
get executed immediately in strict FIFO order, and other
HTTP requests only get executed when there are no top
priority HTTP requests?

The first requests take 50 ms the second requests take 3.5
minutes. Although there may be periods of 100 ms free quite
often periods of 3.5 minutes might never be free.

Maybe the 3.5 minute job needs its own process. Is it easy
to put a process to sleep and wake it up again?

"David Schwartz" <davids(a)webmaster.com> wrote in message
news:1618b298-12f0-4228-942c-abc2055b535c(a)e7g2000yqf.googlegroups.com...
On Apr 6, 5:36 am, "Peter Olcott" <NoS...(a)OCR4Screen.com>
wrote:

> --Suppose the memory allocator lock is held by a
> lower-priority task. If

> There is no need for a memory allocator lock because
> memory
> will not be shared across threads.

Memory is shared across threads no matter what you do.
Multi-threaded
programs have locks around their memory allocators. There's
nothing
you can do about it.

Any function you call may wind up acquiring a lock somewhere
in its
path -- if the thread is pre-empted during the execution of
that
function, no other thread that needs that lock can make
forward
progress until all threads at a higher priority than the
pre-empted
thread are no longer ready-to-run.

Trust me, priorities are *not* the solution to your problem.
Coding
your threads not to do work you don't want done is.

DS