Can extra processing threads help in this case?
in [MFC]
|
From: Joseph M. Newcomer on 13 Apr 2010 15:43 See below... On Mon, 12 Apr 2010 20:39:10 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:6fe7s55u4fnmhjnq2qj3ctroi81iomsnk2(a)4ax.com... >> See below... >> On Mon, 12 Apr 2010 15:53:23 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >> >> Actually, the priority queue works better on a single core >> machine, but I've tried to >> explain why this works, and you have ignored the >> explanations. > >Yeah it works because of time slicing, but how is this >better than one of my two original proposals that also uses >time slicing? *** Can you prove that you are correct? **** > >>>How else do you block a DoS attack? >> **** >> YOU don't. You let your ISP block it for you. Note that >> your belief that the IP address >> is going to be a reasonable approach shows how little you >> understand about Internet >> protocols or (as I have explained in the past) how D-O-S >> attacks work (each packet has a >> forged IP address which is fictional, PRECISELY so you >> can't use the IP address to detect >> and block the attack!) >> joe > >Then what other basis would the ISP have? **** Bad news: none. They use more complex techniques and sophisticated front-end traffic analyzers. And they tend not to want to talk about the details because it might be possible to reverse-engineer a successful attack if you knew the details. Never mind that criminals don't mind buying a $100,000 Cisco box and reverse-engineering it by disassembling the code that runs in it. A million dollar investment, perhaps (counting people time) that will pay off in tens of millions of dollars of criminal activity... Utlmately, in the black-hat/white-hat battle for network security, the black hats currently have so many advantages that you just have to live with what happens. joe **** > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 13 Apr 2010 15:45 See below... On Mon, 12 Apr 2010 21:02:28 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:jpe7s55a5462g4436crrm86c2nva939loa(a)4ax.com... >> See below... >> On Mon, 12 Apr 2010 15:05:51 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >> >>>That sure sounds screwy to me. Of the 40 different >>>priority >>>levels available on Linux, a process with priority of 0 >>>would starve a process with priority of 1? That sure >>>sounds >>>screwy to me. Can you prove this? > >> **** >> Well, I believe I gave you a citation to the explanation >> of the linux scheduler and its >> anti-starvation algorithm, and even gave you the google >> search phrase >> linux scheduler starvation >> by which you could find it. And fundamentally, the answer >> is YES, highest priority thread >> wins, period. That's how it has worked for decades. The >> reason is that back in the days > > http://oreilly.com/catalog/linuxkernel/chapter/ch10.html > This may be one of several options now. I am not sure if I >fully understand this material yet, and since it is outdated >I am waiting form my updated copy. > >> when we were inventing timesharing, the schedulers tried >> really hard to give CPU >> percentage guarantees, and when we measured performance >> bottlenecks, we found that on >> multimillion-dollar mainframes with the computing >> horsepower of a 286, 37% of the CPU time >> was being spent in the scheduler. So in later systems >> (1970 and beyond) we opted for >> lean, mean schedulers that had trivial algorithms >> (highest-priority thread wins) and moved >> "policy" to other parts of the system (e.g., the Balance >> Set Manager in Windows, working >> set trimmers. etc.) because this refactoring reduced OS >> overheads and essentially >> guaranteed more CPU cycles to the apps, instead of to "OS >> maintenance". And it worked, >> and all modern systems use these patterns. > >I hope that you are right. > >> You're big on pattens. Recognize that the best patterns >> we know are not necessarily the >> patterns used to design PFOS. >> >> I can prove it for Windows just by pointing to Solomon & >> Russinovich's book; and you might >> try the google phrase I gave above, which details why >> linux needs and how it has an >> antistarvation algorithm. >> joe > >>>What I am saying is that telling me that it is bad without >>>telling me what is bad about it is far worse than useless. >>>In more than half of the cases now what was bad about my >>>design was not the design itself but the misconception of >>>it. Without explaining why you think it is bad, and only >>>saying that it is bad is really harassment and not >>>helpful. >> **** >> I though I had pointed out conclusively why MQMS >> architectures have problems and SQMS >> architectures work better. > >Yes *** As in "Yes, you have pointed it out conclusively, and I choose to ignore the computations by which you showed it is superior"? **** >and repeating that even an infinite number of times will >never count as sufficient reasoning. **** I choose not to repeat what I already wrote, and you have not disproven what I wrote. **** > >> And the paper you gave us to read about linux throughput >> emphasized, time and again, how they were using SQMS >> architectures to improve performance, >> and you continue your lengthy diatribes about how you are >> going to build magical >> mechanisms to stop low-priority threads from running, >> mechanisms which do not need to >> exist because they are solving nonexistent problems. > >Now that I understand how the Linux Scheduler probably works >that point is finally made. **** Oh, so we're right? joe **** > > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 13 Apr 2010 16:08 See below... On Mon, 12 Apr 2010 23:39:06 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:kqi7s5d0fprv9hj06qs523814qdadvovq9(a)4ax.com... >> See below... >> On Mon, 12 Apr 2010 14:51:04 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >> > >>>Whoops you already lost something there. There are TWO >>>count >>>em TWO options. Only one of the options involves yielding, >>>the other one involves time slicing. >> **** >> SO why do you even need an option of yielding? Gratuitous >> complexity that serves no >> useful purpose. THat's what the OS is for, so why do you >> need to duplicate its >> functionality? >> **** > >It depends upon exactly how the Linux scheduler works >whether or not it would provide any benefit. I really want >to provide absolute priority of the high priority jobs over >the lower priority jobs. If Linux can already do this then >the yielding option makes no sense. > >>>THIS DESIGN REQUIREMENT IS IMMUTABLE >>>The design goal is to somehow provide the means for the >>>high >>>priority jobs to has absolute priority over the lower >>>priority jobs these lower priority jobs all have equal >>>priority relative to one another. The proposed approach >>>must meet the above stated design goal as closely as >>>possible on a PENTIUM 4 computer. >> **** > >> MY PROSE IS IMMUTABLE said Hemingway, one day. Please >> reconcile your Hemingway allusion >> with a belief that your design is perfect. The notion >> that low priority threads "sleep" >> is at best an implementation consideration, NOT a design >> requirement, unless by > >EXACTLY WHERE DID I SAY SLEEP IN THE ABOVE DESIGN >REQUIREMENT? *** You didn't. You said it as part of the implementation strategy, which you keep confusing with a requirements document. **** > >Even when pointed out to you countless times that you read >things that I have not said you continue making this same >mistake. **** I don't want to waste the time going back to find the exact quote, but you DID say, in one message, that you were going to signal all the other processes to go to sleep while you processed the high-priority thread. If you deny this, I WILL go back and find the quote. Do you really want me to expend effort to prove you said it? **** > >Can I ask you a personal question? Are you in your eighties? >If these mistakes that you are making are not your fault I >will be much kinder and gentler with you. I had assumed that >they are merely a form of harassment. If you are doing the >best that you can and these mistakes are unintentional, I >sincerely and humbly apologize for uttering the slightest >nuance of any unkind word. **** I am 63. I have been in this business since 1964. That works out to being in my 46th year building systems. I've been doing Windows since 1990, and Unix from 1975-1990; multiprocessor operating systems starting in 1975 (including multithreading), I/O systems since 1965, device drivers since 1965, I have been an operating system developer and maintainer, I've done banking and payroll systems, compilers, large-scale real-time systems, and delivered real products to real customers, and provided customer support. I've done network programming since TCP/IP first came out, and delivered my first Windows network-based program in 1999. I have designed and taught courses in Windows Device Drivers, Windows Network Programming, and Windows System Programming, and have taught Introductory MFC, Advanced MFC, Win32 API programming, and COM programming. I have a Ph.D. from Carnegie Mellon University, was on the faculty for 7 years, was a founding scientist of the Software Engineering Institute, and was Senior Staff Programmer at Tartan (a compiler company, where I was the chief toolmaker) and Senior Research Scientist and CMU's Information Technology Center. Along this path, I learned a LOT about how to engineer reliable, robust, large-scale rapid-response systems. I've worked closely with people who have built large distributed file systems, and I went to a lot of lectures on file system design for transacted file systems, often presented by the inventors of these systems. I am not stupid, and I know what the limitations of the technology are, and how to deal with building and measuring large-scale systems. My undergradudate degree is in mathematics, and if they had not discontinued minor degrees the year I graduated, I would have been awarded a "minor degree" in physics. I know something about measurement of software systems, having done it for 15 years (and developed the tools to do it!). And I know when someone is making asinine statements that cannot be backed up with any form of reality. Your design methodology is worthy of a not-to-bright freshman; your technical decisions, apparently based on "sound reasoning", are completely wrong, because they have no actual justification for their behavior (other than a badly-remembered textbook, not fully read, and hearsay, conjecture, and quite possibly Tarot cards); your understanding of operating systems and their principles is marginal at best. And I do NOT feel generous or feel like being gentle. You really are totally clueless, and you have systematically ignored the advice of any number of experts, myself included, who clearly know more about any topic you are addressing than you do. You are two-faced, applying a double-standard; demanding WE illustrate "sound reasoning" but not holding yourself to the same standard. Apparently, all you have to do is say "I think X ought to behave THIS way" and it MUST behave that way because that is how you think it should behave, and if anyone even QUESTIONS your "sound reasoning", you tell them they are stupid. Yet you make some of the most foolish and outrageous statements on which you have based your implementation decisions, and without any substantiation, expect us to believe them! Get a life! joe > > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Peter Olcott on 13 Apr 2010 16:16 "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:gji9s55da0t1r8hfo49h3iac9vpi3adhsu(a)4ax.com... > See below... > On Mon, 12 Apr 2010 23:39:06 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: > >> >>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>message news:kqi7s5d0fprv9hj06qs523814qdadvovq9(a)4ax.com... >>> See below... >>> On Mon, 12 Apr 2010 14:51:04 -0500, "Peter Olcott" >>> <NoSpam(a)OCR4Screen.com> wrote: >>> >> >>>>Whoops you already lost something there. There are TWO >>>>count >>>>em TWO options. Only one of the options involves >>>>yielding, >>>>the other one involves time slicing. >>> **** >>> SO why do you even need an option of yielding? >>> Gratuitous >>> complexity that serves no >>> useful purpose. THat's what the OS is for, so why do >>> you >>> need to duplicate its >>> functionality? >>> **** >> >>It depends upon exactly how the Linux scheduler works >>whether or not it would provide any benefit. I really want >>to provide absolute priority of the high priority jobs >>over >>the lower priority jobs. If Linux can already do this then >>the yielding option makes no sense. >> >>>>THIS DESIGN REQUIREMENT IS IMMUTABLE >>>>The design goal is to somehow provide the means for the >>>>high >>>>priority jobs to has absolute priority over the lower >>>>priority jobs these lower priority jobs all have equal >>>>priority relative to one another. The proposed approach >>>>must meet the above stated design goal as closely as >>>>possible on a PENTIUM 4 computer. >>> **** >> >>> MY PROSE IS IMMUTABLE said Hemingway, one day. Please >>> reconcile your Hemingway allusion >>> with a belief that your design is perfect. The notion >>> that low priority threads "sleep" >>> is at best an implementation consideration, NOT a design >>> requirement, unless by >> >>EXACTLY WHERE DID I SAY SLEEP IN THE ABOVE DESIGN >>REQUIREMENT? > *** > You didn't. >>>The notion >>> that low priority threads "sleep" >>> is at best an implementation consideration, NOT a design >>> requirement, OK so you are not senile, you are just a very unkind person.
From: Joseph M. Newcomer on 13 Apr 2010 16:19
See below... On Mon, 12 Apr 2010 22:42:43 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:9of7s5ts91k4e8jerscd9mm86rml0d6ap4(a)4ax.com... >> Some useful terminology: > > >> There is no "load balancing" here; instead, the queues are >> disjoint and orthogonal. If >> one queue is empty, the process is dead, and will not >> execute at all. Thus, reducing >> throughput and maximizing delay times. Only when it gets >> something to do will it start >> running, which is why MQMS always fails relative to SQMS. > >Not really and you already know this you are just trying to >[force a square peg into a round hole with a sledgehammer] >find any possible excuse even false contrived excuses just >to criticize me. Everyone knows that on a single core >machine with four concurrent processes if any one of these >processes quits using the CPU the others get the rest. **** Clueless. Denser than depleted uranium. Why does a process have to "quit" to allow another process to have some CPU resource? [it is called 'time slicing', one of those Really Advanced Concepts] **** > >>>First, you are giving LINUX/UNIX a bad reputation with >>>this primitive >>>thinking. Second, SQL is not a packed record concept. >>>Sorry. You >>>want an ISAM or even varying record concept. But not SQL. >>>Wrong TOOL. >> *** >> His unholy fascination with doing seeks and somehow >> thinking this improves performance >> shows how little he understands about file systems. THos >> 12463 bytes could be fragmented > >Ah so then the more disk seeks the faster the throughput >right? NOT! **** Yes, but at no point have you come to any conclusion that there is ONE seek, or that the logic of the seeks is not handled within the file system cache, or anything else that could actually affect the performance. Nor do you have any data to show that seek time actually matters, or proposed any scalability strategy that could factor it out. The rest of us say "Now, if I/O time is a factor, we can remove it from the equation by..." and go on with a piece of scalable design. But since your design will be perfect, it will have no need for abstraction, refactorization, or scalability. **** > >> into potentially 24 sectors scatter ALL OVER the disk, >> with MASSIVE seek time between >> each, and the exact sector address is not derivable from >> first principles based on the >> file offset; he seems to work in a delusional system in >> which all files are physically >> contiguous on the disk (but remember, this is the guy who >> wanted an application to >> allocate contiguous physical memory!) So his >> understanding of file systems is every bit >> as bad as his understanding of virtual memory, or >> scheduling, or pretty much any other OS >> concept! > >I don't think that this is the way that this works, **** You are basing this one what? One half-read, badly-remembered student text? Years of doing file system maintenance? Your vast experience in writing file system drivers? Building actual file systems? Building previous programs that were disk I/O limited? **** >I could >be wrong, **** OMG! I didn't recognize the possibility of this! This changes the WHOLE PICTURE! **** >but, it would make much more sense to keep the >file allocation table (or whatever its called on whichever >platform) cached in RAM, **** Ah, the "wishful thinking" approach. Given that you are locking down massive number of pages, where do you think they come from? Guess what, they come from physical memory, where these file system pages would be cached. But they might NOT be cached because your OCR processes are hogging all the memory! **** >thus all this complex seeking could >be done in RAM, and the final actual hardware disk drive >seek could be singular. **** Or even zero! Of course, having worked out this Really Good Idea, you know the operating system will actually work in accordance with your fantasy... **** > I do know that file fragmentation >can drastically slow down file copies. According to my >possibly incorrect theory there is no need for it to slow >down seeks to a single record that does not span multiple >sectors. **** Ohh, I LOVE the optimism of the totally ignorant! A THEORY of how a file system work will necessarily dictate how a REAL file system works! ***** > >> If I have a giant BLOB, it also might be fragmented all >> over the disk. But hey, what's a >> little friendly fragmentation among friends? Disk seeks >> obviously take 0 ms. > >4 ms to 9 ms **** Based on what? Some vendor's spec sheet? Did you read the details? Are you measuring adjacent-cylinder seek time, or average (half-disk) seek time? Do you know how these work on A PARTICULAR INSTANCE OF A PARTICULAR FILE RUNNING AT A PARTICULAR INSTANT OF TIME IN A PARTICULAR OPERATING SYSTEM? Oh, I forgot, we are using PFOS, which is always perfectly in sync with your theories! **** > >> I have no idea how many experts it takes to convince him >> he's wrong. But at a guess, an >> infinite number is probably too small; we need >> second-order infinities to express the >> number of experts needed (map the experts to real numbers, >> Aleph-1 infinity of experts. >> And he probably still won't believe us!) > >Only one person providing complete and sound reasoning or at >least a link to reasonably concise complete and sound >reasoning. *** I keep telling you, go get a book on queueing theory. You need to see the whole mathematical development! **** > >> It sounds like a cool idea, and it can probably be used to >> create a seek address, which >> would work well if files were not fragmented and had zero >> latency to fetch the sectors. > >It is far better to seek to a specific record number than it >is to seek to an indexed value to find the record number to >seek to., twice as many disk seeks, thus half the potential >throughput. **** You are really totally clueless... You have NO IDEA how real file systems work! (Try following a Tony Mason lecture on file system drivers someday; he's possibly the best file system expert outside Microsoft today, and if you ever went to one of his lectures, you would quickly revise your harebrained theories) joe **** > > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm |