Can extra processing threads help in this case?
in [MFC]
|
From: Joseph M. Newcomer on 12 Apr 2010 12:44 See below... On Sun, 11 Apr 2010 20:00:43 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message >news:enbDMkd2KHA.3568(a)TK2MSFTNGP04.phx.gbl... >> Peter Olcott wrote: >> >>> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>> message news:%23%23iEA1c2KHA.4332(a)TK2MSFTNGP02.phx.gbl... >>>> Peter Olcott wrote: >>>> >>>>>> So how do your HTTP request get delegated? Four >>>>>> separate IP address, sub domains? >>>>>> >>>>>> free.peter.com >>>>>> 1penny.peter.com >>>>>> nickle.peter.com >>>>>> peso.peter.com >>>>>> >>>>>> What happens when they cross domain attempts occur? >>>>> I would not be using the complex design that you are >>>>> referring to. One domain, one web server, four OCR >>>>> processes. >>>> >>>> So you back to a Many to One Fifo queue. And what >>>> happens with the HTTP responses? >>> >>> They have another FIFO queue in the opposite direction. >> >> >> So its an synchronized serialization? Or two threads? Do >> you need any reader/writer? > >The FIFO in the other direction will have four OCR processes >that are writers and one web server that is the reader. > >> >> >>>> By the web server needs to do a +1 and one of the OCR >>>> has to do a -1. >>> >>> No not on this memory location. This memory location is a >>> copy of another memory location that does these things to >>> the original value. I don't want to slow down the read of >>> this value by using a lock because it will be read in a >>> very tight loop. >> >> >> This is so a classic SYNC 101 problem - See Bains, maybe >> 1st or 2nd chapter. > >Updating the original value is a little tricky, I have books >on that. Updating the copy only requires that the copy be >made immediately following an update of the original. **** If you are adding or subtracting 1, it is not "a little tricky"; the phrase we like to use is "trivial", and it describes InterlockedIncrement and InterlockedDecrement. There is nothing "tricky" about using these. Otherwise, you would use a CRITICAL_SECTION, which is a spin lock at that level, and is not tricky at all. It is only slightly less than trivial. And unless you understand what a CRITICAL_SECTION is, please do not come back and tell us that its overhead is unacceptable. I don't really feel like explaining the obvious yet once more. **** > >> >>> In other words the original memory location may be read >>> or written to as often as once every 10 ms. It has to be >>> locked to make sure that it is updated correctly. >> >> >> But you said there is no lock in the paragraph above. >> >>> This copy of the memory location could easily be read a >>> million times a second or more, don't want to slow this >>> down with a lock. >> >> >> Oy vey! >> >>>> No conflicts, no reader/locker locks? No Interlock >>>> increments and decrements? >>> >>> As long as a simultaneous read and write can not garble >>> each other there is no need for any of these things, on >>> this copy of the original memory location. >> >> >> But do you have both a write and read potention that can >> occur at the same time or priority inversion due to task >> switching? > >No priority inversion is possible with this design because >every low priority job checks at least 100 times a second >whether it needs to put itself to sleep. ***** This entire sentence sounds stupid. If you actually implemented it this way, this would be an incredibly stupid design requiring polling; the correct way is to use a queue with a semaphore, so there is no need to poll, ever. This sounds like someone who has never heard of synchronization making pretensions to doing design. ***** > >There is no memory lock on the copy of the original >NumberOfHighPriorityJobsPending, since this is the only >thing that is shared across processes, there can be no >possible lock contention, thus no priority inversion. **** How? Exactly HOW do you share a variable across processes (I know how to do it in Windows, but do you? And I certainly don't know how to do it in linux, although I could probably discover it if I wanted to waste the time doing the research). But you have obviously done the research, or you would not assert this is possible, so please explain how you are mapping address spaces across process boundaries. Are you using shared data segments established at link time, or memory-mapped files? Can you write the Microsoft C code to do this in Windows? And if you could, why would you want to, when you should be using a named semaphore to handle this efficiently (I though performance mattered, and any design that requires polling a hundred times a second to see if there is something to do is really amateurish! This kind of design is what gets students in Operating Systems 101 an "F" in their exams. Oh, did I mention I've taught these courses?) And, of course, even if you had shared data segments, this would be a really stupid design. A global semaphore would make sense. But I'm not a linux programmer, so I don't know how linux semaphores work. But I know they exist. You really need to take this course: http://www.traininghott.com/Courses/Windows-System-Network-Programming-Hands-On-Training-Course-Class-Seminar-MFC.htm which I teach. It would be a start. It would at least help you dispel most of the myths you have about how operating systems in general and Windows in particular works. Each of the three labs in this course in involve concurrency: a multithreaded file printer, running a child process using anonymous pipes to capture stdout, and a multiprocess producer-consumer model that uses shared memory queues to pass information from the produce processes to the consumer processes. Learn about real synchronization primitives, what processes and threads are like, how a scheduler works, shared data segments, memory-mapped files, what thread priorities mean, how memory allocation works, and so on. I WROTE this course, I TEACH this course, I know something about these topics! And I don't make any more money if you take it because I get paid by the week, not by the number of students. So this is not a scheme to make more money off anyone. They'll just slot you into some already-scheduled class. joe **** > >> >> -- >> HLS > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 12 Apr 2010 12:55 See below... On Sun, 11 Apr 2010 23:18:07 -0600, Jerry Coffin <jerryvcoffin(a)yahoo.com> wrote: >In article <LuidnT3tuaC7p1_WnZ2dnUVZ_gCdnZ2d(a)giganews.com>, >NoSpam(a)OCR4Screen.com says... > >[ ... ] > >> Alternative (a) There are four processes with four queues >> one for each process. These processes only care about >> executing the jobs from their own queue. They don't care >> about the jobs in any other queue. The high priority process >> is given a relative process priority that equates to 80% of >> the CPU time of these four processes. The remaining three >> processes get about 7% each. This might degrade the >> performance of the high priority jobs more than the next >> alternative. > >There is no such thing with any OS of which I'm aware. At least with >a typical OS, the highest priority task is the *only* one that will >run at any given time. Windows (for one example) does attempt to >prevent starvation of lower priority threads by waking one lower >priority thread every four seconds. **** Sadly, you have to realize that schedulers work like he *imagines*, not like any real scheduler we have ever seen! So he thinks that schedulers worry about percentages of CPU time when no scheduler in history has EVER worked this way. But what does reality matter in these designs? **** > >Though the specific details differ, Linux works reasonably similarly. **** There is an article I cited in this thread on how Linux anti-starvation works, and it is a really sad design. "Kludge" comes to mind. **** > >Neither, however, provides any set of priorities that will give >anything similar to what you've described. It just doesn't exist. **** But he won't listen when we tell him these things. This is not how Peter's Fantasy Operating System would run, so it doesn't matter! **** > >> Alternative (b) each of the low priority jobs checks to see >> if a high priority job is in the queue or is notified by a >> signal that a high priority job is waiting. If a high >> priority job is waiting then each of these low priority jobs >> immediately sleeps for a fixed duration. As soon as they >> wake up these jobs check to see if they should go back to >> sleep or wake up. **** But why not use semaphores? Duh! **** > >This requires that each of those tasks is aware of its own process >scheduling AND of the scheduling of other processes of higher >priority. Worse, without a lot of care, it's subject to race >conditions -- e.g. if a high priority task shows up, for this scheme >to work, it has to stay in the queue long enough for every other task >to check the queue and realize that it needs to sleep, *before* you >start the high priority task -- otherwise, the task that's supposed >to have lower priority will never see that it's in the queue, and >will continue to run **** A lack of any understanding of synchronization and queueing theory is essential to this design succeeding. Unfortunately, real operating systems are not encumbered by this lack of understanding of how they work, so they will do SOMETHING, just not the something that PFOS would have done had it existed. **** > >Bottom line: you're ignoring virtually everything the world has >learned about process scheduling over the last 50 year or so. You're >trying to start over from the beginning on a task that happens to be >quite difficult. **** Actually, we were doing better than this in 1967. But then, we were a bunch of people who were in touch with reality (read the Multics literature of the era). Nobody was trying to build fantasy operating systems. > >> These processes could even simply poll a shared memory >> location that contains the number of high priority jobs >> currently in the queue. From what the hardware guys have >> told me memory writes and reads can not possibly garble each >> other. > >This has the same problem outlined above. It adds the requirement for >a shared memory location, and adding polling code to the OCR tasks. >See above about ignoring what the world has learned about process >scheduling over the last 5 decades or so. **** Why? PFOS can do this, so why shouldn't a real operating system? Never mind that this design would fail and Operating Systems 101 design course! And we knew how to do this better in 1968 (read Dijkstra's "Cooperating Sequential Processes" paper http://userweb.cs.utexas.edu/users/EWD/transcriptions/EWD01xx/EWD123.html) which is the basis for nearly all modern operating systems. In fact, the notation he uses is suspiciously like OpenMP or the PPL (Parallel Patterns Library, VS 2010). This was not an accident that these systems adopted his ideas. That paper, which we saw in preprint form at CMU, was about 1968, and was my introduction to concurrent programming. **** > >[ ... ] > >> Neither of these designs has any of the behavior that you >> mentioned. > >No -- they're substantially worse. At least all that did was >occasionally start a lower-priority task out of order. > >[ ... ] > >> I already figured out a way around that. Everyone must have >> their own user account that must be created by a live human. >> All users are always authenticated against this user >> account. I don't see any loopholes in this on single form of >> protection. > >Not even close, and you clearly don't understand the problem at all >yet. The problem is that to authenticate the user you've *already* >created a thread for his connection. The fact that you eventually >decide not to do the OCR for him doesn't change the fact that you've >already spawned a thread. If he makes a zillion attempts at >connecting, even if you eventually reject them all, he's still gotten >you to create a zillion threads to carry out the attempted >authentication for each, and then reject it. **** The end result being a successful Denial-of-service (D-O-S) attack on his site! **** > >Of course, that also ignores the fact that doing authentication well >is non-trivial itself. **** Not in PFOS! joe **** Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 12 Apr 2010 13:22 See below... On Mon, 12 Apr 2010 10:05:29 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Jerry Coffin" <jerryvcoffin(a)yahoo.com> wrote in message >news:MPG.262bf58bf8ba735989863(a)news.sunsite.dk... >> In article >> <LuidnT3tuaC7p1_WnZ2dnUVZ_gCdnZ2d(a)giganews.com>, >> NoSpam(a)OCR4Screen.com says... >> >> [ ... ] >> >>> Alternative (a) There are four processes with four queues >>> one for each process. These processes only care about >>> executing the jobs from their own queue. They don't care >>> about the jobs in any other queue. The high priority >>> process >>> is given a relative process priority that equates to 80% >>> of >>> the CPU time of these four processes. The remaining three >>> processes get about 7% each. This might degrade the >>> performance of the high priority jobs more than the next >>> alternative. >> >> There is no such thing with any OS of which I'm aware. At >> least with >> a typical OS, the highest priority task is the *only* one >> that will >> run at any given time. Windows (for one example) does >> attempt to >> prevent starvation of lower priority threads by waking one >> lower >> priority thread every four seconds. > >The alternative that you show quoted above is called time >slicing and has been available for many decades. **** And therefore timeslicing takes ZERO overhead, right? And you are still thinking that mucking with thread priorities is going to result in a flawless design (and if you start ranting about how "thread" and "processes" are different, you will only confirm that you are completely and utterly clueless) >> >> Though the specific details differ, Linux works reasonably >> similarly. >> >> Neither, however, provides any set of priorities that will >> give >> anything similar to what you've described. It just doesn't >> exist. > > Here is is: > http://en.wikipedia.org/wiki/Nice_(Unix) >I will have to run my own tests to see how the process >priority number map to the relative process priorities that >I provided above. Ultimately the schedule algorithm boils >down to essentially the frequency and duration of a time >slice. There is no need to map to the exact percentage >numbers that I provided. **** As I said, clueless. I knew about 'nice' in 1975, and also understood what the consequences are. You are reading a superficial article that does NOT really tell you what is happening to the thread. **** > > >>> Alternative (b) each of the low priority jobs checks to >>> see >>> if a high priority job is in the queue or is notified by >>> a >>> signal that a high priority job is waiting. If a high >>> priority job is waiting then each of these low priority >>> jobs >>> immediately sleeps for a fixed duration. As soon as they >>> wake up these jobs check to see if they should go back to >>> sleep or wake up. >> >> This requires that each of those tasks is aware of its own >> process >> scheduling AND of the scheduling of other processes of >> higher >> priority. Worse, without a lot of care, it's subject to >> race >> conditions -- e.g. if a high priority task shows up, for >> this scheme >> to work, it has to stay in the queue long enough for every >> other task >> to check the queue and realize that it needs to sleep, >> *before* you >> start the high priority task -- otherwise, the task that's >> supposed >> to have lower priority will never see that it's in the >> queue, and >> will continue to run. > >My scheduler will signal all of the low priority jobs that >they need to sleep now. When the high priority queue is >empty, and all of the high priority jobs are completed the >low priority jobs get a signal to wake up now. **** Oh, a whole NEW mechanism! Has anyone said "this is without a doubt the dumbest design I have ever seen"? Well, it is. You really don't understand operating systems, because the ideas you state here show a fundamental confusion about the right way to build applications. I presume you mean the linux 'signal' operation. Have you really studied the problems involved in using it? **** > >> Bottom line: you're ignoring virtually everything the >> world has >> learned about process scheduling over the last 50 year or >> so. You're >> trying to start over from the beginning on a task that >> happens to be >> quite difficult. > >I see no other way to provide absolute priority to the high >priority jobs (paying customers) over the low priority jobs >(free users). Also I see no way that this would not work >well. If I get enough high priority jobs that the lower >priority jobs never ever get a chance to run that would be >fantastic. The whole purpose of the free jobs is to get more >paying jobs. **** The notion that there is an 'absolute' priority is part of your problem. I think you don't really understand the basic mechanisms of the operating system. I would not waste time trying to build such a complex solution to a trivial problem; I would build the obvious, trivial solution and get the performance I needed, without introducing complex mechanisms that do not need to exist. **** > >If you see something specifically wrong with this approach >please point out the specific dysfunctional aspect. I see no >possible dysfunctional aspects with this design. **** The whole notion of having to "signal" the "low priority threads" to "sleep" and to "wake up" is ill-founded. You have a LOT of problems you are creating for yourself in how you would implement this. You seem to think that such signaling is trivial; it is not. There are fundamental race conditions which you have not comprehended, which will potentially be fatal to such designs (read about the "wakeup-waiting" flag that Multics had to implement to handle these race conditions...oh, wait a minute, that is a classic paper by Jerry Salzer, and therefore would not be part of what you read, which is only the first half of some student text on operating systems) **** > >> This has the same problem outlined above. It adds the >> requirement for >> a shared memory location, and adding polling code to the >> OCR tasks. >> See above about ignoring what the world has learned about >> process >> scheduling over the last 5 decades or so. > >I already changed this to signals, they don't waste as much >time. **** Really? Do you know how 'signal' actually works? And what the race conditions are? **** > >> No -- they're substantially worse. At least all that did >> was >> occasionally start a lower-priority task out of order. > >I see no possible sequence of events where this would ever >occur, if you do please point it out detail by detail. **** Look for what happens if a signal comes in between two instructions that are involved in the synchronization. Most of these techniques are subject to serious race-condition problems, which is why 'signal' is not even supported in Windows--too many problems, and people who use it create programs that won't work correctly. **** > >>> I already figured out a way around that. Everyone must >>> have >>> their own user account that must be created by a live >>> human. >>> All users are always authenticated against this user >>> account. I don't see any loopholes in this on single form >>> of >>> protection. >> >> Not even close, and you clearly don't understand the >> problem at all >> yet. The problem is that to authenticate the user you've >> *already* >> created a thread for his connection. The fact that you >> eventually >> decide not to do the OCR for him doesn't change the fact >> that you've >> already spawned a thread. If he makes a zillion attempts >> at >> connecting, even if you eventually reject them all, he's >> still gotten >> you to create a zillion threads to carry out the attempted >> authentication for each, and then reject it. > >Block IP long before that. **** WHat IP? You clearly have some bizarre notion that the IP is unforgeable. If I wanted to do a D-O-S on your site, I'd change the IP with every transmission. You have some awfully naive views about the goodness of the world. Or are we talking about Peter's Fantasy Internet, where everyone is well-behaved, there are not D-O-S attacks, and nobody EVER emails a virus? **** > >> >> Of course, that also ignores the fact that doing >> authentication well >> is non-trivial itself. >> >> -- >> Later, >> Jerry. > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Jerry Coffin on 12 Apr 2010 13:25 In article <370a869f-3f56-4521-ad9d-66784df27a12@ 5g2000yqj.googlegroups.com>, sant9442(a)gmail.com says... > > > > This is a scenario where something like Amazon's EC2 would work well > > -- you pay only for processor time you actually use, but if you get a > > big job, it can run your task on dozens or even thousands of > > processors so (for example) all the pages from a book are OCRed in > > parallel, the results put back together, and your customer gets his > > result back quickly. Then your system goes idle again, and you quit > > paying for *any* processor time until another task comes along. > > Interesting proposal. > > But this won't eliminate coding requirements? He only leverages the > computing power and scaling needs which is good as one less (big) > issue to deal with, but he still needs to frame his code around the > same principles. In fact, this current framework of Many Threads to 1 > FIFO queue might cost him more based on my readings. He will need to > make it dynamic loading for EC2 to minimize his cost.. Right? No, it wouldn't eliminate coding, but it would let him concentrate primarily on the OCR engine proper. I haven't written anything for it myself, but the general idea wouldn't be a lot different than he's working on right now -- you build a complete system on Linux, create a virtual machine image of it, and upload it to Amazon's servers. They charge you a little for storage and per-processor time for the actual processing. The difference from what he's doing right now is that instead of being restricted to running on one single-core processor, it'll automatically load up multiple instances of his virtual machine onto multiple processors and run them all concurrently when/if there's enough load to justify it. They provide a billing system that already works, load balancing, more processing time than you could hope to put to good use, geographically distributed top-tier data centers, etc. In short, they already take care of a lot of what he's worrying about, and quite a bit more that he doesn't know to worry about yet. Of course, Amazon isn't the only possibility -- Google App Engine and Windows Azure do the same sorts of things. The obvious difference is that EC2 let's you put your "stuff" in a VM image, so his current code should remain usable. Google App Engine requires that you code in Python or Java. IIRC, Windows Azure supports .NET, so he _might_ be able to use existing code, but it's not at all certain. -- Later, Jerry.
From: Hector Santos on 12 Apr 2010 13:39
Peter Olcott wrote: >>>> if (NumberOfHighPriorityJobsPending !=0) >>>> nanosleep(20); >>> 20 milliseconds >> >> Ok, fair enough, you honestly cited the wrong function, >> but we get > > No I didn't > http://linux.die.net/man/2/nanosleep > I did provide the wrong parameters, it should have been > nanosleep(0, 20000000) > 20000000 nanoseconds = 20 ms Hence, you used the wrong function for the data you provided - period. >> Who does the HTTP delegation? Or will you have FOUR >> different web servers with four different FORM pages? >> How do you control the cross domain security and business >> thread entry points where someone is trying to cheat you >> out of a few dollars? > > (1) Everyone authenticates How do does this take? > One physical CPU with a single core and two hyperthreads. Is that possible? >> Do you see the conflicts all based on your own >> ever-changing descriptions? > > No but I do see the conflicts based on your misreading what > I said. You say 10 different things each message! and you still have a conflict with each one! >> A single CORE does not have HYPERTHRREADING!!! That is >> only possible with a multi-core CPU. > > Not according to Intel terminology. Hyperthreading was what > Intel came out with before they came out with multiple > cores. Now that they came out with multiple cores, some of > these multi-core CPUs also have two hyperthreads per core, > and some do not. Show a reference where it says 1 CPU with one CORE offers Hyperthreading. Maybe the mix up is what CORE means. You can not have hyperthreading without atleast TWO LOGICAL processors. These can be called cores. http://en.wikipedia.org/wiki/Hyper-threading Hyper-threading is an Intel-proprietary technology used to improve parallelization of computations (doing multiple tasks at once) performed on PC microprocessors. For each processor core that is physically present, the operating system addresses two virtual processors, and shares the workload between them when possible. Hyper-threading requires not only that the operating system support multiple processors, You are confusing multi-core with two physically separate multi-cpu and you are confusing the marketing usage and changes with "HT" versus dual vs multi-core. In other words, when intel use HT - it means the processor has multiple cores. There is no such thing AFAIK as a 1 CPU with 1 CORE that offers HyperThreading. http://en.wikipedia.org/wiki/Multi-core_(computing) A multi-core processor is a processing system composed of two or more independent cores. and from the horse mouth: http://www.intel.com/technology/platform-technology/hyper-threading Today's Intel� Hyper-Threading Technology (Intel� HT Technology) delivers thread-level parallelism on each processor resulting in more efficient use of processor resources.... Note it says and its guilty also of using PROCESSOR in lieu of CORE, but nonetheless it says EACH PROCESSOR. Not ONE processor! >> So you will have a Multiple-CORE machine where each >> process affinity is set, all a SINGLE CPU machine where >> they is no hyperthreading? > No that is not what I said and yet another false assumption > on your part. See above. Do you know how to read anything even technical documents on the web? >>> This is not a given, but, using time to synchronize is >>> not the best idea. >> >> It is never a good idea. So why did you show it? > > It would have probably worked well enough, but, changing it > is better. It would of worked perfectly until it broke - Classic SYNC 101 error! You are no exception. > I always design completely before I begin coding, and you do it so poorly you never get anything done. You would be totally lost without the help of people on the net. >> Not related to your design solution, but a cause >> contributing towards your design problem. > > So then are there are specific dysfunctional aspects with > the simple time slicing that I proposed immediately above > that you can point out? First, you are not time slicing, you are yielding the rest of your quantum when you sleep. The OS time slicing YOU. You need to learn how to use the terms and ideas correctly. Do you mean signaling like I TOLD YOU to use (you did not propose it)? It depends on what you believe is best performance wise. Are you attempting to throttle a request? Slow it down? Put incoming low priority request to "sleep" until the 1 single fifo processor is complete with the current high priority request? If you want to throttle the new thread (slow it done) and give more quantum to the other high priority thread, you can yield (give up your quantum). If you want block it to give even MORE quantum to the HP thread, then you can wait on a kernel object which will be a VERY efficient "SLEEP" until it is signaled to wake up. IMO, with my experience, it is better to leave your threads alone and only boost the high priority thread and see how that performs. If you see a lower thread hogging up your high priority, then you probably to make it wait for HP completion or throttle it. But the whole point of all this is you don't really know until you see this all in action with a LOADING - how your request are coming in various rates and frequencies. There is no one equation here and 99.99% you need to make this configurable or at some intelligence to make it learn. The lesson here is not to lock yourself into one way that it puts constraints on all the other parts and as a result, those other parts puts constraints on these tunning factors. I can give you one recent specific experience. POP3 server. We tried to increase the priority on one thread for an important action it is terribly affected the rest of the server. By far, by letting the OS do its job with scheduling, we obtain the best performance across the board. But trying to give the HP thread full power, you might find the the rest is not satisfactory, even its a low tier request item. Remember, first impressions count - if they see a poor slow trial, they might even bother subscribing to your service. >> Well, until we get straight what you THINK a "single core" >> processor means, it depends. I think you mean each process >> is assigned a CPU affinity on a multi-core machine. > > PENTIUM 4, it might not even have hyperthreading. Ok, so you going with a pure single processor. Hey, that might be better, but you don't know until you tried a multi-core or multi-cpu with multiple threads. Remember what you will be saving, context switching. >> But as I pointed out, they are not as isolated as you >> think: >> >> - HTTP request delegator or proxy? > > The web server does this. > >> - common customer database for authentication and ACL >> ideas >> (access control list) or four separate databases? > > single customer database >> - One log for each one or one single log? > > A single log, and additionally a separate log for each paid > job stored in the customer database with the output data. So you realize these need syncing. The logging can use fast appending, but the the customer database will need syncing specially if using SQLITE3. For this, go MYSQL to avoid the full locks SQLITES provides. >> So each OCR can handle any job type. Fine. But you need a >> single source manager as indicated above which negates >> your design. > > Not at all. The web server delegaes the jobs to queues. That is what what I said and you will be load balancing othewise YOU will be hosed. >> Don't forget the HTTP response requirements - which is >> another bottle neck since you have to update/log state >> points. > > OCR process tells the web server when a specific job is > completed. Well requires the WEB SERVER threads to wait. >>> As long as the ROW lookup maps to the file byte offset we >>> are good. >> >> Then get another data manager because SQLITE will not give >> you this. > > And you know this how? Because we, many of customers and 3rd party developers use SQLITE3 in many our products. It uses ISAM/BTREE behind the scenes but that doesn't mean you have access to - you will be violating the integrity. Now, that said, if you have the cojones, you can use whats called SQLITE3 VFS 'Virtual File System' and VFM, Virtual File Methods which allows you to do custom designs into the HOOKS of the SQLITE3 api. http://www.sqlite.org/c3ref/io_methods.html http://www.sqlite.org/c3ref/vfs_find.html http://www.sqlite.org/c3ref/vfs.html But why would anyone do this for SQL is beyond me. The only legitimate reason you might want to use this and the SQLITE3 designers will advise you against it, is to begin changing its locking scheme. The fact you think you need SQL with file objects access means you clearly are applying it wrong. >>> If the ROW lookup must read and maintain an index just to >>> be able to get to the rows in sequential order, this may >>> not be acceptable. >> >> Huh, you SELECT gives you the order you want - you declare >> it in the SELECT. But its not a FILE OFFSET thing. > Eventually it has to always be a file byte offset thing in > the underlying physical implementation because Unix/Linux > only knows about files in terms of a sequences of bytes. If > I tell the file manager that I want record number 12463 and > each record has exactly 12 bytes then it must seek (12463 -1 > * 12) byte offset. First, you are giving LINUX/UNIX a bad reputation with this primitive thinking. Second, SQL is not a packed record concept. Sorry. You want an ISAM or even varying record concept. But not SQL. Wrong TOOL. Now, can you create a TABLE with a field that is a BLOB, where you save a fixed length binary field? SURE! you can do that. But you will be accessing sql FIELDS, not direct file byte offsets. >> Since your HTTP request are not bulk, each one is handled >> separated. At the rate you expect for the TPS, you will be >> MURDERED. > > It will merely be reduced to the maximum number of > transactions that can be written to disk. Most of this time > would be disk drive seek time. My cheap drive has about 9 ms > seek time that's 111 seeks per second. So SQLite might be a > bottleneck unless a single transaction takes a single seek. Gawd, you such a bad designer. >> Again, it is explained to you, yet you refused to listen. >> SQLITE is not what you want as a shared SQL database among >> multiple WEB and OCR accessors. > > The transaction write time may still be a bottleneck for any > other SQL processor. Not with SQL or database systems that allow for SQL row/record/cursor level I/O access. >> You don't know what you are talking about. By definition, >> Record numbers are not out of sequence. And what are the >> indices you are > > Record numbers can and indeed are capable of being placed > out-of-sequence in SQLite. Not in view YOU get them. You can get it ascending or descending. By definition, ROW NUMBERS are sequential! Now the insider, it can have fragmentation, but the PURPOSE of SQL if to give you the records in the order you want. Not in how its STORED! In other words, if you did: SELECT * FROM PETER_TABLE you are not going to see row ids 10, 100, 20, 5, 2, 6, 7. You will see: 2 5 6 7 10 20 100 Its not like you think where it uses the first EMPTY ISAM RECORD that was last deleted and before you packed the database. Sorry Charlie. It is not the NON-SQL Databasing you did for 10 years, 10 years ago. >> referring to? What (fields) are they based on? > > The record number. So you make the record number the primary key? you could, by why is the reason again? >> You see, "Single Core?" you are using the wrong terms. >> Yet, joe and I have a good idea of what you trying to say. > > PENTIUM 4 > I have been using the right terms all along and you keep > insisting on ignoring them. See above. -- HLS |