Maximum (practical) number of threads?
in [TCL]
|
Prev: Help flag for widgets
Next: expr oddity?
From: Bob Techentin on 12 May 2010 02:39 > Doesn't Tcl use POSIX threads? Yes, but they're buried under the Tcl threading model, which is a little more robust than your typical C application. > If this is what > you need, and want to use Tcl, figure out how to reduce the memory > footprint for each interp. So I run tclsh on linux, and 'ps' tells me that the process uses 1504 KBytes (RSS). I create 100 interps, and 'ps' tells me that my process size has increased to 8604 KBytes. That's only 71 KBytes per interp, which is not too bad. (No warranty expressed in my interpretation of 'ps') I guess I was wondering if there were other limitations. For example, I'm not afraid of throwing a million elements into an array. Is there a slave interpreter data structure that would blow up at 10,000, or should I fear some latent synchronization limitation that won't work with so many threads? Bob
From: Uwe Klein on 12 May 2010 03:52 Bob Techentin wrote: > I guess I was wondering if there were other limitations. For example, > I'm not afraid of throwing a million elements into an array. Is there > a slave interpreter data structure that would blow up at 10,000, or > should I fear some latent synchronization limitation that won't work > with so many threads? > What about galloping horde wake up from poll ;-? Afaik/r such issues ( with forex apache ) lead to epoll and similar constructs. ( Actually never worked with threads except under RT-Linux ) uwe
From: Donal K. Fellows on 12 May 2010 10:27 On 11 May, 15:37, Bob Techentin <techentin.rob...(a)mayo.edu> wrote: > I found some info in the wiki on thread-specific data structures, some > benchmarks, and guidance. But I can't seem to find any information > about inherent limitations on the number of threads that you can have. That's because nobody's really tested with that size of hardware behind it. Be aware that the inter-thread communications are likely to be not uniformly expensive; on-board comms will be faster than things that have to go across to the other side of the room. :-) > Can I make an application with a million threads? I think you'll hit a limit on the number of file descriptors before that. Tcl needs one FD per thread in order to communicate with the notifier thread, and that uses select() which doesn't scale too well. Rewriting the notifier to use a more efficient low-level API is tricky enough that I've always had something else to be doing... Donal.
From: tom.rmadilo on 12 May 2010 11:39 On May 12, 7:27 am, "Donal K. Fellows" <donal.k.fell...(a)manchester.ac.uk> wrote: > On 11 May, 15:37, Bob Techentin <techentin.rob...(a)mayo.edu> wrote: > > > I found some info in the wiki on thread-specific data structures, some > > benchmarks, and guidance. But I can't seem to find any information > > about inherent limitations on the number of threads that you can have. > > That's because nobody's really tested with that size of hardware > behind it. Be aware that the inter-thread communications are likely to > be not uniformly expensive; on-board comms will be faster than things > that have to go across to the other side of the room. :-) > > > Can I make an application with a million threads? > > I think you'll hit a limit on the number of file descriptors before > that. Tcl needs one FD per thread in order to communicate with the > notifier thread, and that uses select() which doesn't scale too well. > Rewriting the notifier to use a more efficient low-level API is tricky > enough that I've always had something else to be doing... If you check out the link provided in the OP it seems that our typical assumptions don't apply. I doubt that Tcl could be used without some major rewriting, if it is even possible. But the XMT system is divided up so as to avoid the typical I/O issues. Each processor has 128 data channels (hardware threads). These run a simplified UNIX based upon BSD. Most data used within the processor is stored in a large local cache, but data can be moved from/to any processor. There are four processors per module and each module has four I/O processors which handle communication, File I/O and IPC.
From: Alexandre Ferrieux on 12 May 2010 12:39
On May 12, 4:27 pm, "Donal K. Fellows" <donal.k.fell...(a)manchester.ac.uk> wrote: > > Tcl needs one FD per thread in order to communicate with the > notifier thread [...] Wrong. There's a single triggerPipe shared among all threads. -Alex |