socket limitations question
in [TCL]
|
Prev: Announce: Runtime Library 3.0 Gestalt Items 1.1 and Hugelist 1.4,Tcl-Mode for XCode
Next: http::geturl intermittent hang - looks for ideas...
From: Alexandre Ferrieux on 14 Oct 2009 17:15 On Oct 14, 9:34 pm, "tom.rmadilo" <tom.rmad...(a)gmail.com> wrote: > On Oct 14, 11:01 am, Alexandre Ferrieux <alexandre.ferri...(a)gmail.com> > wrote: > > > On 14 oct, 18:20, "tom.rmadilo" <tom.rmad...(a)gmail.com> wrote: > > > When poll isn't available, an emulation using select is provided. Of > > > course, it isn't epoll. > > > Then why mention this ? Poll is just like select wrt scaling up: it > > doesn't; only epoll does. > > My point is that you must first harmonize epoll with the other two > possibilities. epoll must first work with the rest of the io code. The > example I gave shows that a successful abstraction of poll only > involved one tiny ifdef. But if epoll requires extensive modification > to other levels of the io API, it probably can't be used effectively. > Since this code was stolen from the GNU C Library, maybe an epoll > emulation also exists. No. Guessing is not necessary nor useful here. No interaction with the "rest of the io code". Just read the manpages: you'll find that epoll_wait() as the distinguished feature or returning *only* the hot fds in a compact list, while select/poll force you to scan the whole list looking for the hot ones. Hence emulation at the API level is useless, it will not solve the fundamental performance issue. That's the kind of situation that calls for a new syscall. In Linuxland that's also the minimum. -Alex
From: Tomas Friml on 10 Nov 2009 16:57 On Oct 15, 2:48 am, Alexandre Ferrieux <alexandre.ferri...(a)gmail.com> wrote: > On Oct 14, 3:30 pm, "Donal K. Fellows" > > <donal.k.fell...(a)manchester.ac.uk> wrote: > > On 14 Oct, 01:41, Alexandre Ferrieux <alexandre.ferri...(a)gmail.com> > > wrote: > > > > However, as a recent discussion on tclcore showed, there is another > > > one that kicks in, and that is not configurable at runtime: > > > FD_SETSIZE, which is the static size of the bit arrays passed to select. > > > IIRC it is 1024 for glibc. > > > On at least some OSes, you can set it at build-time. > > Yes. Recompile your libc, reboot, and pray. > > -Alex Hi, I recently hit this same problem described here, after sock1024 is used the socket command basically stops working. I was trying to find out what need to do get it working for more than 1024 sockets but I don't really understand the exact relation between tcl and libc (I'm tcl newbie). How do I find out which libc my tcl uses? Cheers, Tom
From: Alexandre Ferrieux on 10 Nov 2009 17:07 On Nov 10, 10:57 pm, Tomas Friml <instantni....(a)gmail.com> wrote: > On Oct 15, 2:48 am, Alexandre Ferrieux <alexandre.ferri...(a)gmail.com> > wrote: > > > > > > > On Oct 14, 3:30 pm, "Donal K. Fellows" > > > <donal.k.fell...(a)manchester.ac.uk> wrote: > > > On 14 Oct, 01:41, Alexandre Ferrieux <alexandre.ferri...(a)gmail.com> > > > wrote: > > > > > However, as a recent discussion on tclcore showed, there is another > > > > one that kicks in, and that is not configurable at runtime: > > > > FD_SETSIZE, which is the static size of the bit arrays passed to select. > > > > IIRC it is 1024 for glibc. > > > > On at least some OSes, you can set it at build-time. > > > Yes. Recompile your libc, reboot, and pray. > > > -Alex > > Hi, I recently hit this same problem described here, after sock1024 is > used the socket command basically stops working. I was trying to find > out what need to do get it working for more than 1024 sockets but I > don't really understand the exact relation between tcl and libc (I'm > tcl newbie). How do I find out which libc my tcl uses? Well, it uses _the_ libc present on the system (unless you're intentionally juggling with several versions, you have just one of them). Then look at the value of the FD_SETSIZE in the associated include files. -Alex
From: phil on 10 Nov 2009 17:38 poll() can support > 1024 fds and is quite portable -- well before 1995. poll() is also a lot more efficient than select(). One can preallocate the "pollfd array" ONCE and reuse it over and over. Unlike select() where you generally make FD_Sets over from scratch or memcpy it back... And its possible to make maps that map the "pollfd array" directly to your "internal fd system structure" with the same index. So switching to poll() would definately be a win/win in a couple areas. ps: epoll is "recent"-linux thing and should be used when available. I'm on RHEL3 and RHEL4 systems that don't have epoll.
From: Alexandre Ferrieux on 10 Nov 2009 19:15
On Nov 10, 11:38 pm, phil <pedi...(a)west.com> wrote: > > poll() is also a lot more efficient than select(). > One can preallocate the "pollfd array" ONCE and reuse it over and > over. > Unlike select() where you generally make FD_Sets over from scratch or > memcpy it back... Yes but you still have to iterate over the whole array to find the hot ones, so it's O(N). Only epoll() is O(1) in that it picks them out for you (and doesn't iterate under the hood either; they are "pushed" by the kernel instead of being "pulled" by the caller.) -Alex |