adjustable arrays
in [Lisp]
|
Prev: Comparing Lisp to Python, what you consider more important: speed or macros.
Next: LispBox - truncated result?
From: Peter Keller on 29 Apr 2010 13:14 Mirko <mirko.vukovic(a)gmail.com> wrote: > Could it be that a different data structure would be more > appropriate? What about a tree? (also, until recently, I had not > realized how many different kinds of tree there are (2-3, 2-3-4, left > leaning, etc). That would impact your access time, but in your case it > may be manageable, or you could first store in tree, and then transfer > into array. I know a fair number of data structures, but most of them require consing which is something I need to minimize. Luckily, the storage requirements of my data aren't tree-like. Thank you for your suggesstion. To give an example of the problems I'm facing, suppose I have 10,000 clients: If I have a 16k initial read buffer for each client which can grow in size up to 128k as needed, then just for the read buffers alone I'll need: 156MB up to 1.250GB of memory needed. If I have about 20k data buffers I need for each client to write, then the buffers required for writing are 195MB. If I have about 20K in memory structures for each client, then another 195MB. So, under worst case conditions (the clients send me 128KB results all the time instead of averaging around 16K) for 10,000 clients, I need in total 546MB up to 1.640GB. Now this needed memory for which I can easily account! That is a lot of memory to keep resident and perform churn on and hence why I'm very paranoid about consing or calling make-array. I use other libraries and things and who knows what their memory usages are. But at the scaling levels I'm desiring, I have to pay attention to it. -pete
From: Peter Keller on 29 Apr 2010 13:16 Giovanni Gigante <giov(a)cidoc.iuav.it> wrote: > >> My question is: This adjustment happens at the _end_ of the array, can I >> expand the array at the beginning? > > > Naive idea: > what about building a list (you can cons to the beginning of that) and > mapping it to an array only when you are done? That doesn't solve the problem, suppose: (#(1 2 3) (#4 5 6)) I still have to allocate a 6 element array somewhere to map all of that into a single array: #(1 2 3 4 5 6) -pete
From: Peter Keller on 29 Apr 2010 13:27 RG <rNOSPAMon(a)flownet.com> wrote: > In article <4bd9afe9$0$9898$80265adb(a)spool.cs.wisc.edu>, > Peter Keller <psilord(a)merlin.cs.wisc.edu> wrote: >> The context is that when writing a data array to a client, the buffer >> management layer needs to wrap a header around the data before sending it. >> I thought about not doing the wrapping at all--just write the small header >> then the payload, but then that leads to alternating writes of a very >> small amount of data and then a large amount of data. That oscillation >> sucks from a network scheduling/bandwidth perspective. I need to write >> as many bytes as I can as often as I can. > > Seems to me then that what you want is not a growable array but rather > smarter write buffering. I do not deny that, have any references? The nice thing about lisp is I can just go back into that layer and bash the code into something else with relative ease. >> I guess at this point, I'll just assemble the arrays naively and if the >> profiler tells me there is a problem there, I'll look into it deeper. > > That sounds like a fine plan. Premature optimization and all that. > Memcpy is pretty frickin' fast, particularly when compared to network > I/O. Speaking of memcpy, in lisp what is the equivalent of memcpy? replace? Hrm... I should use that in a few places I stupidly wrote the byte copies with a do loop. Thanks for the oblique suggestsion. :) -pete
From: RG on 29 Apr 2010 14:02 In article <4bd9c172$0$9895$80265adb(a)spool.cs.wisc.edu>, Peter Keller <psilord(a)merlin.cs.wisc.edu> wrote: > RG <rNOSPAMon(a)flownet.com> wrote: > > In article <4bd9afe9$0$9898$80265adb(a)spool.cs.wisc.edu>, > > Peter Keller <psilord(a)merlin.cs.wisc.edu> wrote: > >> The context is that when writing a data array to a client, the buffer > >> management layer needs to wrap a header around the data before sending it. > >> I thought about not doing the wrapping at all--just write the small header > >> then the payload, but then that leads to alternating writes of a very > >> small amount of data and then a large amount of data. That oscillation > >> sucks from a network scheduling/bandwidth perspective. I need to write > >> as many bytes as I can as often as I can. > > > > Seems to me then that what you want is not a growable array but rather > > smarter write buffering. > > I do not deny that, have any references? Sorry, no. But surely it's not that hard? rg
From: Peter Keller on 29 Apr 2010 14:21
RG <rNOSPAMon(a)flownet.com> wrote: > In article <4bd9c172$0$9895$80265adb(a)spool.cs.wisc.edu>, > Peter Keller <psilord(a)merlin.cs.wisc.edu> wrote: > >> RG <rNOSPAMon(a)flownet.com> wrote: >> > In article <4bd9afe9$0$9898$80265adb(a)spool.cs.wisc.edu>, >> > Peter Keller <psilord(a)merlin.cs.wisc.edu> wrote: >> >> The context is that when writing a data array to a client, the buffer >> >> management layer needs to wrap a header around the data before sending it. >> >> I thought about not doing the wrapping at all--just write the small header >> >> then the payload, but then that leads to alternating writes of a very >> >> small amount of data and then a large amount of data. That oscillation >> >> sucks from a network scheduling/bandwidth perspective. I need to write >> >> as many bytes as I can as often as I can. >> > >> > Seems to me then that what you want is not a growable array but rather >> > smarter write buffering. >> >> I do not deny that, have any references? > > Sorry, no. But surely it's not that hard? Yeah, it shouldn't be that hard. When it comes to it, I'll just look in the usual place for papers or descriptions of what people already do. Thanks for your help. Later, -pete |