Parallel Compression 1.0 .....
in [General Programming]
|
Prev: off topic but please forgive me me and answer
Next: AUTHENTIC DESIGNER HANDBAGS WWW.VOGUELANDE.COM. CHANEL, LOUIS VUITTON, GUCCI
From: aminer on 2 Apr 2010 03:16 Skybuck wrote: > What if people wanna roll there own versions ? ;) > They would much better be "served" by algorithms/pseudo > code than real code which could be system/language specific ;) It's easy to EXTRACT algorithms from Object Pascal code... Look for example inside pbzip.pas, i am using this in the main body of my program: name:='msvcr100.dll'; It's the 'test' file that i am using - it's inside the zip file also - one you compile and execute pbzip.pas it will generate a file msvcr100.dll.bz. And as you have noticed i am using a - portable - compound filesystem, look at ParallelStructuredStorage.pas inside the zip file. After that i am opening it with: fstream1:=TFileStream.create(name, fmOpenReadWrite); and i am reading chunks of streams and 'distributing' them to my Thread Pool Engine to be compressed - in parallel - by myobj.BZipcompress method, look at: for i:=0 to e do begin if (i=e) and (r=0) then break; stream1:=TMemoryStream.create; if (r > 0) and (i=e) then stream1.copyfrom(fstream1,r) else stream1.copyfrom(fstream1,d); stream1.position:=0; obj:=TJob.create; obj.stream:=stream1; obj.directory:=directory; obj.compressionlevel:=9; obj.streamindex:=inttostr(i); obj.r:=r; obj.number:=e; TP.execute(myobj.BZipcompress,pointer(obj)); end; I am doing the same thing in PZlib.pas... http://pages.videotron.com/aminer/ Sincerely, Amine Moulay Ramdane.
From: aminer on 2 Apr 2010 03:30 Hello again, And after that i am reading those compressed files from the compound filesystem - look inside pzlib.pas - and i am 'distributing' those compressed files, as streams, to my Thread Pool Engine to be decompressed - look inside pzlib.pas - by myobj.Zlibdecompress method, look at: -------------------------------------------- names:=TStringlIST.create; storage.foldernames('/',names); len:=strtoint(names[0]); if r=0 then len:=len+ 1 ; for i:=0 to len do begin if (i=len) and (r=0) then break; obj:=TJob.create; obj.directory:=directory; obj.streamindex:=inttostr(i); obj.index:=i; obj.number:=e; obj.r:=r; TP.execute(myobj.Zlibdecompress,pointer(obj)); end; -------------------------------------------------- http://pages.videotron.com/aminer/ Sincerely, Amine Moulay Ramdane.
From: aminer on 2 Apr 2010 04:42 I wrote: > And as you have noticed i am using a portable > compound filesystem, look at ParallelStructuredStorage.pas > inside the zip file. Why ? Cause you can parallel compress your files and store those compound filesystem .zlb (zlib) or .bz (bzip) compressed files in a portable compound filesystem and after that you can distribute your compound filesystem... And of course you can uncompress files - or all the content of your compound file system - from your compound file system. And of course that's easy with Parallel Compression 1.0 :) http://pages.videotron.com/aminer/ Sincerely, Amine Moulay Ramdane.
From: aminer on 2 Apr 2010 11:10 Skybuvk wrote: >[...] an algorithm really ;) >What's so special about it ? Parallel bzip and zlib is not just pbzip.pas and pzlib.pas the parallel bzip and zlib algorithm include my Thread Pool Engine algorithm + Parallel Queue algorithm ... I am calling it algorithm cause it uses a finite number of instructions and rules to resolve a problem - parallel compression and decompression - Do you understand ? And as i said you can parallel compress your files and store those compound filesystem .zlb (zlib) or .bz (bzip) compressed files in a portable compound filesystem and after that you can distribute your compound filesystem... And of course you can uncompress files - or all the content of your compound file system - from your compound file system. > I see a whole bunch of pascal/delphi files thrown together, >a whole bunch of dll's and god-forbid ms*.dll files... Those dlls are mandatory for now... and you can easily write a batch file etc. and reorganize ... > I see some "test programs" which are described as "modules" which they > simply are not... That's VERY easy to convert those pzlib.pas and pbzip.pas to units, and that's what i will do in the next step... Parallel Compression will still be enhanced in the future... > It shouldn't be that hard... set your editor to "use tab character" (turn > tabs to spaces off) I am not using the delphi editor, just the notpad.exe or write.exe... and i am compiling from the dos prompt... >So far it seems like you are inserting your >threads/syncronizations >everywhere in single-thread-design algorithms ? No, it's not just insertting threads/syncronizations .. I have reasoned - and used logic - look for example at parallelhashlist.pas inside the zip file, i am using MEWs etc. carefully in the right places and i have also a little bit modified the serial code... and it uses a hash based method , with an array of MREW... The Thread Pool Engine Engine i have constructued it from zero - and i have used my ParallelQueue - an efficent lock-free queue - etc.... The parallel bzip and zlib, i have constructed it by using also my Thread Pool Engine construction etc... etc. That's not just 'inserting' threads/syncronizations. >But my estimate would be that for now on low core systems... the >"compression" would take far more time... No. pbzlib.pas takes for example 3.3x on 4 cores... http://pages.videotron.com/aminer/ParallelCompression/parallelbzip.htm Skybuck also wrote > [...] or anything extraordinary... Don't be stupid Skybuck. It's in fact: 1- Usefull 2 - A good thing for educational purpose. Amine Moulay Ramdane.
From: aminer on 2 Apr 2010 13:59
Skybuck wrote: >The thread pool concept is retarded. >Any good delphi programmer is capable of creating an array of threads. >So my advice to you: >1. Delete your thread pool, because it's junk. >2. Write a serious/big application that uses many threads, >and simply derive from TThread to see how easy it is. How can you be so stupid ? My Thread Pool Engine is not just an array of threads, it uses effient lock-free queues - example lock-free ParalleQueue - for each worker thread and it uses work-stealing - for more efficiency - etc ... And it easy the work for you - you can 'reuse' the TThreadPool Class... - and it is very useful... Please read again: http://pages.videotron.com/aminer/threadpool.htm Amine Moulay Ramdane. |