Prev: How do I produce phonic letter sounds with espeak?
Next: Cannot obtain IP using udhcpc for wifi connection..
From: Ignoramus28865 on 17 Nov 2009 12:32
Another update. This one tries to calculate the default parallel level
based on the number of CPUs on the system.

#!/usr/bin/perl

#
# This script reads its standard input, or from files given on
# command line (<>).
#
# It executes every line as a separate shell command. If --parallel
# argument is given, it executes as many jobs in parallel as
# possible, but no more tham "--parallel" at any given time.
#
# This can be helpful to speed up some tasks.
#
# The default level of parallel is based on the number of CPUs.
#
# You need module Parallel::ForkManager. It is available as
# an ubuntu package.
#
# Copyright(C) Igor Chudov, 2009. All rights reserved.
# This script is made available to the public under the latest
# GPL Version found at http://www.gnu.org/licenses/gpl.html
#
# No warranty is given or implied. Refunds will not be provided.
#
# Igor Chudov, http://igor.chudov.com/
#

use strict;
use warnings;

use Getopt::Long;
use Parallel::ForkManager;

my $parallel = undef;

GetOptions(
"parallel=i" => \$parallel,
);

unless( defined $parallel ) {
if( open( CPU, "/proc/cpuinfo" ) ) {
$parallel = 0;
while( <CPU> ) {
$parallel++ if /^processor\b/;
}
close( CPU );
} else {
$parallel = 0;
}
}

my $pm = new Parallel::ForkManager( $parallel );

while( <> ) {
chomp;
my $pid = $pm->start and next;
system( $_ );
$pm->finish;
}

$pm->wait_all_children;
From: despen on 17 Nov 2009 14:44
Ignoramus28865 <ignoramus28865(a)NOSPAM.28865.invalid> writes:

> On 2009-11-17, despen(a)verizon.net <despen(a)verizon.net> wrote:
>> Ignoramus30118 <ignoramus30118(a)NOSPAM.30118.invalid> writes:
>>
>>> #!/usr/bin/perl
>>>
>>> #
>>> # This script reads its standard input, or from files given on
>>> # command line (<>).
>>> #
>>> # It executes every line as a separate shell command. If --parallel
>>> # argument is given, it executes as many jobs in parallel, as
>>> # possible, but no more tham "--parallel" at any given time.
>>> #
>>> # This can be helpful to speed up some tasks.
>>
>> Make with the -j option.
>>
>> -j = as many as possible
>> -j nnn up to nnn at a time.
>>
>> In addition job interdedpence can be handled.
>>
>> Ie. don't start these 3 jobs until these other 2 run to completion
>> successfully.
>>
>
> This is similar to what my script does, except that it does not
> require a makefile.
>
> My immediate use of this would be to speed up image conversion with
> imagemagick. I have a script that would download pictures from my
> camera, and then "convert" them to make it 50%x50% of original size.
>
> I want to speed up "convert" by parallelizing it.

That's exactly the kind of thing I would use make for.
In fact, here's part of a makefile to create thumbnails for
jpgs and gifs:

small_%.jpg: %.jpg
convert -geometry 320x240 $< $@
chmod a+r $@
small_%.gif: %.gif
convert -geometry 320x240 -interlace plane $< $@
chmod a+r $@

Thumbnails only get created when the image is updated.
The same Makefile, uploads everything that's changed to the web server.
From: Ignoramus28865 on 17 Nov 2009 17:15
On 2009-11-17, despen(a)verizon.net <despen(a)verizon.net> wrote:
> Ignoramus28865 <ignoramus28865(a)NOSPAM.28865.invalid> writes:
>
>> On 2009-11-17, despen(a)verizon.net <despen(a)verizon.net> wrote:
>>> Ignoramus30118 <ignoramus30118(a)NOSPAM.30118.invalid> writes:
>>>
>>>> #!/usr/bin/perl
>>>>
>>>> #
>>>> # This script reads its standard input, or from files given on
>>>> # command line (<>).
>>>> #
>>>> # It executes every line as a separate shell command. If --parallel
>>>> # argument is given, it executes as many jobs in parallel, as
>>>> # possible, but no more tham "--parallel" at any given time.
>>>> #
>>>> # This can be helpful to speed up some tasks.
>>>
>>> Make with the -j option.
>>>
>>> -j = as many as possible
>>> -j nnn up to nnn at a time.
>>>
>>> In addition job interdedpence can be handled.
>>>
>>> Ie. don't start these 3 jobs until these other 2 run to completion
>>> successfully.
>>>
>>
>> This is similar to what my script does, except that it does not
>> require a makefile.
>>
>> My immediate use of this would be to speed up image conversion with
>> imagemagick. I have a script that would download pictures from my
>> camera, and then "convert" them to make it 50%x50% of original size.
>>
>> I want to speed up "convert" by parallelizing it.
>
> That's exactly the kind of thing I would use make for.
> In fact, here's part of a makefile to create thumbnails for
> jpgs and gifs:
>
> small_%.jpg: %.jpg
> convert -geometry 320x240 $< $@
> chmod a+r $@
> small_%.gif: %.gif
> convert -geometry 320x240 -interlace plane $< $@
> chmod a+r $@
>
> Thumbnails only get created when the image is updated.
> The same Makefile, uploads everything that's changed to the web server.

I think that you have a great approach, but in my case, I do not want
to keep the originals. (which may be addressed by adding rm in the
makefile, I am not sure)

Also, would this makefile make small_small_small_mypic.jpg if you
invoke it a few times, starting with mypic.jpg? Am I missing anything?

i
From: despen on 17 Nov 2009 21:15
Ignoramus28865 <ignoramus28865(a)NOSPAM.28865.invalid> writes:

> On 2009-11-17, despen(a)verizon.net <despen(a)verizon.net> wrote:
>> Ignoramus28865 <ignoramus28865(a)NOSPAM.28865.invalid> writes:
>>
>>> On 2009-11-17, despen(a)verizon.net <despen(a)verizon.net> wrote:
>>>> Ignoramus30118 <ignoramus30118(a)NOSPAM.30118.invalid> writes:
>> small_%.jpg: %.jpg
>> convert -geometry 320x240 $< $@
>> chmod a+r $@
>> small_%.gif: %.gif
>> convert -geometry 320x240 -interlace plane $< $@
>> chmod a+r $@
>>
>> Thumbnails only get created when the image is updated.
>> The same Makefile, uploads everything that's changed to the web server.
>
> I think that you have a great approach, but in my case, I do not want
> to keep the originals. (which may be addressed by adding rm in the
> makefile, I am not sure)
>
> Also, would this makefile make small_small_small_mypic.jpg if you
> invoke it a few times, starting with mypic.jpg? Am I missing anything?

No, you don't invoke it a few times.
Here is an example:

large:=$(wildcard *.jpg *.gif)

all: $(addprefix small_,$large)

small_%.jpg: %.jpg
convert -geometry 320x240 $< $@
chmod a+r $@
small_%.gif: %.gif
convert -geometry 320x240 -interlace plane $< $@
chmod a+r $@

What that does is first wildcard expand all the .jpg and .gif files
in the current directory into the variable "large".

Then it declares the default target "all" as all the .jpg/gif files
with a "small_" prefix.

So now make has all these targets, small_x.jpg, small_z.gif.
It looks for a rule that matches and finds the 2 rules shown.

If the file small_x.jpg already exists and was last updated AFTER
x.jpg was, it does nothing.

Otherwise it runs the rule shown.

How many it runs at one time depends on the -j option to make.

It can run no commands or hundreds, depending on how many files there
are and whether the "small_" file is current.


The real power in Makefiles comes from the chains of dependency.
You start with the images, make the thumbnails, then make uploading
dependent on the thumbnails.

Only whats needed gets done and you run as many processes at a time
as you want.
From: Ben Morrow on 17 Nov 2009 22:34
[f'ups set to clpmisc]

Quoth despen(a)verizon.net:
> Ignoramus28865 <ignoramus28865(a)NOSPAM.28865.invalid> writes:
>
> > Also, would this makefile make small_small_small_mypic.jpg if you
> > invoke it a few times, starting with mypic.jpg? Am I missing anything?
>
> No, you don't invoke it a few times.
> Here is an example:
>
> large:=$(wildcard *.jpg *.gif)

If any small_*.jpg files exist, this will include them.

> all: $(addprefix small_,$large)

This will then try to create a corresponding small_small_*.jpg file.
While it is possible to fix this, you quickly run up against the extreme
ugliness of trying to do anything complicated with make.

(For completeness I should also point out that this makefile is
gmake-specific. BSD make has facilities that would make this easier, but
it's still not going to be pretty.)

A straightforward Perl solution could look something like (untested):

use Parallel::ForkManager;

my $J = Parallel::ForkManager->new(3);

my @todo =
grep { -M $_ > -M "small_$_" }
grep !/^small_/,
<*.jpg *.gif>

for (@todo) {
$J->start and next;

# process the file in $_, using system "convert ..." or
# Image::Magick or ...

$J->finish;
}

$J->wait_all_children;

Ben

First  |  Prev  | 
Pages: 1 2
Prev: How do I produce phonic letter sounds with espeak?
Next: Cannot obtain IP using udhcpc for wifi connection..