Prev: Pool Module: iterator does not yield consistently with different chunksizes
Next: delegation pattern via descriptor
From: Dave Angel on 2 Jul 2010 07:55
syockit wrote:
> I've been playing around with custom iterators to map into Pool. When
> I run the code below:
>
> def arif(arr):
> return arr
>
> def permutate(n):
> k = 0
> a = list(range(6))
> while k<n:
> for i in range(6):
> a.insert(0, a.pop(5)+6)
> #yield a[:] <-- produces correct results
> yield a
> k += 1
> return
>
> def main():
> from multiprocessing import Pool
> pool = Pool()
> chksize = 15
> for x in pool.imap_unordered(arif, permutate(100), chksize):
> print(x)
>
> if __name__=="__main__":
> main()
>
> .... will output something like this:
>
>
> [36, 37, 38, 39, 40, 41]
> [36, 37, 38, 39, 40, 41]
> [36, 37, 38, 39, 40, 41]
> [36, 37, 38, 39, 40, 41]
> [36, 37, 38, 39, 40, 41]
> [36, 37, 38, 39, 40, 41]
> [72, 73, 74, 75, 76, 77]
> [72, 73, 74, 75, 76, 77]
> [72, 73, 74, 75, 76, 77]
> [72, 73, 74, 75, 76, 77]
> [72, 73, 74, 75, 76, 77]
> [72, 73, 74, 75, 76, 77]
> [108, 109, 110, 111, 112, 113]
> [108, 109, 110, 111, 112, 113]
> [108, 109, 110, 111, 112, 113]
> [108, 109, 110, 111, 112, 113]
> [108, 109, 110, 111, 112, 113]
> [108, 109, 110, 111, 112, 113]
> [144, 145, 146, 147, 148, 149]
>
> ... where results are duplicated number of times equal to chunk size,
> and the results between the gap are lost. Using a[:] instead, i get:
>
> [6, 7, 8, 9, 10, 11]
> [12, 13, 14, 15, 16, 17]
> [18, 19, 20, 21, 22, 23]
> [24, 25, 26, 27, 28, 29]
> [30, 31, 32, 33, 34, 35]
> [36, 37, 38, 39, 40, 41]
> [42, 43, 44, 45, 46, 47]
> [48, 49, 50, 51, 52, 53]
>
> .... it comes out okay. Any explanation for such behavior?
>
> Ahmad Syukri
>
>
While I didn't actually try to follow all your code, I suspect your
problem is that when you yield the same object multiple times, they're
being all saved, and then when evaluated, they all have the final
value. If the values are really independent, somebody has to copy the
list, and the [:] does that.

DaveA

 | 
Pages: 1
Prev: Pool Module: iterator does not yield consistently with different chunksizes
Next: delegation pattern via descriptor