Prev: Is ElementTree development still active?
Next: How to input values of the matrix from keyboard in python
From: EW on 11 Aug 2010 13:57
On Aug 11, 1:55 pm, MRAB <pyt...(a)mrabarnett.plus.com> wrote:
> EW wrote:
>
> [snip]
>
>
>
> > So here the P2 thread has ended and gone away but I still have his
> > Queue lingering.
>
> > So on a thread I can use is_alive() to check status and use join() to
> > clean up but I don't see any analogous functionality for Queues.  How
> > do I kill them?  I thought about putting a suicide message on the
> > Queue and then C1 would read it and set the variable to None but i'm
> > not sure setting the variable to None actually makes the Queue go
> > away.  It could just end up sitting in memory unreferenced - and
> > that's not good.  Additionally, I could have any number of consumer
> > threads reading that Queue so once the first one get the suicide note
> > the other consumer threads never would.
>
> > I figure there has to be an elegant way for managing my Queues but so
> > far I can't find it.  Any suggestions would be appreciated and thanks
> > in advance for any help.
>
> An object will be available for garbage collection when nothing refers
> to it either directly or indirectly. If it's unreferenced then it will
> go away.
>
> As for the suicide note, if a consumer sees it then it can put it back
> into the queue so other consumers will see it and then forget about the
> queue (set the variable which refers to the queue to None, or, if the
> references are in a list, delete it from the list).

Ok great. I wasn't sure about the Garbage collection part of it.
That's actually pretty easy.

Thanks!
From: Paul Rubin on 11 Aug 2010 14:16
EW <ericwoodworth(a)gmail.com> writes:
> I thought about doing it that way and I could do it that way but it
> still seems like there should be a way to clean up Queues on my own.
> If I did it this way then I guess I'd be relying on garbage collection
> when the script ended to clean up the Queues for me.

Oh, I see. As long as it's possible to start new producer or consumer
threads that touch a queue, obviously that queue has to still be around.
If the program starts all its threads at the beginning, then runs til
they exit, then does more stuff, then you could do something like:

# make dictonary of queues, one queue per task type
queues = {'sql': Queue(), 'logging': Queue(), ... }

for i in <whatever threads you want>
threading.Thread(target=your_handler, args=[queues])

del queues

and then when all the threads exit, there are no remaining references to
the queues. But why do you care? Queues aren't gigantic structures,
they're just a list (collections.deque) with an rlock. It's fine to let
the gc clean them up; that's the whole point of having a gc in the first
place.
From: EW on 11 Aug 2010 14:39
On Aug 11, 2:16 pm, Paul Rubin <no.em...(a)nospam.invalid> wrote:
> EW <ericwoodwo...(a)gmail.com> writes:
> > I thought about doing it that way and I could do it that way but it
> > still seems like there should be a way to clean up Queues on my own.
> > If I did it this way then I guess I'd be relying on garbage collection
> > when the script ended to clean up the Queues for me.
>
> Oh, I see.  As long as it's possible to start new producer or consumer
> threads that touch a queue, obviously that queue has to still be around.
> If the program starts all its threads at the beginning, then runs til
> they exit, then does more stuff, then you could do something like:
>
>     # make dictonary of queues, one queue per task type
>     queues = {'sql': Queue(), 'logging': Queue(), ... }
>
>     for i in <whatever threads you want>
>        threading.Thread(target=your_handler, args=[queues])
>
>     del queues
>
> and then when all the threads exit, there are no remaining references to
> the queues.  But why do you care?  Queues aren't gigantic structures,
> they're just a list (collections.deque) with an rlock.  It's fine to let
> the gc clean them up; that's the whole point of having a gc in the first
> place.

Well I cared because I thought garbage collection would only happen
when the script ended - the entire script. Since I plan on running
this as a service it'll run for months at a time without ending. So I
thought I was going to have heaps of Queues hanging out in memory,
unreferenced and unloved. It seemed like bad practice so I wanted to
get out ahead of it.

But the GC doesn't work the way I thought it worked so there's really
no problem I guess. I was just confused on garbage collection it seems.
From: Paul Rubin on 11 Aug 2010 14:52
EW <ericwoodworth(a)gmail.com> writes:
> Well I cared because I thought garbage collection would only happen
> when the script ended - the entire script. Since I plan on running
> this as a service it'll run for months at a time without ending. So I
> thought I was going to have heaps of Queues hanging out in memory,
> unreferenced and unloved. It seemed like bad practice so I wanted to
> get out ahead of it.

Even if GC worked that way it wouldn't matter, if you use just one queue
per type of task. That number should be a small constant so the memory
consumption is small.
From: MRAB on 11 Aug 2010 15:33
Paul Rubin wrote:
> EW <ericwoodworth(a)gmail.com> writes:
>> Well I cared because I thought garbage collection would only happen
>> when the script ended - the entire script. Since I plan on running
>> this as a service it'll run for months at a time without ending. So I
>> thought I was going to have heaps of Queues hanging out in memory,
>> unreferenced and unloved. It seemed like bad practice so I wanted to
>> get out ahead of it.
>
> Even if GC worked that way it wouldn't matter, if you use just one queue
> per type of task. That number should be a small constant so the memory
> consumption is small.

That's basically how _non_-garbage-collected languages work! :-)
First  |  Prev  |  Next  |  Last
Pages: 1 2 3
Prev: Is ElementTree development still active?
Next: How to input values of the matrix from keyboard in python