oom killer: break from infinite loop
in [Kernel]
|
From: KAMEZAWA Hiroyuki on 30 Mar 2010 21:10 On Tue, 30 Mar 2010 13:29:29 -0700 (PDT) David Rientjes <rientjes(a)google.com> wrote: > > diff --git a/mm/oom_kill.c b/mm/oom_kill.c > > index 0cb1ca4..9e89a29 100644 > > --- a/mm/oom_kill.c > > +++ b/mm/oom_kill.c > > @@ -510,8 +510,10 @@ retry: > > if (PTR_ERR(p) == -1UL) > > goto out; > > > > - if (!p) > > - p = current; > > + if (!p) { > > + read_unlock(&tasklist_lock); > > + panic("Out of memory and no killable processes...\n"); > > + } > > > > if (oom_kill_process(p, gfp_mask, 0, points, limit, mem, > > "Memory cgroup out of memory")) > > > > This actually does appear to be necessary but for a different reason: if > current is unkillable because it has OOM_DISABLE, for example, then > oom_kill_process() will repeatedly fail and mem_cgroup_out_of_memory() > will infinitely loop. > > Kame-san? > When a memcg goes into OOM and it only has unkillable processes (OOM_DISABLE), we can do nothing. (we can't panic because container's death != system death.) Because memcg itself has mutex+waitqueue for mutual execusion of OOM killer, I think infinite-loop will not be critical probelm for the whole system. And, now, memcg has oom-kill-disable + oom-kill-notifier features. So, If a memcg goes into OOM and there is no killable process, but oom-kill is not disabled by memcg.....it means system admin's mis-configuraton. He can stop inifite loop by hand, anyway. # echo 1 > ..../group_A/memory.oom_control Thanks, -Kame -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: David Rientjes on 31 Mar 2010 02:10 On Wed, 31 Mar 2010, KAMEZAWA Hiroyuki wrote: > > > diff --git a/mm/oom_kill.c b/mm/oom_kill.c > > > index 0cb1ca4..9e89a29 100644 > > > --- a/mm/oom_kill.c > > > +++ b/mm/oom_kill.c > > > @@ -510,8 +510,10 @@ retry: > > > if (PTR_ERR(p) == -1UL) > > > goto out; > > > > > > - if (!p) > > > - p = current; > > > + if (!p) { > > > + read_unlock(&tasklist_lock); > > > + panic("Out of memory and no killable processes...\n"); > > > + } > > > > > > if (oom_kill_process(p, gfp_mask, 0, points, limit, mem, > > > "Memory cgroup out of memory")) > > > > > > > This actually does appear to be necessary but for a different reason: if > > current is unkillable because it has OOM_DISABLE, for example, then > > oom_kill_process() will repeatedly fail and mem_cgroup_out_of_memory() > > will infinitely loop. > > > > Kame-san? > > > > When a memcg goes into OOM and it only has unkillable processes (OOM_DISABLE), > we can do nothing. (we can't panic because container's death != system death.) > > Because memcg itself has mutex+waitqueue for mutual execusion of OOM killer, > I think infinite-loop will not be critical probelm for the whole system. > > And, now, memcg has oom-kill-disable + oom-kill-notifier features. > So, If a memcg goes into OOM and there is no killable process, but oom-kill is > not disabled by memcg.....it means system admin's mis-configuraton. > > He can stop inifite loop by hand, anyway. > # echo 1 > ..../group_A/memory.oom_control > Then we should be able to do this since current is by definition unkillable since it was not found in select_bad_process(), right? --- diff --git a/mm/oom_kill.c b/mm/oom_kill.c --- a/mm/oom_kill.c +++ b/mm/oom_kill.c @@ -500,12 +500,9 @@ void mem_cgroup_out_of_memory(struct mem_cgroup *mem, gfp_t gfp_mask) read_lock(&tasklist_lock); retry: p = select_bad_process(&points, limit, mem, CONSTRAINT_NONE, NULL); - if (PTR_ERR(p) == -1UL) + if (!p || PTR_ERR(p) == -1UL) goto out; - if (!p) - p = current; - if (oom_kill_process(p, gfp_mask, 0, points, limit, mem, "Memory cgroup out of memory")) goto retry; -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: KAMEZAWA Hiroyuki on 31 Mar 2010 02:20 On Tue, 30 Mar 2010 23:07:08 -0700 (PDT) David Rientjes <rientjes(a)google.com> wrote: > On Wed, 31 Mar 2010, KAMEZAWA Hiroyuki wrote: > > > > > diff --git a/mm/oom_kill.c b/mm/oom_kill.c > > > > index 0cb1ca4..9e89a29 100644 > > > > --- a/mm/oom_kill.c > > > > +++ b/mm/oom_kill.c > > > > @@ -510,8 +510,10 @@ retry: > > > > if (PTR_ERR(p) == -1UL) > > > > goto out; > > > > > > > > - if (!p) > > > > - p = current; > > > > + if (!p) { > > > > + read_unlock(&tasklist_lock); > > > > + panic("Out of memory and no killable processes...\n"); > > > > + } > > > > > > > > if (oom_kill_process(p, gfp_mask, 0, points, limit, mem, > > > > "Memory cgroup out of memory")) > > > > > > > > > > This actually does appear to be necessary but for a different reason: if > > > current is unkillable because it has OOM_DISABLE, for example, then > > > oom_kill_process() will repeatedly fail and mem_cgroup_out_of_memory() > > > will infinitely loop. > > > > > > Kame-san? > > > > > > > When a memcg goes into OOM and it only has unkillable processes (OOM_DISABLE), > > we can do nothing. (we can't panic because container's death != system death.) > > > > Because memcg itself has mutex+waitqueue for mutual execusion of OOM killer, > > I think infinite-loop will not be critical probelm for the whole system. > > > > And, now, memcg has oom-kill-disable + oom-kill-notifier features. > > So, If a memcg goes into OOM and there is no killable process, but oom-kill is > > not disabled by memcg.....it means system admin's mis-configuraton. > > > > He can stop inifite loop by hand, anyway. > > # echo 1 > ..../group_A/memory.oom_control > > > > Then we should be able to do this since current is by definition > unkillable since it was not found in select_bad_process(), right? To me, this patch is acceptable and seems reasnoable. But I didn't joined to memcg development when this check was added and don't know why kill current.. http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=c7ba5c9e8176704bfac0729875fa62798037584d Addinc Balbir to CC. Maybe situation is changed now. Because we can stop inifinite loop (by hand) and there is no rushing oom-kill callers, this change is acceptable. Thanks, -Kame > --- > diff --git a/mm/oom_kill.c b/mm/oom_kill.c > --- a/mm/oom_kill.c > +++ b/mm/oom_kill.c > @@ -500,12 +500,9 @@ void mem_cgroup_out_of_memory(struct mem_cgroup *mem, gfp_t gfp_mask) > read_lock(&tasklist_lock); > retry: > p = select_bad_process(&points, limit, mem, CONSTRAINT_NONE, NULL); > - if (PTR_ERR(p) == -1UL) > + if (!p || PTR_ERR(p) == -1UL) > goto out; > > - if (!p) > - p = current; > - > if (oom_kill_process(p, gfp_mask, 0, points, limit, mem, > "Memory cgroup out of memory")) > goto retry; > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: David Rientjes on 31 Mar 2010 02:40 On Wed, 31 Mar 2010, Balbir Singh wrote: > > To me, this patch is acceptable and seems reasnoable. > > > > But I didn't joined to memcg development when this check was added > > and don't know why kill current.. > > > > The reason for adding current was that we did not want to loop > forever, since it stops forward progress - no error/no forward > progress. It made sense to oom kill the current process, so that the > cgroup admin could look at what went wrong. > oom_kill_process() will fail on current since it wasn't selected as an eligible task to kill in select_bad_process() and we know it to be a member of the memcg, so there's no point in trying to kill it. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: KAMEZAWA Hiroyuki on 31 Mar 2010 02:40
On Wed, 31 Mar 2010 12:00:07 +0530 Balbir Singh <balbir(a)linux.vnet.ibm.com> wrote: > * KAMEZAWA Hiroyuki <kamezawa.hiroyu(a)jp.fujitsu.com> [2010-03-31 15:13:56]: > > > On Tue, 30 Mar 2010 23:07:08 -0700 (PDT) > > David Rientjes <rientjes(a)google.com> wrote: > > > > > On Wed, 31 Mar 2010, KAMEZAWA Hiroyuki wrote: > > > > > > > > > diff --git a/mm/oom_kill.c b/mm/oom_kill.c > > > > > > index 0cb1ca4..9e89a29 100644 > > > > > > --- a/mm/oom_kill.c > > > > > > +++ b/mm/oom_kill.c > > > > > > @@ -510,8 +510,10 @@ retry: > > > > > > if (PTR_ERR(p) == -1UL) > > > > > > goto out; > > > > > > > > > > > > - if (!p) > > > > > > - p = current; > > > > > > + if (!p) { > > > > > > + read_unlock(&tasklist_lock); > > > > > > + panic("Out of memory and no killable processes...\n"); > > > > > > + } > > > > > > > > > > > > if (oom_kill_process(p, gfp_mask, 0, points, limit, mem, > > > > > > "Memory cgroup out of memory")) > > > > > > > > > > > > > > > > This actually does appear to be necessary but for a different reason: if > > > > > current is unkillable because it has OOM_DISABLE, for example, then > > > > > oom_kill_process() will repeatedly fail and mem_cgroup_out_of_memory() > > > > > will infinitely loop. > > > > > > > > > > Kame-san? > > > > > > > > > > > > > When a memcg goes into OOM and it only has unkillable processes (OOM_DISABLE), > > > > we can do nothing. (we can't panic because container's death != system death.) > > > > > > > > Because memcg itself has mutex+waitqueue for mutual execusion of OOM killer, > > > > I think infinite-loop will not be critical probelm for the whole system. > > > > > > > > And, now, memcg has oom-kill-disable + oom-kill-notifier features. > > > > So, If a memcg goes into OOM and there is no killable process, but oom-kill is > > > > not disabled by memcg.....it means system admin's mis-configuraton. > > > > > > > > He can stop inifite loop by hand, anyway. > > > > # echo 1 > ..../group_A/memory.oom_control > > > > > > > > > > Then we should be able to do this since current is by definition > > > unkillable since it was not found in select_bad_process(), right? > > > > To me, this patch is acceptable and seems reasnoable. > > > > But I didn't joined to memcg development when this check was added > > and don't know why kill current.. > > > > The reason for adding current was that we did not want to loop > forever, since it stops forward progress - no error/no forward > progress. It made sense to oom kill the current process, so that the > cgroup admin could look at what went wrong. > Now, notifier is triggered. > > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=c7ba5c9e8176704bfac0729875fa62798037584d > > > > Addinc Balbir to CC. Maybe situation is changed now. > > Because we can stop inifinite loop (by hand) and there is no rushing oom-kill > > callers, this change is acceptable. > > > > By hand is not always possible if we have a large number of cgroups > (I've seen a setup with 2000 cgroups on libcgroup ML). 2000 cgroups * > number of processes make the situation complex. I think using OOM > notifier is now another way of handling such a situation. > "By hand" includes "automatically with daemon program", of course. Hmm, in short, your opinion is "killing current is good for now" ? I have no strong opinion, here. (Because I'll recommend all customers to disable oom kill if they don't want any task to be killed automatically.) Thanks, -Kame -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |