Memory footprint of a mail server
in [Debian]
|
From: Thomas Goirand on 5 May 2010 01:10 Hi, Back when we were running Sarge, we were able to run clamav, spamassassin and amavis, together with apache, mysql and an ftp server, in just 200MB RAM and same as swap. Today, running Lenny, top shows us some crazy results: clamav: 156 MB amavisd-new: 80 MB per process (running 3 processes is the minimum) spamd: 105 MB per process (same remark) That makes the total amount of RAM needed to run these 3 up to something like 700 MB, which makes it a hard fit for smaller end VPSes. What happened between Sarge and Lenny is beyond me. It seems we could replace amavisd-new by clamsmtp, which would save 240MB of RAM, but clamav and spamd will still have to be there, as there's no other open source replacement available. So, is there anyone with an idea on how improve memory efficiency here? Is there a way to improve perl apps memory footprint, so it's not a dog, in a more global way on the server? Is dspam better than spamassassin for it's using less RAM? Has anyone tried clamsmtp and how much RAM does it take (we would remove amavisd-new totally)? How does the Markovian discrimination of DSPAM performs? How about the integration of dspam and clamsmtp in Debian? Thomas -- To UNSUBSCRIBE, email to debian-user-REQUEST(a)lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmaster(a)lists.debian.org Archive: http://lists.debian.org/4BE0F94C.3060700(a)goirand.fr
From: Stan Hoeppner on 5 May 2010 01:40 Thomas Goirand put forth on 5/4/2010 11:51 PM: > Hi, > > Back when we were running Sarge, we were able to run clamav, > spamassassin and amavis, together with apache, mysql and an ftp server, > in just 200MB RAM and same as swap. > > Today, running Lenny, top shows us some crazy results: > > clamav: 156 MB > amavisd-new: 80 MB per process (running 3 processes is the minimum) > spamd: 105 MB per process (same remark) Please post top output so we can see the actual VIRT, RES, CODE, DATA, SHR, and %MEM column data for these processes. My initial guess is that you have a bit of consolidation going on that you're not seeing. You may want to run pmap on some of those duplicate processes to see what portion of their footprint is shared. -- Stan -- To UNSUBSCRIBE, email to debian-user-REQUEST(a)lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmaster(a)lists.debian.org Archive: http://lists.debian.org/4BE1028C.6030802(a)hardwarefreak.com
From: Mihamina Rakotomandimby on 5 May 2010 03:40 > Thomas Goirand <thomas(a)goirand.fr> : >Today, running Lenny, top shows us some crazy results: >clamav: 156 MB >amavisd-new: 80 MB per process (running 3 processes is the minimum) >spamd: 105 MB per process (same remark) >That makes the total amount of RAM needed to run these 3 up to >something like 700 MB, which makes it a hard fit for smaller end VPSes. We use on each openVZ VE: clamav + spamassasin. VEs are 2GB RAM. So yes, your observations are right. But I think that's the way it goes. -- Architecte Informatique chez Blueline/Gulfsat: Administration Systeme, Recherche & Developpement +261 34 29 155 34 / +261 33 11 207 36 -- To UNSUBSCRIBE, email to debian-user-REQUEST(a)lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmaster(a)lists.debian.org Archive: http://lists.debian.org/20100505103722.6c42c584(a)pbmiha.malagasy.com
From: Camaleón on 5 May 2010 05:00 On Wed, 05 May 2010 12:51:24 +0800, Thomas Goirand wrote: > Today, running Lenny, top shows us some crazy results: > > clamav: 156 MB > amavisd-new: 80 MB per process (running 3 processes is the minimum) > spamd: 105 MB per process (same remark) > > That makes the total amount of RAM needed to run these 3 up to something > like 700 MB, which makes it a hard fit for smaller end VPSes. What > happened between Sarge and Lenny is beyond me. It seems we could replace > amavisd-new by clamsmtp, which would save 240MB of RAM, but clamav and > spamd will still have to be there, as there's no other open source > replacement available. Mmmm, amavisd-new can call spamassassin directly, so no need to use "spamd/spamc" daemonized version. OTOH, I also see the same "numbers" on my systems... Greetings, -- Camaleón -- To UNSUBSCRIBE, email to debian-user-REQUEST(a)lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmaster(a)lists.debian.org Archive: http://lists.debian.org/pan.2010.05.05.08.58.42(a)gmail.com
From: Boyd Stephen Smith Jr. on 5 May 2010 11:00
On Tuesday 04 May 2010 23:51:24 Thomas Goirand wrote: > Hi, > > Back when we were running Sarge, we were able to run clamav, > spamassassin and amavis, together with apache, mysql and an ftp server, > in just 200MB RAM and same as swap. > > Today, running Lenny, top shows us some crazy results: > > clamav: 156 MB > amavisd-new: 80 MB per process (running 3 processes is the minimum) > spamd: 105 MB per process (same remark) Is that virtual or resident? While virtual memory usage can be important, resident memory is a better initial measuring stick. For my two-VPS system, I have: vps1: exim4 -- 83M virt; 344K resident (listener; always around) exim4 -- 83M virt; 1.7M resident (worker; not always around) exim4 -- 83M virt; 1.7M resident (worker; not always around) spamd -- 101M virt; 4M resident vps2: exim4 -- 42M virt; 124K resident clamd -- 280M virt; 159M resident clamd -- 280M virt; 159M resident freshclam -- 42M virt; 124K resident You can guess why I moved clamAV to a separate system. Upstream is fine with the memory footprint, so I don't expect it to go down. Very hard to fit with any other services on a 256M VPS. Also, clamAV's memory usage tends to spike at just the wrong times, so I do recommend isolating it. Technically, the exim on vps2 doesn't participate in providing mail services. It's exim handles local delivery, doesn't listen on non-loopback interfaces, and, and uses vps1 (over a "private" network) as a smarthost for all non-local delivery. Virtual memory usage isn't a good measure by itself since virtual memory pages can be a) shared (but top/htop charges each process with all pages) and b) file-backed and never accessed. For example, when a shared library is loaded, the whole .so is mapped into memory. However, only the pages that are actually read by the program are read from disk and copied into RAM. If the program loads a large library but only uses a couple of functions in it, the virtual memory usage of that program will greatly increase but the shared memory usage of that program will only increase slightly and the resident memory usage of that program won't change.[1] > That makes the total amount of RAM needed to run these 3 up to something > like 700 MB, which makes it a hard fit for smaller end VPSes. Virtual memory usage can't simply be added like that, because part of virtual memory usage includes shared pages. exim4+clamav+spamd is not confortable on a 256M vps, mainly because clamd is a memory "hog". I've split them between 2 256M vpses, and it has been working well. I imagine a 512M VPS could also handle it without trouble. ClamAV is the biggest problem, I've seen its resident memory size exceed 250M during scans. > Is there a way to improve perl apps memory footprint, so it's not a dog, > in a more global way on the server? Taking a lot of virtual memory means nothing. Virtual memory addresses are not necessarily backed by either RAM or swap on Linux. The resident memory of perl applications really shouldn't be much more (if any) than the equivalent C++ program using boost or the STL. [C programs are probably a little bit of a win here, if you don't re-implement the garbage collection and dynamic typing of Perl.] > Has anyone tried clamsmtp and how much RAM does > it take (we would remove amavisd-new totally)? If the goal is to get rid of amavisd-new, exim4-daemon-heavy has built-in support for virus scanning through clamav. > How about the integration of dspam and > clamsmtp in Debian? I'm not sure why you need either of these programs. exim4-daemon-heavy already contains support for interfacing with a number of spam/virus detection daemons that run on the same or different systems. -- Boyd Stephen Smith Jr. ,= ,-_-. =. bss(a)iguanasuicide.net ((_/)o o(\_)) ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-' http://iguanasuicide.net/ \_/ |