testing HS/SR - 1 vs 2 performance
in [PgSql]
|
Prev: pg_ctl stop -m immediate on the primary server inflatessequences
Next: [HACKERS] non-reproducible failure of random test on HEAD
From: Dimitri Fontaine on 14 Apr 2010 04:24 Heikki Linnakangas <heikki.linnakangas(a)enterprisedb.com> writes: > Changing the KnownAssignedXids data structure from > hash table into something that's quicker to scan. Preferably something > with O(N), where N is the number of entries in the data structure, not > the maximum number of entries it can hold as it is with the hash table > currently. So that's pretty good news RedBlack Trees made it in 9.0, isn't it? :) > A quick fix would be to check if there's any entries in the hash table > before scanning it. That would eliminate the overhead when there's no > in-progress transactions in the master. But as soon as there's even one, > the overhead comes back. Does not sound like typical, does it? -- dim -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Simon Riggs on 14 Apr 2010 08:08 On Tue, 2010-04-13 at 21:09 +0300, Heikki Linnakangas wrote: > Heikki Linnakangas wrote: > > I could reproduce this on my laptop, standby is about 20% slower. I ran > > oprofile, and what stands out as the difference between the master and > > standby is that on standby about 20% of the CPU time is spent in > > hash_seq_search(). The callpath is GetSnapshotDat() -> > > KnownAssignedXidsGetAndSetXmin() -> hash_seq_search(). That explains the > > difference in performance. > > The slowdown is proportional to the max_connections setting in the > standby. 20% slowdown might still be acceptable, but if you increase > max_connections to say 1000, things get really slow. I wouldn't > recommend max_connections=1000, of course, but I think we need to do > something about this. Changing the KnownAssignedXids data structure from > hash table into something that's quicker to scan. Preferably something > with O(N), where N is the number of entries in the data structure, not > the maximum number of entries it can hold as it is with the hash table > currently. There's a tradeoff here to consider. KnownAssignedXids faces two workloads: one for each WAL record where we check if the xid is already known assigned, one for snapshots. The current implementation is optimised towards recovery performance, not snapshot performance. > A quick fix would be to check if there's any entries in the hash table > before scanning it. That would eliminate the overhead when there's no > in-progress transactions in the master. But as soon as there's even one, > the overhead comes back. Any fix should be fairly quick because of the way its modularised - with something like this in mind. I'll try a circular buffer implementation, with fastpath. Have something in a few hours. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Heikki Linnakangas on 16 Apr 2010 04:29 Simon Riggs wrote: > On Tue, 2010-04-13 at 21:09 +0300, Heikki Linnakangas wrote: >> A quick fix would be to check if there's any entries in the hash table >> before scanning it. That would eliminate the overhead when there's no >> in-progress transactions in the master. But as soon as there's even one, >> the overhead comes back. > > Any fix should be fairly quick because of the way its modularised - with > something like this in mind. > > I'll try a circular buffer implementation, with fastpath. I started experimenting with a sorted array based implementation on Tuesday but got carried away with other stuff. I now got back to that and cleaned it up. How does the attached patch look like? It's probably similar to what you had in mind. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
From: Simon Riggs on 16 Apr 2010 05:10 On Fri, 2010-04-16 at 11:29 +0300, Heikki Linnakangas wrote: > Simon Riggs wrote: > > On Tue, 2010-04-13 at 21:09 +0300, Heikki Linnakangas wrote: > >> A quick fix would be to check if there's any entries in the hash table > >> before scanning it. That would eliminate the overhead when there's no > >> in-progress transactions in the master. But as soon as there's even one, > >> the overhead comes back. > > > > Any fix should be fairly quick because of the way its modularised - with > > something like this in mind. > > > > I'll try a circular buffer implementation, with fastpath. > > I started experimenting with a sorted array based implementation on > Tuesday but got carried away with other stuff. I now got back to that > and cleaned it up. > > How does the attached patch look like? It's probably similar to what you > had in mind. It looks like a second version of what I'm working on and about to publish. I'll take that as a compliment! My patch is attached here also, for discussion. The two patches look same in their main parts, though I have quite a few extra tweaks in there, which you can read about in comments. One tweak I don't have is the use of the presence array that allows a sensible bsearch, so I'll to alter my patch to use that idea but keep the rest of my code. -- Simon Riggs www.2ndQuadrant.com
From: Heikki Linnakangas on 16 Apr 2010 07:47
Simon Riggs wrote: > On Fri, 2010-04-16 at 11:29 +0300, Heikki Linnakangas wrote: >> How does the attached patch look like? It's probably similar to what you >> had in mind. > > It looks like a second version of what I'm working on and about to > publish. I'll take that as a compliment! > > My patch is attached here also, for discussion. > > The two patches look same in their main parts, though I have quite a few > extra tweaks in there, which you can read about in comments. Yeah. Yours seems a lot more complex with all those extra tweaks, I would suggest to keep it simple. I did realize one bug in my patch: I didn't handle xid wraparound correctly in the binary search, need to use TransactionIdFollows instead of plan >. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |