Streaming replication and non-blocking I/O
in [PgSql]
|
Prev: Sought after architectures for the PostgreSQL buildfarm?
Next: [HACKERS] 答复: questions about concurrency control in Postgresql
From: Fujii Masao on 8 Dec 2009 20:42 On Tue, Dec 8, 2009 at 11:23 PM, Heikki Linnakangas <heikki.linnakangas(a)enterprisedb.com> wrote: > The first question is: do we really need to be prepared for that? The > XLogRecPtr acknowledgment messages the client sends are very small, and > if the client is mindful about not sending them too often - perhaps max > 1 ack per 1 received XLOG message - the receive buffer in the backend > should never fill up in practice. It's OK to drop that feature. > If that's deemed not good enough, we could modify just internal_flush() > so that it uses secure_poll to wait for the possibility to either read > or write, instead of blocking for just write. Whenever there's incoming > data, read them into PqRecvBuffer for later processing, which keeps the > OS input buffer from filling up. If PqRecvBuffer fills up, it can be > extended, or we can start dropping old XLogRecPtr messages from it. Extending PqRecvBuffer seems better because XLogRecPtr message has some types (i.e., we cannot just drop old message without parsing all messages in the buffer). Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Heikki Linnakangas on 9 Dec 2009 01:58 Fujii Masao wrote: > On Tue, Dec 8, 2009 at 11:23 PM, Heikki Linnakangas > <heikki.linnakangas(a)enterprisedb.com> wrote: >> If that's deemed not good enough, we could modify just internal_flush() >> so that it uses secure_poll to wait for the possibility to either read >> or write, instead of blocking for just write. Whenever there's incoming >> data, read them into PqRecvBuffer for later processing, which keeps the >> OS input buffer from filling up. If PqRecvBuffer fills up, it can be >> extended, or we can start dropping old XLogRecPtr messages from it. > > Extending PqRecvBuffer seems better because XLogRecPtr message > has some types (i.e., we cannot just drop old message without parsing > all messages in the buffer). True. Another idea I had was to introduce a callback that backend libpq can call when the buffer fills. Walsender would set the callback to ProcessStreamMsgs(). But if everyone is happy with just relying on the OS buffer to not fill up, let's just drop it. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Fujii Masao on 9 Dec 2009 02:55 On Wed, Dec 9, 2009 at 3:58 PM, Heikki Linnakangas <heikki.linnakangas(a)enterprisedb.com> wrote: > True. Another idea I had was to introduce a callback that backend libpq > can call when the buffer fills. Walsender would set the callback to > ProcessStreamMsgs(). > > But if everyone is happy with just relying on the OS buffer to not fill > up, let's just drop it. The OS buffer is expected to be able to store a large number of XLogRecPtr messages, because its size is small. So it's also OK to just drop it. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Tom Lane on 9 Dec 2009 10:00 Fujii Masao <masao.fujii(a)gmail.com> writes: > On Wed, Dec 9, 2009 at 3:58 PM, Heikki Linnakangas > <heikki.linnakangas(a)enterprisedb.com> wrote: >> But if everyone is happy with just relying on the OS buffer to not fill >> up, let's just drop it. > The OS buffer is expected to be able to store a large number of > XLogRecPtr messages, because its size is small. So it's also OK > to just drop it. It certainly seems to be something we could improve later, when and if evidence emerges that it's a real-world problem. For now, simple is beautiful. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Fujii Masao on 10 Dec 2009 01:41
On Thu, Dec 10, 2009 at 12:00 AM, Tom Lane <tgl(a)sss.pgh.pa.us> wrote: >> The OS buffer is expected to be able to store a large number of >> XLogRecPtr messages, because its size is small. So it's also OK >> to just drop it. > > It certainly seems to be something we could improve later, when and > if evidence emerges that it's a real-world problem. For now, > simple is beautiful. I just dropped the backend libpq changes related to non-blocking I/O. git://git.postgresql.org/git/users/fujii/postgres.git branch: replication Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |