Streaming replication and non-blocking I/O
in [PgSql]
|
Prev: Sought after architectures for the PostgreSQL buildfarm?
Next: [HACKERS] 答复: questions about concurrency control in Postgresql
From: Heikki Linnakangas on 17 Dec 2009 08:25 Fujii Masao wrote: > I'm thinking of making the standby send the "walsender-switch-code" the same way > as application_name; walreceiver always specifies the option like > "replication=on" > in conninfo string and calls PQconnectdb(), which sends the code as a part of > startup packet. And, the environment variable for that should not be defined to > avoid user's mis-configuration, I think. Sounds good. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Fujii Masao on 21 Dec 2009 07:56 On Fri, Dec 18, 2009 at 11:42 AM, Fujii Masao <masao.fujii(a)gmail.com> wrote: > Okey. Design clarification again; > > 0. Begin by connecting to the master using PQconnectdb() with new conninfo > option specifying the request of replication. The startup packet with the > request is sent to the master, then the backend switches to the walsender > mode. The walsender goes into the main loop and wait for the request from > the walreceiver. <snip> > 4. Start replication > > Slave -> Master: Query message, with query string "START REPLICATION: > XXXX", where XXXX is the RecPtr of the starting point. > > Master -> Slave: CopyOutResponse followed by a continuous stream of > CopyData messages with WAL contents. Done. Currently there is no new libpq function for replication. The walreceiver uses only existing functions like PQconnectdb, PQexec, PQgetCopyData, etc. git://git.postgresql.org/git/users/fujii/postgres.git branch: replication Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Heikki Linnakangas on 21 Dec 2009 12:26 Fujii Masao wrote: > On Fri, Dec 18, 2009 at 11:42 AM, Fujii Masao <masao.fujii(a)gmail.com> wrote: >> Okey. Design clarification again; >> >> 0. Begin by connecting to the master using PQconnectdb() with new conninfo >> option specifying the request of replication. The startup packet with the >> request is sent to the master, then the backend switches to the walsender >> mode. The walsender goes into the main loop and wait for the request from >> the walreceiver. > <snip> >> 4. Start replication >> >> Slave -> Master: Query message, with query string "START REPLICATION: >> XXXX", where XXXX is the RecPtr of the starting point. >> >> Master -> Slave: CopyOutResponse followed by a continuous stream of >> CopyData messages with WAL contents. > > Done. Currently there is no new libpq function for replication. The > walreceiver uses only existing functions like PQconnectdb, PQexec, > PQgetCopyData, etc. Ok thanks, sounds good, I'll take a look. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Heikki Linnakangas on 21 Dec 2009 12:31 Fujii Masao wrote: > On Tue, Dec 15, 2009 at 4:11 AM, Tom Lane <tgl(a)sss.pgh.pa.us> wrote: >> Hm. Perhaps it should be a loadable plugin and not hard-linked into the >> backend? Compare dblink. > > You mean that such plugin is supplied in shared_preload_libraries, > a new process is forked and the shared-memory related to walreceiver > is created by using shmem_startup_hook? Since this approach would > solve the problem discussed previously, ISTM this makes sense. > http://archives.postgresql.org/pgsql-hackers/2009-11/msg00031.php > > Some additional code might be required to control the termination > of walreceiver. I'm not sure which problem in that thread you're referring to, but I can see two options: 1. Use dlopen()/dlsym() in walreceiver to use libpq. A bit awkward, though we could write a bunch of macros to hide that and make the libpq calls look normal. 2. Move walreceiver altogether into a loadable module, which is linked as usual to libpq. Like e.g contrib/dblink. Thoughts? Both seem reasonable to me. I tested the 2nd option (see 'replication' branch in my git repository), splitting walreceiver.c into two: the functions that run in the walreceiver process, and the functions that are called from other processes to control walreceiver. That's a quite nice separation, though of course we could do that with the 1st approach as well. PS. I just merged with CVS HEAD. Streaming replication is pretty awesome with Hot Standby! -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Tom Lane on 21 Dec 2009 13:21
Heikki Linnakangas <heikki.linnakangas(a)enterprisedb.com> writes: > Fujii Masao wrote: > I'm not sure which problem in that thread you're referring to, but I can > see two options: > 1. Use dlopen()/dlsym() in walreceiver to use libpq. A bit awkward, > though we could write a bunch of macros to hide that and make the libpq > calls look normal. > 2. Move walreceiver altogether into a loadable module, which is linked > as usual to libpq. Like e.g contrib/dblink. > Thoughts? Both seem reasonable to me. From a packager's standpoint the second is much saner. If you want to use dlopen() then you will have to know the exact name of the .so file (e.g. libpq.so.5.3) and possibly its location too. Or you will have to persuade packagers that they should ship bare "libpq.so" symlinks, which is contrary to packaging standards on most Linux distros. (walreceiver.so wouldn't be subject to those standards, but libpq is because it's a regular library that can also be hard-linked by applications.) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |