Exposing the Xact commit order to the user
in [PgSql]
|
Prev: [HACKERS] Exposing the Xact commit order to the user
Next: [HACKERS] Synchronization levels in SR
From: Greg Stark on 26 May 2010 10:04 On Sun, May 23, 2010 at 9:21 PM, Jan Wieck <JanWieck(a)yahoo.com> wrote: > Each record of the Transaction Commit Info consists of > > txid xci_transaction_id > timestamptz xci_begin_timestamp > timestamptz xci_commit_timestamp > int64 xci_total_rowcount > So I think you're going about this backwards. Instead of discussing implementation I think you should start with the API the replication system needs. In particular I'm not sure you really want a server-side query at all. I'm wondering if you wouldn't be better off with a public machine-parsable text format version of the WAL. Ie, at the same time as writing out all the nitty gritty to the binary wal we would write out a summary of public data to an xml version containing just parts of the data stream that we can promise won't change, such as transaction id, lsn, timestamp. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Jan Wieck on 26 May 2010 11:11 On 5/26/2010 10:04 AM, Greg Stark wrote: > On Sun, May 23, 2010 at 9:21 PM, Jan Wieck <JanWieck(a)yahoo.com> wrote: >> Each record of the Transaction Commit Info consists of >> >> txid xci_transaction_id >> timestamptz xci_begin_timestamp >> timestamptz xci_commit_timestamp >> int64 xci_total_rowcount >> > > So I think you're going about this backward > > Instead of discussing implementation I think you should start with the > API the replication system needs. In particular I'm not sure you > really want a server-side query at all. I'm wondering if you wouldn't > be better off with a public machine-parsable text format version of > the WAL. Ie, at the same time as writing out all the nitty gritty to > the binary wal we would write out a summary of public data to an xml > version containing just parts of the data stream that we can promise > won't change, such as transaction id, lsn, timestamp. Since the actual row level change information and other event data is found inside of regular tables, identified by TXID and sequence number, I am pretty sure I want that data in a server-side query. What you are proposing is to read the xid's and timestamps with an external process, that now forcibly needs to reside on the DB server itself (neither Londiste nor Slony have that requirement as of today), then bring it back into the DB at least inside the WHERE clause of a query. Jan -- Anyone who trades liberty for security deserves neither liberty nor security. -- Benjamin Franklin -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Jan Wieck on 26 May 2010 12:10 On 5/26/2010 10:04 AM, Greg Stark wrote: > Instead of discussing implementation I think you should start with the > API the replication system needs. .... but to answer that request, actually I don't even think we should be discussing API specifics. During PGCon, Marco Kreen, Jim Nasby and I were discussing what the requirements of a unified message queue, shared by Londiste and Slony may look like. For some use cases of pgq, there isn't even any interest in user table changes. These are simply a reliable, database backed message passing system. Today both systems use an "agreeable" order of changes selected by rather expensive queries based on serializable snapshot information and a global, non cacheable serial number. This could be replaced with a logic based on the actual commit order of the transactions. This order does not need to be 100% accurate. As long as the order is recorded after all user actions have been performed (trigger queue shut down) and while the transaction is still holding onto its locks, that order is good enough. This will not allow a conflicting transaction, waiting on locks to be released, to appear having committed before the lock conflict winner. It is obvious that in cases where only small portions or even none of the user table changes are needed, holding on to or even parsing the ENTIRE WAL sounds suboptimal for this use case. Jan -- Anyone who trades liberty for security deserves neither liberty nor security. -- Benjamin Franklin -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Greg Stark on 26 May 2010 12:38 On Wed, May 26, 2010 at 5:10 PM, Jan Wieck <JanWieck(a)yahoo.com> wrote: > ... but to answer that request, actually I don't even think we should be > discussing API specifics. > How about just API generalities? Like, where do you need this data, on the master or on the slave? Would PGXC like it on the transaction coordinator? What question do you need to answer, do you need to pull out sets of commits in certain ranges or look up specific transaction ids and find out when they committed? Or do you only need to answer which of two transaction ids committed first? -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Heikki Linnakangas on 26 May 2010 13:17
Could you generate the commit-order log by simply registering a commit hook (RegisterXactCallback(XACT_EVENT_COMMIT)) that writes such a log somewhere in the data directory? That would work with older versions too, no server changes required. It would not get called during recovery, but I believe that would be sufficient for Slony. You could always batch commits that you don't know when they committed as if they committed simultaneously. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |