From: Takahiro Itagaki on 29 Jan 2010 02:29 Greg Smith <greg(a)2ndquadrant.com> wrote: > Attached is a patch that fixes a long standing bug in pgbench: it won't > handle scale factors above ~4000 (around 60GB) because it uses 32-bit > integers for its computations related to the number of accounts, and it > just crashes badly when you exceed that. This month I've run into two > systems where that was barely enough to exceed physical RAM, so I'd > expect this to be a significant limiting factor during 9.0's lifetime. > A few people have complained about it already in 8.4. +1 for the fix. Do we also need to adjust "tuples done" messages during dataload? It would be too verbose for large scale factor. I think a message every 1% is reasonable. if (j % 10000 == 0) fprintf(stderr, INT64_FORMAT " tuples done.\n", j); Regards, --- Takahiro Itagaki NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Tom Lane on 29 Jan 2010 11:09 Greg Smith <greg(a)2ndquadrant.com> writes: > Was looking for general feedback on whether the way I've converted this > to use 64 bit integers for the account numbers seems appropriate, and to > see if there's any objection to fixing this in general given the > potential downsides. In the past we've rejected proposed patches for pgbench on the grounds that they would make results non-comparable to previous results. So the key question here is how much this affects the speed. Please be sure to test that on a 32-bit machine, not a 64-bit one. > ! retval = (int64) strtol(res, &endptr, 19); That bit is merely wishful thinking :-( regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Robert Haas on 29 Jan 2010 11:20 On Fri, Jan 29, 2010 at 11:09 AM, Tom Lane <tgl(a)sss.pgh.pa.us> wrote: > Greg Smith <greg(a)2ndquadrant.com> writes: >> Was looking for general feedback on whether the way I've converted this >> to use 64 bit integers for the account numbers seems appropriate, and to >> see if there's any objection to fixing this in general given the >> potential downsides. > > In the past we've rejected proposed patches for pgbench on the grounds > that they would make results non-comparable to previous results. Perhaps we need an option indicating whether or not the use of bigint columns is OK. ....Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Greg Smith on 29 Jan 2010 14:33 Tom Lane wrote: > In the past we've rejected proposed patches for pgbench on the grounds > that they would make results non-comparable to previous results. So the > key question here is how much this affects the speed. Please be sure to > test that on a 32-bit machine, not a 64-bit one. > Sheesh, who has a 32-bit machine anymore? I'll see what older hardware I can dig up. I've realized there are two separate issues to be concerned about: 1) On small scale data sets, what's the impact of the main piece of data being shuffled around in memory (the account number in the accounts table) now being 64 bits? That part might be significantly worse on 32-bit hardware. 2) How does the expansion in size of the related primary key on that data impact the breakpoint where the database doesn't fit in RAM anymore? I did just updated my pgbench-tools package this month so that it happily runs against either 8.3 or 8.4/9.0 and I've done two rounds of extensive test runs lately, so plenty of data to compare against here. >> ! retval = (int64) strtol(res, &endptr, 19); >> > > That bit is merely wishful thinking :-( > I did specificially say I didn't trust that call one bit. There is a middle ground position here, similar to what Robert suggested, that I just add a "large mode" to the program for people who need it without touching the current case. That might allow me to sidestep some of these issues I may not have a good answer to with getting the \setshell feature working right in 64 bits, could just make that one specific to "regular mode". In any case, I think this limitation in what pgbench can do has risen to be a full-on bug at this point for the expected users of the next version, and I'll sit on this until there's something better we can make available. -- Greg Smith 2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support greg(a)2ndQuadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
|
Pages: 1 Prev: out-of-scope cursor errors Next: WARNING: pgstat wait timeout |