Prev: [HACKERS] how to use advanced gist options
Next: [HACKERS] WIP: simple allocator
From: Tom Lane on 15 Mar 2010 22:02
Takahiro Itagaki <itagaki.takahiro(a)oss.ntt.co.jp> writes:
> Greg Smith <greg(a)2ndquadrant.com> wrote:
>> It looks like the switch between clients running on separate workers can
>> lead to a mix of their respective lines showing up though.

> Oops. There might be two solutions for the issue:
> 1. Use explicit locks. The lock primitive will be pthread_mutex for
> multi-threaded implementations or semaphore for multi-threaded ones.
> 2. Use per-thread log files.
> File names would be "pgbench_log.<main-process-id>.<thread-id>".

I think #1 is out of the question, as the synchronization overhead will
do serious damage to the whole point of having a multithreaded pgbench.
#2 might be a reasonable idea.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

From: Takahiro Itagaki on 15 Mar 2010 22:13

Tom Lane <tgl(a)sss.pgh.pa.us> wrote:

> Takahiro Itagaki <itagaki.takahiro(a)oss.ntt.co.jp> writes:
> > Oops. There might be two solutions for the issue:
> > 1. Use explicit locks. The lock primitive will be pthread_mutex for
> > multi-threaded implementations or semaphore for multi-threaded ones.
> > 2. Use per-thread log files.
> > File names would be "pgbench_log.<main-process-id>.<thread-id>".
>
> I think #1 is out of the question, as the synchronization overhead will
> do serious damage to the whole point of having a multithreaded pgbench.
> #2 might be a reasonable idea.

Ok, I'll go for #2.

Regards,
---
Takahiro Itagaki
NTT Open Source Software Center



--
Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

From: Takahiro Itagaki on 15 Mar 2010 23:09

Takahiro Itagaki <itagaki.takahiro(a)oss.ntt.co.jp> wrote:

> > > 2. Use per-thread log files.
> > > File names would be "pgbench_log.<main-process-id>.<thread-id>".

Here is a patch to implement per-thread log files for pgbench -l.

The log filenames are "pgbench_log.<main-process-id>.<thread-serial-number>"
for each thread, but the first thread (including single-threaded) still uses
"pgbench_log.<main-process-id>" for the name because of compatibility.

Example:
$ pgbench -c16 -j4 -l
$ ls
pgbench_log.2196 pgbench_log.2196.1 pgbench_log.2196.2 pgbench_log.2196.3

Comments and suggenstions welcome.

Regards,
---
Takahiro Itagaki
NTT Open Source Software Center

From: Greg Smith on 18 Mar 2010 14:49
Takahiro Itagaki wrote:
> The log filenames are "pgbench_log.<main-process-id>.<thread-serial-number>"
> for each thread, but the first thread (including single-threaded) still uses
> "pgbench_log.<main-process-id>" for the name because of compatibility.
>

Attached is an updated version that I think is ready to commit. Only
changes are docs--I rewrote those to improve the wording some. The code
looked and tested fine to me. I just added support for the new format
to pgbench-tools and am back to happily running large batches of tests
using it again.

I confirmed a few things:

-On my CentOS system, the original problem is masked if you have
"--enable-thread-safety" on; the multi-threaded output shows up without
any broken lines into the single file. As I suspected it's only the
multi-process implementation that shows the issue here. Since Tom
points out that's luck rather than something that should be relied upon,
I don't think that actually changes what to do here, it just explains
why this wasn't obvious in earlier testing--normally I have thread
safety on nowadays.

-Patch corrects the problem. I took a build without thread safety on,
demonstrated the issue with its pgbench. Apply the patch, rebuild just
pgbench, run again; new multiple log files have no issue.

-It's easy to convert existing scripts to utilize the new multiple log
format. Right now the current idiom you're forced into using when
running pgbench scripts is to track the PID it's run as, then use
something like:

mv pgbench_log.${PID} pgbench.log

To convert to a stable filename for later processing. Now you just use
something like this instead:

cat pgbench_log.${PID}* > pgbench.log
rm -f pgbench_log.${PID}*

And that works fine.

--
Greg Smith 2ndQuadrant US Baltimore, MD
PostgreSQL Training, Services and Support
greg(a)2ndQuadrant.com www.2ndQuadrant.us

From: Takahiro Itagaki on 22 Mar 2010 21:38

Greg Smith <greg(a)2ndquadrant.com> wrote:

> Attached is an updated version that I think is ready to commit. Only
> changes are docs--I rewrote those to improve the wording some.

Thanks for the correction. Applied.

Regards,
---
Takahiro Itagaki
NTT Open Source Software Center



--
Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

First  |  Prev  |  Next  |  Last
Pages: 1 2 3
Prev: [HACKERS] how to use advanced gist options
Next: [HACKERS] WIP: simple allocator