Unicode UTF-8 table formatting for psql text output
in [PgSql]
|
Prev: Hot Standby 0.2.1
Next: Rejecting weak passwords
From: Roger Leigh on 26 Oct 2009 18:58 On Mon, Oct 26, 2009 at 01:33:19PM -0400, Tom Lane wrote: > Greg Stark <gsstark(a)mit.edu> writes: > > While i agree this looks nicer I wonder what it does to things like > > excel/gnumeric/ooffice auto-recognizing table layouts and importing > > files. I'm not sure our old format was so great for this so maybe this > > is actually an improvement I'm asking for. > > Yeah. We can do what we like with the UTF8 format but I'm considerably > more worried about the aspect of making random changes to the > plain-ASCII output. On the other hand, we changed that just a release > or so ago (to put in the multiline output in the first place) and > I didn't hear complaints about it that time. I checked (using strace) gnumeric (via libgda and gnome-database-properties) openoffice (oobase) Both spreadsheets require a connection to be set up first for them to use as a handle, so I did that and traced from there. Neither made any use of psql; they both appear to use libpq via their respective database abstraction libs--no forking of any children observed. Excel is a bit tougher, I bought my first copy last week for other reasons, but I lack both windows expertise and debugging tools to trace things, and I also dual boot my computer with the postgres install on the Linux partition, making connecting to the database rather hard! I think someone else is better suited to check this one! On a related note, there's something odd with the pager code. The output of \l with the pager off: rleigh=# \l List of databases Name â Owner â Encoding â Collation â Ctype â Access privileges ââââââââââââââââââ¼âââââââââââ¼âââââââââââ¼ââââââââââââââ¼ââââââââââââââ¼âââââââââââââââââââââââ [...] (header line is 91 characters, 273 bytes) And with the pager on: rleigh=# \l List of databases Name â Owner â Encoding â Collation â Ctype â Access privileges ââââââââââââââââââ¼âââââââââââ¼âââââââââââ¼ââââââââââââââ¼ââââââââââââââ¼âââââââââââââââââ ��âââââ [...] (longest header line 85 characters, 255 bytes, 256 bytes inc. LF, remainder on second line) Note that the pager wasn't required and so wasn't actually invoked, but the output was corrupted. A newline was inserted almost at the end of the line and the continuation lacks a leading \342 which (since these UTF-8 codes are all three-byte) leads to two bytes which are invalid UTF-8. Since this spurious newline got inserted exactly on a 256 byte boundary, I was wondering if there was some buffer either internal to psql or in the termios/pty layer that was getting flushed. It also lost the first byte of the second line (possibly swapped for the \n). Another wierdness: it only happens if the terminal width is > 85 columns wide, otherwise it just wraps around as one would expect! AFAICT there are no 255/256 length buffers in the code, and the code doing the printing is just doing stdio to fout which is either stdout or a pipe! Because of this, I can't see how the spurious \n appears in the middle of a simple loop. If border=2, you'll see this for all top mid and bottom ruled lines. I do see strace showing some termios fiddling, could that be at fault or is that just readline ncurses initialisation? Regards, Roger -- .''`. Roger Leigh : :' : Debian GNU/Linux http://people.debian.org/~rleigh/ `. `' Printing on GNU/Linux? http://gutenprint.sourceforge.net/ `- GPG Public Key: 0x25BFB848 Please GPG sign your mail.
From: Tom Lane on 26 Oct 2009 19:19 Roger Leigh <rleigh(a)codelibre.net> writes: > On Mon, Oct 26, 2009 at 01:33:19PM -0400, Tom Lane wrote: >> Yeah. We can do what we like with the UTF8 format but I'm considerably >> more worried about the aspect of making random changes to the >> plain-ASCII output. > I checked (using strace) > gnumeric (via libgda and gnome-database-properties) > openoffice (oobase) Even if that were the entire universe of programs we cared about, whether their internal ODBC logic goes through psql isn't really the point here. What I'm worried about is somebody piping the text output of psql into another program. > On a related note, there's something odd with the pager code. Hm, what platform are you testing that on? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Roger Leigh on 26 Oct 2009 19:33 On Mon, Oct 26, 2009 at 07:19:24PM -0400, Tom Lane wrote: > Roger Leigh <rleigh(a)codelibre.net> writes: > > On Mon, Oct 26, 2009 at 01:33:19PM -0400, Tom Lane wrote: > >> Yeah. We can do what we like with the UTF8 format but I'm considerably > >> more worried about the aspect of making random changes to the > >> plain-ASCII output. > > > I checked (using strace) > > gnumeric (via libgda and gnome-database-properties) > > openoffice (oobase) > > Even if that were the entire universe of programs we cared about, > whether their internal ODBC logic goes through psql isn't really > the point here. What I'm worried about is somebody piping the > text output of psql into another program. > > > On a related note, there's something odd with the pager code. > > Hm, what platform are you testing that on? Debian GNU/Linux (unstable) linux 2.6.30 eglibc 2.10.1 libreadline6 6.0.5 libncurses5 5.7 gcc 4.3.4 This is the trace of the broken write: 16206 write(1, " Name \342\224\202 Owner \342\224"..., 102) = 102 16206 write(1, "\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342 \224"..., 256) = 256 16206 write(1, "\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\n", 18) = 18 I'll attach the whole thing for reference. What's clear is that the first write was *exactly* 256 bytes, which is what was requested, presumably by libc stdio buffering (which shouldn't by itself be a problem). Since we use 3-byte UTF-8 and 256/3 is 85 + 1 remainder, this is where the wierd 85 char forced newline comes from. Since it only happens when the terminal window is >85 chars, that's where I'm assuming some odd termios influence comes from (though it might just be the source of the window size and be completely innocent). The fact that libc did the two separate writes kind of rules out termios mangling the output post-write(). Regards, Roger -- .''`. Roger Leigh : :' : Debian GNU/Linux http://people.debian.org/~rleigh/ `. `' Printing on GNU/Linux? http://gutenprint.sourceforge.net/ `- GPG Public Key: 0x25BFB848 Please GPG sign your mail.
From: Peter Eisentraut on 27 Oct 2009 02:43 On Mon, 2009-10-26 at 10:12 -0700, Greg Stark wrote: > While i agree this looks nicer I wonder what it does to things like > excel/gnumeric/ooffice auto-recognizing table layouts and importing > files. I'm not sure our old format was so great for this so maybe this > is actually an improvement I'm asking for. But as long as we're > changing the format... It would at at least be good to test the > behaviour What exactly are you referring to here? -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Roger Leigh on 27 Oct 2009 07:34
On Mon, Oct 26, 2009 at 11:33:40PM +0000, Roger Leigh wrote: > On Mon, Oct 26, 2009 at 07:19:24PM -0400, Tom Lane wrote: > > Roger Leigh <rleigh(a)codelibre.net> writes: > > > On Mon, Oct 26, 2009 at 01:33:19PM -0400, Tom Lane wrote: > > >> Yeah. We can do what we like with the UTF8 format but I'm considerably > > >> more worried about the aspect of making random changes to the > > >> plain-ASCII output. > > > > > I checked (using strace) > > > gnumeric (via libgda and gnome-database-properties) > > > openoffice (oobase) > > > > Even if that were the entire universe of programs we cared about, > > whether their internal ODBC logic goes through psql isn't really > > the point here. What I'm worried about is somebody piping the > > text output of psql into another program. > > > > > On a related note, there's something odd with the pager code. > > > > Hm, what platform are you testing that on? > > Debian GNU/Linux (unstable) > linux 2.6.30 > eglibc 2.10.1 > libreadline6 6.0.5 > libncurses5 5.7 > gcc 4.3.4 > > This is the trace of the broken write: > > 16206 write(1, " Name \342\224\202 Owner \342\224"..., 102) = 102 > 16206 write(1, "\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342 > \224"..., 256) = 256 > 16206 write(1, "\224\200\342\224\200\342\224\200\342\224\200\342\224\200\342\224\200\n", 18) = 18 Further tracing showed this to be a bug in the "util-linux" version of "more" which had a static 256 byte line buffer. The above was a red herring--it's writing to a pipe. I've sent a patch to fix this by increasing the buffer size. -- .''`. Roger Leigh : :' : Debian GNU/Linux http://people.debian.org/~rleigh/ `. `' Printing on GNU/Linux? http://gutenprint.sourceforge.net/ `- GPG Public Key: 0x25BFB848 Please GPG sign your mail. |