UTF8 with BOM support in psql
in [PgSql]
|
From: Andrew Dunstan on 17 Nov 2009 12:14 Peter Eisentraut wrote: > On tis, 2009-11-17 at 00:59 -0800, Chuck McDevitt wrote: > >> Or is there a plan to read and convert the UTF-16 or UTF-32 to UTF-8, >> so psql and PostgreSQL understand it? >> (BTW, that would actually be nice on Windows, where UTF-16 is common). >> > > Well, someone could implement UTF-16 or UTF-whatever as client encoding. > But I have not heard of any concrete proposals about that. > > Doesn't the nul byte problem make that seriously hard? cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Tom Lane on 17 Nov 2009 13:01 Andrew Dunstan <andrew(a)dunslane.net> writes: > Peter Eisentraut wrote: >> Well, someone could implement UTF-16 or UTF-whatever as client encoding. >> But I have not heard of any concrete proposals about that. > Doesn't the nul byte problem make that seriously hard? Just about impossible. It would require a protocol bump, and removal of C-style string usage *everywhere* on the client side. Again, this is something that might be more feasible with encoding conversion inside psql --- translating UTF16 to UTF8 immediately upon reading it from any external file would confine the problem to possibly manageable bounds. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Andrew Dunstan on 17 Nov 2009 13:22 Tom Lane wrote: > Andrew Dunstan <andrew(a)dunslane.net> writes: > >> Peter Eisentraut wrote: >> >>> Well, someone could implement UTF-16 or UTF-whatever as client encoding. >>> But I have not heard of any concrete proposals about that. >>> > > >> Doesn't the nul byte problem make that seriously hard? >> > > Just about impossible. It would require a protocol bump, and removal of > C-style string usage *everywhere* on the client side. > > Again, this is something that might be more feasible with encoding > conversion inside psql --- translating UTF16 to UTF8 immediately upon > reading it from any external file would confine the problem to possibly > manageable bounds. > > > Well, it might be a good idea to provide at least some support in libpq. Making each client do it from scratch seems a bit inefficient. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Tom Lane on 17 Nov 2009 13:51 Andrew Dunstan <andrew(a)dunslane.net> writes: > Well, it might be a good idea to provide at least some support in libpq. > Making each client do it from scratch seems a bit inefficient. Encoding conversion seems far outside libpq's charter, and as for "from scratch" there are other libraries for that. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Itagaki Takahiro on 17 Nov 2009 22:52
Peter Eisentraut <peter_e(a)gmx.net> wrote: > Together, that should cover a lot of cases. Not perfect, but far from > useless. For Japanese users on Windows, the client encoding are always set to SJIS because of the restriction of cmd.exe. But the script file can be written in UTF8 with BOM. I don't think we should depend on client encoding. Regards, --- ITAGAKI Takahiro NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |