pg_read_file() and non-ascii input file
in [PgSql]
|
Prev: update_process_title=off and logger, wal, ...processes
Next: [HACKERS] More frame options in window functions
From: Robert Haas on 30 Dec 2009 16:13 On Mon, Nov 30, 2009 at 4:36 AM, Itagaki Takahiro <itagaki.takahiro(a)oss.ntt.co.jp> wrote: > If we want to keep backward compatibility, the issue can be fixed > by adding pg_verifymbstr() to the function. We can also have the > binary version in another name, like pg_read_binary_file(). I don't feel good about changing the return type of an existing function, so I guess +1 from me on the approach quoted above. ....Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Takahiro Itagaki on 3 Jan 2010 21:10 Robert Haas <robertmhaas(a)gmail.com> wrote: > > If we want to keep backward compatibility, the issue can be fixed > > by adding pg_verifymbstr() to the function. > > I don't feel good about changing the return type of an existing > function, so I guess +1 from me on the approach quoted above. Ok, I just added pg_verifymbstr() instead of changing the result type. I didn't add any additinal file reading functions in the patch, but I'm willing to add them if someone want them: - pg_read_file_with_encoding() - pg_read_binary_file() RETURNS bytea - pg_read_text_file() RETURNS SETOF text -- returns set of lines One thing bothering me is the HINT message on error is just pointless; The encoding is controlled by "server_encoding" here. We will have the same error message in server-side COPY commands. We'd better improving the message, though it should be done by another patch. =# SELECT pg_read_file('invalid.txt', 0, (pg_stat_file('invalid.txt')).size); ERROR: invalid byte sequence for encoding "UTF8": 0x93 HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding". =# COPY tbl FROM 'invalid.txt'; -- server-side copy from a local file. (the same message -- but the encoding should match with "server_encoding") Regards, --- Takahiro Itagaki NTT Open Source Software Center
From: Robert Haas on 3 Jan 2010 22:36
On Sun, Jan 3, 2010 at 9:10 PM, Takahiro Itagaki <itagaki.takahiro(a)oss.ntt.co.jp> wrote: > > Robert Haas <robertmhaas(a)gmail.com> wrote: > >> > If we want to keep backward compatibility, the issue can be fixed >> > by adding pg_verifymbstr() to the function. >> >> I don't feel good about changing the return type of an existing >> function, so I guess +1 from me on the approach quoted above. > > Ok, I just added pg_verifymbstr() instead of changing the result type. Sounds fine. > I didn't add any additinal file reading functions in the patch, but > I'm willing to add them if someone want them: > - pg_read_file_with_encoding() > - pg_read_binary_file() RETURNS bytea > - pg_read_text_file() RETURNS SETOF text -- returns set of lines OK. > One thing bothering me is the HINT message on error is just pointless; > The encoding is controlled by "server_encoding" here. We will have the > same error message in server-side COPY commands. We'd better improving > the message, though it should be done by another patch. Interestingly, this same issue is being discussed on the thread entitled "invalid UTF-8 via pl/perl". I suppose whatever solution is adopted there can be used here as well. FWIW, repurposing the third argument of pg_verifymbstr seems like the most sensible option to me. ....Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |