From: Fujii Masao on
On Tue, Dec 22, 2009 at 8:49 PM, Heikki Linnakangas
<heikki.linnakangas(a)enterprisedb.com> wrote:
>> Umm.., I still cannot find the place where the walreceiver module is
>> loaded by using load_external_function() in your 'replication' branch.
>> Also the compilation of that branch fails. Is the 'pushed' branch the
>> latest? Sorry if I'm missing something.
>
> Ah, I see. The changes were not included in the merge commit after all,
> but I had simple forgot to "git add" them. Sorry about that, should be
> there now.

This change which moves walreceiver process into a dynamically loaded
module caused the following compile error on my MinGW environment.

---------------------------
gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith
-Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing
-fwrapv -g -I. -I../../../../src/interfaces/libpq
-I../../../../src/include -I./src/include/port/win32 -DEXEC_BACKEND
"-I../../../../src/include/port/win32" -DBUILDING_DLL -c -o
walreceiverproc.o walreceiverproc.c
dlltool --export-all --output-def libwalreceiverprocdll.def walreceiverproc.o
dllwrap -o walreceiverproc.dll --dllname walreceiverproc.dll --def
libwalreceiverprocdll.def walreceiverproc.o -L../../../../src/backend
-lpostgres -L../../../../src/interfaces/libpq -L../../../../src/port
-lpq
Info: resolving _pg_signal_mask by linking to __imp__pg_signal_mask
(auto-import)
Info: resolving _pg_signal_queue by linking to __imp__pg_signal_queue
(auto-import)
Info: resolving _InterruptPending by linking to
__imp__InterruptPending (auto-import)
Info: resolving _assert_enabled by linking to __imp__assert_enabled
(auto-import)
Info: resolving _WalRcv by linking to __imp__WalRcv (auto-import)
Info: resolving _proc_exit_inprogress by linking to
__imp__proc_exit_inprogress (auto-import)
Info: resolving _BlockSig by linking to __imp__BlockSig (auto-import)
Info: resolving _sync_method by linking to __imp__sync_method (auto-import)
Info: resolving _MyProcPid by linking to __imp__MyProcPid (auto-import)
Info: resolving _CurrentResourceOwner by linking to
__imp__CurrentResourceOwner (auto-import)
Info: resolving _TopMemoryContext by linking to
__imp__TopMemoryContext (auto-import)
Info: resolving _CurrentMemoryContext by linking to
__imp__CurrentMemoryContext (auto-import)
Info: resolving _PG_exception_stack by linking to
__imp__PG_exception_stack (auto-import)
Info: resolving _UnBlockSig by linking to __imp__UnBlockSig (auto-import)
Info: resolving _ThisTimeLineID by linking to __imp__ThisTimeLineID
(auto-import)
Info: resolving _error_context_stack by linking to
__imp__error_context_stack (auto-import)
Info: resolving _InterruptHoldoffCount by linking to
__imp__InterruptHoldoffCount (auto-import)
c:\MinGW\bin\..\lib\gcc\mingw32\3.4.2\..\..\..\..\mingw32\bin\ld.exe:
warning: auto-importing has been activated without
--enable-auto-import specified on the command line.
This should work unless it involves constant data structures
referencing symbols from auto-imported DLLs.
fu000001.o:(.idata$2+0xc): undefined reference to `libpostgres_a_iname'
fu000003.o:(.idata$2+0xc): undefined reference to `libpostgres_a_iname'
fu000005.o:(.idata$2+0xc): undefined reference to `libpostgres_a_iname'
fu000006.o:(.idata$2+0xc): undefined reference to `libpostgres_a_iname'
fu000008.o:(.idata$2+0xc): undefined reference to `libpostgres_a_iname'
fu000009.o:(.idata$2+0xc): more undefined references to
`libpostgres_a_iname' follow
nmth000000.o:(.idata$4+0x0): undefined reference to `_nm__pg_signal_mask'
nmth000002.o:(.idata$4+0x0): undefined reference to `_nm__pg_signal_queue'
nmth000004.o:(.idata$4+0x0): undefined reference to `_nm__InterruptPending'
nmth000007.o:(.idata$4+0x0): undefined reference to `_nm__assert_enabled'
nmth000012.o:(.idata$4+0x0): undefined reference to `_nm__WalRcv'
nmth000018.o:(.idata$4+0x0): undefined reference to `_nm__proc_exit_inprogress'
nmth000020.o:(.idata$4+0x0): undefined reference to `_nm__BlockSig'
nmth000023.o:(.idata$4+0x0): undefined reference to `_nm__sync_method'
nmth000026.o:(.idata$4+0x0): undefined reference to `_nm__MyProcPid'
nmth000028.o:(.idata$4+0x0): undefined reference to `_nm__CurrentResourceOwner'
nmth000030.o:(.idata$4+0x0): undefined reference to `_nm__TopMemoryContext'
nmth000032.o:(.idata$4+0x0): undefined reference to `_nm__CurrentMemoryContext'
nmth000035.o:(.idata$4+0x0): undefined reference to `_nm__PG_exception_stack'
nmth000037.o:(.idata$4+0x0): undefined reference to `_nm__UnBlockSig'
nmth000039.o:(.idata$4+0x0): undefined reference to `_nm__ThisTimeLineID'
nmth000041.o:(.idata$4+0x0): undefined reference to `_nm__error_context_stack'
nmth000043.o:(.idata$4+0x0): undefined reference to `_nm__InterruptHoldoffCount'
collect2: ld returned 1 exit status
c:\MinGW\bin\dllwrap.exe: c:\MinGW\bin\gcc exited with status 1
make[2]: *** [walreceiverproc.dll] Error 1
make[2]: Leaving directory
`/c/postgres/mmm/src/backend/postmaster/walreceiverproc'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/c/postgres/mmm/src'
make: *** [all] Error 2
---------------------------

Though I marked the variables shown in the above message as PGDLLIMPORT,
the "make" still fails in the same way. I struggled with this issue
for some time, but
could not fix it yet :(

Frankly I'm not familiar with that area. So it would be nice if
someone could analyze
this issue.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

--
Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

From: Magnus Hagander on
On Tue, Jan 12, 2010 at 17:58, Fujii Masao <masao.fujii(a)gmail.com> wrote:
> On Tue, Dec 22, 2009 at 8:49 PM, Heikki Linnakangas
> <heikki.linnakangas(a)enterprisedb.com> wrote:
>>> Umm.., I still cannot find the place where the walreceiver module is
>>> loaded by using load_external_function() in your 'replication' branch.
>>> Also the compilation of that branch fails. Is the 'pushed' branch the
>>> latest? Sorry if I'm missing something.
>>
>> Ah, I see. The changes were not included in the merge commit after all,
>> but I had simple forgot to "git add" them. Sorry about that, should be
>> there now.
>
> This change which moves walreceiver process into a dynamically loaded
> module caused the following compile error on my MinGW environment.

That sounds strange - it should pick those up from the -lpostgres. Any
chance you have an old postgres binary around from a non-syncrep build
or something?


> ---------------------------
>
> Though I marked the variables shown in the above message as PGDLLIMPORT,
> the "make" still fails in the same way. I struggled with this issue
> for some time, but
> could not fix it yet :(
>
> Frankly I'm not familiar with that area. So it would be nice if
> someone could analyze
> this issue.

Do you have an environment to try to build it under msvc? in my
experience, that gives you easier-to-understand error messages in a
lot of cases like this - it removets the mingw black magic.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

--
Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

From: Fujii Masao on
Thanks for your advice!

On Wed, Jan 13, 2010 at 3:37 AM, Magnus Hagander <magnus(a)hagander.net> wrote:
>> This change which moves walreceiver process into a dynamically loaded
>> module caused the following compile error on my MinGW environment.
>
> That sounds strange - it should pick those up from the -lpostgres. Any
> chance you have an old postgres binary around from a non-syncrep build
> or something?

No, there is no old postgres binary.

> Do you have an environment to try to build it under msvc?

No, unfortunately.

> in my
> experience, that gives you easier-to-understand error messages in a
> lot of cases like this - it removets the mingw black magic.

OK. I'll try to build it under msvc.

But since there seems to be a long way to go before doing that,
I would appreciate if someone could give me some advice.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

--
Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

From: Heikki Linnakangas on
Fujii Masao wrote:
> Done. Currently there is no new libpq function for replication. The
> walreceiver uses only existing functions like PQconnectdb, PQexec,
> PQgetCopyData, etc.
>
> git://git.postgresql.org/git/users/fujii/postgres.git
> branch: replication

Thanks!

I'm afraid we haven't quite nailed the select/poll issue yet. You copied
pq_wait() from the libpq pqSocketCheck(), but there's one big difference
between the backend and the frontend: the frontend always puts the
connection to non-blocking mode, while the backend uses blocking mode.
At least with SSL, I think it's possible for pq_wait() to return false
positives, if the SSL layer decides to renegotiate the connection
causing data to flow in the other direction in the underlying TCP
connection. A false positive would lead cause walsender to block
indefinitely on the pq_getbyte() call.

I don't even want to think about the changes required to put the backend
socket to non-blocking mode, I don't know that code well enough. Maybe
we could temporarily put it to non-blocking mode, read to see if there's
any data available, and put it back to blocking mode. But even then I
think we'd need to modify at least secure_read() to work correctly with
SSL in non-blocking mode.

Another idea is to use poll() to check for POLLHUP, on those platforms
that have poll(). AFAICS there is no equivalent for that in select(), so
for platforms that don't have poll() we would have to simply ignore the
issue or write some other platform-specific work-around (Windows
WSAEventSelect() seems to have a FD_CLOSE event for that). That would be
a quite localized change.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

From: Fujii Masao on
On Wed, Jan 13, 2010 at 7:27 PM, Heikki Linnakangas
<heikki.linnakangas(a)enterprisedb.com> wrote:
> the frontend always puts the
> connection to non-blocking mode, while the backend uses blocking mode.

Really? By default (i.e., without the expressly setting by using
PQsetnonblocking()), the connection is set to blocking mode even
in frontend. Am I missing something?

> At least with SSL, I think it's possible for pq_wait() to return false
> positives, if the SSL layer decides to renegotiate the connection
> causing data to flow in the other direction in the underlying TCP
> connection. A false positive would lead cause walsender to block
> indefinitely on the pq_getbyte() call.

Sorry. I could not understand that issue scenario. Could you explain
it in more detail?

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

--
Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers