How to handle mouse events during a long-running database transaction?
in [TCL]
|
Prev: tcl-snmptools
Next: ANNOUNCE: RamDebugger 7.0 released
From: Teemu Salmivesi on 15 Sep 2009 13:40 On Sep 15, 8:19 pm, "tom.rmadilo" <tom.rmad...(a)gmail.com> wrote: > On Sep 15, 9:54 am, Teemu Salmivesi <salmiv...(a)gmail.com> wrote: > > > > > On 15 syys, 15:04, Alexandre Ferrieux <alexandre.ferri...(a)gmail.com> > > wrote: > > > > On Sep 15, 1:30 pm, Teemu Salmivesi <salmiv...(a)gmail.com> wrote: > > > > > We have a very large application which is not planned to use threads > > > > and it is not thread safe. > > > > Then look at the contradiction in full light: either you manage to > > > isolate your blocking transaction in another thread, or you'll have to > > > do the finer-grained scheduling by hand (assuming you can split one > > > long wait into several shorter-timeout operations). No way out. > > > I agree. > > The application includes c and c++ code about 5.000.000 lines. > > That means that an another thread is not easy to implement because the > > code is not thread safe as I mentioned earlier. > > The application has TCP/IP, UDP/IP connections, local memory database > > and the GUI. > > I know that the design of the application is not the best one, > > but the code is very efficient except the huge task which will update > > the local database thru the TCP/IP connection. > > > We are planning to do scheduling by hand. Our plan is to interrupt the > > long running task (not always a db transaction) and > > handle mouse events so often that response of the GUI will be good > > enough. I know that there is a possibility to handle collapsed mouse > > events in Tcl > > but I don't know how to do it inside the TCL_DoOneEvent function. It > > seems that we are not able to get mouse motion events at all. > > Do you have any idea how to get mouse motion events from the event > > queue of Tcl? > > > - Teemu > > > > Note that isolating your database part in a separate *process*, if the > > > needed I/O bandwidth permits, will save you the hassle of making > > > anything thread-safe. It will morph the problem into one of > > > serialization, which is way simpler. > > Why not update the GUI via tcp/ip or udp? Usually a GUI is only a > snapshot of the actual state (especially if you are using a database), > so just decouple the GUI by using either a client poll or server > push. > > Or maybe what you are saying is that your application makes a local > copy of the data and then uses a long transaction to update the real > database. If that is the case, the design issue is trying to > synchronize two copies of essentially one chunk of data. That is risky > and difficult. But if that is what is happening, you could make a copy > of the data from your GUI application and then run an update from the > copy (a headless version of your application) to the database, kind of > like an auto-save feature, but it could avoid a GUI freeze (the exact > same thing happens with your typical desktop application during an > auto-save operation). The application is handling the task received from TCP/IP (socket) and updating the local database. At the same time user is drawing something eg. free draw on a canvas. Sure we have to catch all mouse motion events to show the drawing correctly. We are using Tcl event queues to update data from the socket and we are using Tcl queues to handle all mouse events. But we are not able to get mouse events if we are already inside the TCL_DoOneEvent function. It is possible to call TCL_DoOneEvent recursively, but how we are able to handle mouse motion events only? - Teemu
From: tom.rmadilo on 15 Sep 2009 15:52 On Sep 15, 10:40 am, Teemu Salmivesi <salmiv...(a)gmail.com> wrote: > On Sep 15, 8:19 pm, "tom.rmadilo" <tom.rmad...(a)gmail.com> wrote: > > > > > On Sep 15, 9:54 am, Teemu Salmivesi <salmiv...(a)gmail.com> wrote: > > > > On 15 syys, 15:04, Alexandre Ferrieux <alexandre.ferri...(a)gmail.com> > > > wrote: > > > > > On Sep 15, 1:30 pm, Teemu Salmivesi <salmiv...(a)gmail.com> wrote: > > > > > > We have a very large application which is not planned to use threads > > > > > and it is not thread safe. > > > > > Then look at the contradiction in full light: either you manage to > > > > isolate your blocking transaction in another thread, or you'll have to > > > > do the finer-grained scheduling by hand (assuming you can split one > > > > long wait into several shorter-timeout operations). No way out. > > > > I agree. > > > The application includes c and c++ code about 5.000.000 lines. > > > That means that an another thread is not easy to implement because the > > > code is not thread safe as I mentioned earlier. > > > The application has TCP/IP, UDP/IP connections, local memory database > > > and the GUI. > > > I know that the design of the application is not the best one, > > > but the code is very efficient except the huge task which will update > > > the local database thru the TCP/IP connection. > > > > We are planning to do scheduling by hand. Our plan is to interrupt the > > > long running task (not always a db transaction) and > > > handle mouse events so often that response of the GUI will be good > > > enough. I know that there is a possibility to handle collapsed mouse > > > events in Tcl > > > but I don't know how to do it inside the TCL_DoOneEvent function. It > > > seems that we are not able to get mouse motion events at all. > > > Do you have any idea how to get mouse motion events from the event > > > queue of Tcl? > > > > - Teemu > > > > > Note that isolating your database part in a separate *process*, if the > > > > needed I/O bandwidth permits, will save you the hassle of making > > > > anything thread-safe. It will morph the problem into one of > > > > serialization, which is way simpler. > > > Why not update the GUI via tcp/ip or udp? Usually a GUI is only a > > snapshot of the actual state (especially if you are using a database), > > so just decouple the GUI by using either a client poll or server > > push. > > > Or maybe what you are saying is that your application makes a local > > copy of the data and then uses a long transaction to update the real > > database. If that is the case, the design issue is trying to > > synchronize two copies of essentially one chunk of data. That is risky > > and difficult. But if that is what is happening, you could make a copy > > of the data from your GUI application and then run an update from the > > copy (a headless version of your application) to the database, kind of > > like an auto-save feature, but it could avoid a GUI freeze (the exact > > same thing happens with your typical desktop application during an > > auto-save operation). > > The application is handling the task received from TCP/IP (socket) and > updating the local database. > At the same time user is drawing something eg. free draw on a canvas. > Sure we have to catch all mouse motion events to show the drawing > correctly. > We are using Tcl event queues to update data from the socket and we > are using Tcl queues to handle all mouse events. > But we are not able to get mouse events if we are already inside the > TCL_DoOneEvent function. > It is possible to call TCL_DoOneEvent recursively, but how we are able > to handle mouse motion events only? So all I have been able to figure out so far is that you have a 5 million line application that isn't thread safe and is used to draw something on a canvas. Obviously there must be an application state, and with 5 million lines of code, the state might be pretty big. Also, for some unknown reason the application needs both udp and tcp access to the outside world (is this the database connection?). Occasionally a long running transaction with the database freezes your GUI. But what relationship, if any, do these things have to each other? What is the transaction doing? Are there multiple users with GUIs of their own? Because a one user application isn't really large just because it takes up a lot of space. If it is a multiuser application then in theory, the freeze could affect everyone, even the slow response from the database will be a huge issue for multiple users (even with a non-blocking api). I also don't understand how you got to a 5 million line application without realizing this freezing problem many moons ago. Maybe you just stuck a few existing tools together and they aren't working as expected? That's a lot of code held up by a very basic application design issue.
From: Alexandre Ferrieux on 15 Sep 2009 17:07 On Sep 15, 7:40 pm, Teemu Salmivesi <salmiv...(a)gmail.com> wrote: > > The application is handling the task received from TCP/IP (socket) and > updating the local database. > At the same time user is drawing something eg. free draw on a canvas. > Sure we have to catch all mouse motion events to show the drawing > correctly. OK so for the third time, here is how I would solve this nightmare: - Isolate the GUI in process A (wish) - Put all the rest in process B (C) - make the two communicate through sockets or pipes In addition to solving your thread safety and responsiveness problem, you'll get an extra bonus: the GUI can be developped purely in script (remember that funny language everybody is talking about on this newsgroup ;-). -Alex
From: Teemu Salmivesi on 16 Sep 2009 02:18 On 15 syys, 22:52, "tom.rmadilo" <tom.rmad...(a)gmail.com> wrote: > On Sep 15, 10:40 am, Teemu Salmivesi <salmiv...(a)gmail.com> wrote: > > > > > On Sep 15, 8:19 pm, "tom.rmadilo" <tom.rmad...(a)gmail.com> wrote: > > > > On Sep 15, 9:54 am, Teemu Salmivesi <salmiv...(a)gmail.com> wrote: > > > > > On 15 syys, 15:04, Alexandre Ferrieux <alexandre.ferri...(a)gmail.com> > > > > wrote: > > > > > > On Sep 15, 1:30 pm, Teemu Salmivesi <salmiv...(a)gmail.com> wrote: > > > > > > > We have a very large application which is not planned to use threads > > > > > > and it is not thread safe. > > > > > > Then look at the contradiction in full light: either you manage to > > > > > isolate your blocking transaction in another thread, or you'll have to > > > > > do the finer-grained scheduling by hand (assuming you can split one > > > > > long wait into several shorter-timeout operations). No way out. > > > > > I agree. > > > > The application includes c and c++ code about 5.000.000 lines. > > > > That means that an another thread is not easy to implement because the > > > > code is not thread safe as I mentioned earlier. > > > > The application has TCP/IP, UDP/IP connections, local memory database > > > > and the GUI. > > > > I know that the design of the application is not the best one, > > > > but the code is very efficient except the huge task which will update > > > > the local database thru the TCP/IP connection. > > > > > We are planning to do scheduling by hand. Our plan is to interrupt the > > > > long running task (not always a db transaction) and > > > > handle mouse events so often that response of the GUI will be good > > > > enough. I know that there is a possibility to handle collapsed mouse > > > > events in Tcl > > > > but I don't know how to do it inside the TCL_DoOneEvent function. It > > > > seems that we are not able to get mouse motion events at all. > > > > Do you have any idea how to get mouse motion events from the event > > > > queue of Tcl? > > > > > - Teemu > > > > > > Note that isolating your database part in a separate *process*, if the > > > > > needed I/O bandwidth permits, will save you the hassle of making > > > > > anything thread-safe. It will morph the problem into one of > > > > > serialization, which is way simpler. > > > > Why not update the GUI via tcp/ip or udp? Usually a GUI is only a > > > snapshot of the actual state (especially if you are using a database), > > > so just decouple the GUI by using either a client poll or server > > > push. > > > > Or maybe what you are saying is that your application makes a local > > > copy of the data and then uses a long transaction to update the real > > > database. If that is the case, the design issue is trying to > > > synchronize two copies of essentially one chunk of data. That is risky > > > and difficult. But if that is what is happening, you could make a copy > > > of the data from your GUI application and then run an update from the > > > copy (a headless version of your application) to the database, kind of > > > like an auto-save feature, but it could avoid a GUI freeze (the exact > > > same thing happens with your typical desktop application during an > > > auto-save operation). > > > The application is handling the task received from TCP/IP (socket) and > > updating the local database. > > At the same time user is drawing something eg. free draw on a canvas. > > Sure we have to catch all mouse motion events to show the drawing > > correctly. > > We are using Tcl event queues to update data from the socket and we > > are using Tcl queues to handle all mouse events. > > But we are not able to get mouse events if we are already inside the > > TCL_DoOneEvent function. > > It is possible to call TCL_DoOneEvent recursively, but how we are able > > to handle mouse motion events only? > > So all I have been able to figure out so far is that you have a 5 > million line application that isn't thread safe and is used to draw > something on a canvas. Obviously there must be an application state, > and with 5 million lines of code, the state might be pretty big. Also, > for some unknown reason the application needs both udp and tcp access > to the outside world (is this the database connection?). Occasionally > a long running transaction with the database freezes your GUI. > > But what relationship, if any, do these things have to each other? > What is the transaction doing? Are there multiple users with GUIs of > their own? Because a one user application isn't really large just > because it takes up a lot of space. If it is a multiuser application > then in theory, the freeze could affect everyone, even the slow > response from the database will be a huge issue for multiple users > (even with a non-blocking api). > > I also don't understand how you got to a 5 million line application > without realizing this freezing problem many moons ago. Maybe you > just stuck a few existing tools together and they aren't working as > expected? That's a lot of code held up by a very basic application > design issue. The application is for a one user only. The way we use Tcl/Tk is described here http://wiki.tcl.tk/1527. We are handling events in callback functions. During a long running handling of data (data is from TCP/IP socket and callback is triggered by Tcl to handle the data) we would like to (have to) handle mouse motion event from Tcl queue. One possibility is to call TCL_DoOneEvent recursively but for some reason we are not getting mouse motion event callbacks from Tcl even the X Window is generating mouse motion events. - Teemu
From: tom.rmadilo on 16 Sep 2009 11:08
On Sep 15, 11:18 pm, Teemu Salmivesi <salmiv...(a)gmail.com> wrote: > On 15 syys, 22:52, "tom.rmadilo" <tom.rmad...(a)gmail.com> wrote: > > > > > On Sep 15, 10:40 am, Teemu Salmivesi <salmiv...(a)gmail.com> wrote: > > > > On Sep 15, 8:19 pm, "tom.rmadilo" <tom.rmad...(a)gmail.com> wrote: > > > > > On Sep 15, 9:54 am, Teemu Salmivesi <salmiv...(a)gmail.com> wrote: > > > > > > On 15 syys, 15:04, Alexandre Ferrieux <alexandre.ferri...(a)gmail.com> > > > > > wrote: > > > > > > > On Sep 15, 1:30 pm, Teemu Salmivesi <salmiv...(a)gmail.com> wrote: > > > > > > > > We have a very large application which is not planned to use threads > > > > > > > and it is not thread safe. > > > > > > > Then look at the contradiction in full light: either you manage to > > > > > > isolate your blocking transaction in another thread, or you'll have to > > > > > > do the finer-grained scheduling by hand (assuming you can split one > > > > > > long wait into several shorter-timeout operations). No way out. > > > > > > I agree. > > > > > The application includes c and c++ code about 5.000.000 lines. > > > > > That means that an another thread is not easy to implement because the > > > > > code is not thread safe as I mentioned earlier. > > > > > The application has TCP/IP, UDP/IP connections, local memory database > > > > > and the GUI. > > > > > I know that the design of the application is not the best one, > > > > > but the code is very efficient except the huge task which will update > > > > > the local database thru the TCP/IP connection. > > > > > > We are planning to do scheduling by hand. Our plan is to interrupt the > > > > > long running task (not always a db transaction) and > > > > > handle mouse events so often that response of the GUI will be good > > > > > enough. I know that there is a possibility to handle collapsed mouse > > > > > events in Tcl > > > > > but I don't know how to do it inside the TCL_DoOneEvent function. It > > > > > seems that we are not able to get mouse motion events at all. > > > > > Do you have any idea how to get mouse motion events from the event > > > > > queue of Tcl? > > > > > > - Teemu > > > > > > > Note that isolating your database part in a separate *process*, if the > > > > > > needed I/O bandwidth permits, will save you the hassle of making > > > > > > anything thread-safe. It will morph the problem into one of > > > > > > serialization, which is way simpler. > > > > > Why not update the GUI via tcp/ip or udp? Usually a GUI is only a > > > > snapshot of the actual state (especially if you are using a database), > > > > so just decouple the GUI by using either a client poll or server > > > > push. > > > > > Or maybe what you are saying is that your application makes a local > > > > copy of the data and then uses a long transaction to update the real > > > > database. If that is the case, the design issue is trying to > > > > synchronize two copies of essentially one chunk of data. That is risky > > > > and difficult. But if that is what is happening, you could make a copy > > > > of the data from your GUI application and then run an update from the > > > > copy (a headless version of your application) to the database, kind of > > > > like an auto-save feature, but it could avoid a GUI freeze (the exact > > > > same thing happens with your typical desktop application during an > > > > auto-save operation). > > > > The application is handling the task received from TCP/IP (socket) and > > > updating the local database. > > > At the same time user is drawing something eg. free draw on a canvas. > > > Sure we have to catch all mouse motion events to show the drawing > > > correctly. > > > We are using Tcl event queues to update data from the socket and we > > > are using Tcl queues to handle all mouse events. > > > But we are not able to get mouse events if we are already inside the > > > TCL_DoOneEvent function. > > > It is possible to call TCL_DoOneEvent recursively, but how we are able > > > to handle mouse motion events only? > > > So all I have been able to figure out so far is that you have a 5 > > million line application that isn't thread safe and is used to draw > > something on a canvas. Obviously there must be an application state, > > and with 5 million lines of code, the state might be pretty big. Also, > > for some unknown reason the application needs both udp and tcp access > > to the outside world (is this the database connection?). Occasionally > > a long running transaction with the database freezes your GUI. > > > But what relationship, if any, do these things have to each other? > > What is the transaction doing? Are there multiple users with GUIs of > > their own? Because a one user application isn't really large just > > because it takes up a lot of space. If it is a multiuser application > > then in theory, the freeze could affect everyone, even the slow > > response from the database will be a huge issue for multiple users > > (even with a non-blocking api). > > > I also don't understand how you got to a 5 million line application > > without realizing this freezing problem many moons ago. Maybe you > > just stuck a few existing tools together and they aren't working as > > expected? That's a lot of code held up by a very basic application > > design issue. > > The application is for a one user only. > The way we use Tcl/Tk is described herehttp://wiki.tcl.tk/1527. > We are handling events in callback functions. During a long running > handling of data (data is from TCP/IP socket and callback is triggered > by Tcl to handle the data) we would like to (have to) handle mouse > motion event from Tcl queue. One possibility is to call TCL_DoOneEvent > recursively but for some reason we are not getting mouse motion event > callbacks from Tcl even the X Window is generating mouse motion > events. Okay, so basically you have a tiny single user application. Follow Alexandre Ferrieux's advice and save yourself (and everyone else working on this project) lots of grief. You don't need to keep explaining what you "want to do" because the basics of it is that you have a long running blocking event which screws up your GUI responsiveness. Have you ever used a web browser when it decides to update your page, replacing the article you were only half way finished reading with something you have no interest in? Have you experienced this delay? Or when MS Word auto-saves? Nobody has solved the problem you are talking about (very well) in a single process because the development model allows GUI events to compete with TCP/IP (or file dump) events and they don't mix very well. You aren't going to solve this problem with TCL_DoOneEvent. |