Prev: Create a Word document using Matlab
Next: weld detection classifier
From: yash on 21 Jul 2010 13:36
Hello,

I'm writing a matlab program that fetches lots of data from a
database. Given the structure of the tables and the nature of the
data, I'm forced to run multiple queries multiple times. For instance,
I may decide that I want to get a dozen attributes for each of 1000
stocks. Getting each attribute requires a separate DB query.

In order to get better performance, I'm using a parfor loop to loop
through the 1000 stocks and fetch data. Since I'll have multiple
parallel loops getting data, I don't want to have them all use a
single connection, and so I create a connection in the parfor loop and
close it before exiting.

As you can imagine, the repeated opening and closing of connections is
very costly and slows down the program. Ideally I'd love to leverage
some kind of connection pooling to avoid the connect/disconnect. I see
that there's no native connection pooling in Matlab and so wrote my
own class to do so.

The issue now is that I believe Matlab serializes objects when passing
data to workers using parfor. The connection object can't be
serialized. So, when the code in my parfor loop tries to request a
connection from the pool, it fails since the pool would have to be
serialized for the worker to see it, and connections can't be
serialized.

Thus, I can't use a connection in the parfor loop that was created
outside the loop, and creating connections in the loop is very costly.
Does anyone have any ideas on how I can implement some kind of
connection pooling in this scenario, or some other way to speed things
up by avoiding the frequent connect/disconnect.

FYI - the database is SQL 2008, which does not implement connection
pooling on the server side. If it did, I could leave the pooling to
SQLServer and not have this issue...

Thanks!
 | 
Pages: 1
Prev: Create a Word document using Matlab
Next: weld detection classifier