Prev: Buy Genuine Google Adsense Account only for Rs.200/- for indian people.
Next: Getting Windows ACLs in Java
From: Ken on 2 Apr 2010 16:47
On Apr 1, 3:01 pm, Jim Janney <jjan...(a)shell.xmission.com> wrote:
> Arne Vajhøj <a...(a)vajhoej.dk> writes:
> > On 01-04-2010 16:08, Jim Janney wrote:
> >> Arne Vajhøj<a...(a)vajhoej.dk>  writes:
> >>> On 01-04-2010 15:11, Jim Janney wrote:
> >>>> I need to maintain a data base of small text snippets keyed by
> >>>> arbitrary strings, without the overhead of a full SQL relational
> >>>> database.  We will have several people putting data into it so it
> >>>> needs to support concurrent access over a network.
>
> >>>> JDBM catches my eye (http://jdbm.sourceforge.net/) but I can't find
> >>>> any place that says whether it supports concurrent access or not.
> >>>> Berkeley DB would be perfect but would need a license from Oracle
> >>>> since this is a commercial product.  Gdbm apparently does not support
> >>>> concurrent access.  TDB (http://sourceforge.net/projects/tdb/) looks
> >>>> good but would require a JNI interface and probably some work to get
> >>>> it to compile under Windows.
>
> >>>> Any recommendations?  I can do JNI if neccessary but would be just as
> >>>> happy not to need to.  I need something I can get going quickly --
> >>>> management wants it yesterday, as usual...
>
> >>> Why are you so sure that a full SQL relational database has
> >>> too much overhead?
>
> >> Five years of experience using a relational database in the
> >> application for which this is intended.  They're great for some
> >> things, and we use it for those things.  This is not one of them.
>
> > And the simple solution of buying some more powerful
> > database hardware is not an option?
>
> Tell all of our customers they have to upgrade their network and buy a
> faster AS/400?  No, that's not going to fly.
>

You have an AS/400... I don't understand what the issue is... are you
trying to replace the AS/400?

If the AS/400 is fast enough and for some reason think that SQL is too
slow, if it's an old dieing 400 I could see that. But if you only
have a few hundred users and it's a half way modern iSeries I couldn't
imagine them being able to stress it with SQL, unless they're day
traders!!!

Anyways if the AS/400 is fast enough what is wrong with its record
level access? I've never tried to use Java for record level access
(but I very much bet it's possible via JTOpen) I've only it in RPG (if
you don't program in RPG see SETLL and CHAIN) and even then I'd rather
use sql than doing record level. The problem with it is if you
support more than one client. Different clients are going to have
different data needs and so an optimal set of CHAINs for one customer
is not going to be the same for another and if this simple group of
strings ever has a dependency with data elsewhere... it's just so much
harder to manage than simply using transactions.

I haven't done much research on Express-C (isn't it free?), it might
have the same functionality and you'd probably be more comfortable
with a flavor of DB2.

>
> >>> They work. They are widely used. They are optimized a lot.
> >>> They support ACID.
>
> >>> Chosing some rarely used ISAM package, that may have
> >>> concurrency problems&  other bugs and may not even be optimized
> >>> very well seems as a high risk for little gain to me.
>
> >>> Sure Google, Yahoo, Facebook etc. uses NOSQL databases
> >>> today. But they can also afford to spend 2 digit millions
> >>> of dollars on the development without blinking. Can you?
>
> >> If I had time to debate this I wouldn't have asked the question.  I
> >> was hoping someone might have a quick, constructive suggestion.
>
> > The fact that using a relational database does not perform
> > well does not guarantee that a non-relational database will
> > perform well.
>
> > That is just wishful thinking.
>
> > If you ask questions for free in a public forum, then
> > you should be prepared for suggestions about other
> > approaches than the one you prefer.
>
> In the abstract it is undoubtably an interesting technical question.
> I don't have time for it now.
>
> --
> Jim Janney

From: Jim Janney on 2 Apr 2010 17:47
Patrick May <patrick(a)softwarematters.org> writes:

> Jim Janney <jjanney(a)shell.xmission.com> writes:
>> I need to maintain a data base of small text snippets keyed by
>> arbitrary strings, without the overhead of a full SQL relational
>> database. We will have several people putting data into it so it
>> needs to support concurrent access over a network.
> [ . . . ]
>> Berkeley DB would be perfect but would need a license from Oracle
>> since this is a commercial product.
>
> BDB was my first thought when reading your requirements. It's fast
> and trustworthy. How expensive is the commercial license? I suspect
> it's worth the amount of risk it eliminates.

Their web page doesn't say. Apparently you tell them what you plan to
do with it and they offer you a price.

--
Jim Janney
From: Jim Janney on 2 Apr 2010 18:23
Ken <ken(a)kenmcwilliams.com> writes:

> On Apr 1, 3:01 pm, Jim Janney <jjan...(a)shell.xmission.com> wrote:
>> Arne Vajhøj <a...(a)vajhoej.dk> writes:
>> > On 01-04-2010 16:08, Jim Janney wrote:
>> >> Arne Vajhøj<a...(a)vajhoej.dk>  writes:
>> >>> On 01-04-2010 15:11, Jim Janney wrote:
>> >>>> I need to maintain a data base of small text snippets keyed by
>> >>>> arbitrary strings, without the overhead of a full SQL relational
>> >>>> database.  We will have several people putting data into it so it
>> >>>> needs to support concurrent access over a network.
>>
>> >>>> JDBM catches my eye (http://jdbm.sourceforge.net/) but I can't find
>> >>>> any place that says whether it supports concurrent access or not.
>> >>>> Berkeley DB would be perfect but would need a license from Oracle
>> >>>> since this is a commercial product.  Gdbm apparently does not support
>> >>>> concurrent access.  TDB (http://sourceforge.net/projects/tdb/) looks
>> >>>> good but would require a JNI interface and probably some work to get
>> >>>> it to compile under Windows.
>>
>> >>>> Any recommendations?  I can do JNI if neccessary but would be just as
>> >>>> happy not to need to.  I need something I can get going quickly --
>> >>>> management wants it yesterday, as usual...
>>
>> >>> Why are you so sure that a full SQL relational database has
>> >>> too much overhead?
>>
>> >> Five years of experience using a relational database in the
>> >> application for which this is intended.  They're great for some
>> >> things, and we use it for those things.  This is not one of them.
>>
>> > And the simple solution of buying some more powerful
>> > database hardware is not an option?
>>
>> Tell all of our customers they have to upgrade their network and buy a
>> faster AS/400?  No, that's not going to fly.
>>
>
> You have an AS/400... I don't understand what the issue is... are you
> trying to replace the AS/400?
>
> If the AS/400 is fast enough and for some reason think that SQL is too
> slow, if it's an old dieing 400 I could see that. But if you only
> have a few hundred users and it's a half way modern iSeries I couldn't
> imagine them being able to stress it with SQL, unless they're day
> traders!!!
>
> Anyways if the AS/400 is fast enough what is wrong with its record
> level access? I've never tried to use Java for record level access
> (but I very much bet it's possible via JTOpen) I've only it in RPG (if
> you don't program in RPG see SETLL and CHAIN) and even then I'd rather
> use sql than doing record level. The problem with it is if you
> support more than one client. Different clients are going to have
> different data needs and so an optimal set of CHAINs for one customer
> is not going to be the same for another and if this simple group of
> strings ever has a dependency with data elsewhere... it's just so much
> harder to manage than simply using transactions.
>
> I haven't done much research on Express-C (isn't it free?), it might
> have the same functionality and you'd probably be more comfortable
> with a flavor of DB2.

Our customers have AS/400s, some of them dating back to 2001. Either
SQL or record level access would be more than fast enough if the code
ran directly on the 400, but when you connect over a network the
latency goes way up, and I wanted to be able to do a bunch of
unrelated queries (roughly 10 to 200) with no perceptible delay to the
user (this is part of displaying a screen).

I'm beginning to think that the same problem would apply to anything I
do over a network, so I'm currently rethinking my whole approach to
the problem. The data is small enough I can cache it all locally, but
I need to reload it when it changes.

--
Jim Janney


From: Robert Klemme on 3 Apr 2010 06:07
On 04/03/2010 12:23 AM, Jim Janney wrote:
> Ken <ken(a)kenmcwilliams.com> writes:
>
>> On Apr 1, 3:01 pm, Jim Janney <jjan...(a)shell.xmission.com> wrote:
>>> Arne Vajhøj <a...(a)vajhoej.dk> writes:
>>>> On 01-04-2010 16:08, Jim Janney wrote:
>>>>> Arne Vajhøj<a...(a)vajhoej.dk> writes:
>>>>>> On 01-04-2010 15:11, Jim Janney wrote:
>>>>>>> I need to maintain a data base of small text snippets keyed by
>>>>>>> arbitrary strings, without the overhead of a full SQL relational
>>>>>>> database. We will have several people putting data into it so it
>>>>>>> needs to support concurrent access over a network.
>>>>>>> JDBM catches my eye (http://jdbm.sourceforge.net/) but I can't find
>>>>>>> any place that says whether it supports concurrent access or not.
>>>>>>> Berkeley DB would be perfect but would need a license from Oracle
>>>>>>> since this is a commercial product. Gdbm apparently does not support
>>>>>>> concurrent access. TDB (http://sourceforge.net/projects/tdb/) looks
>>>>>>> good but would require a JNI interface and probably some work to get
>>>>>>> it to compile under Windows.
>>>>>>> Any recommendations? I can do JNI if neccessary but would be just as
>>>>>>> happy not to need to. I need something I can get going quickly --
>>>>>>> management wants it yesterday, as usual...
>>>>>> Why are you so sure that a full SQL relational database has
>>>>>> too much overhead?
>>>>> Five years of experience using a relational database in the
>>>>> application for which this is intended. They're great for some
>>>>> things, and we use it for those things. This is not one of them.
>>>> And the simple solution of buying some more powerful
>>>> database hardware is not an option?
>>> Tell all of our customers they have to upgrade their network and buy a
>>> faster AS/400? No, that's not going to fly.
>>>
>> You have an AS/400... I don't understand what the issue is... are you
>> trying to replace the AS/400?
>>
>> If the AS/400 is fast enough and for some reason think that SQL is too
>> slow, if it's an old dieing 400 I could see that. But if you only
>> have a few hundred users and it's a half way modern iSeries I couldn't
>> imagine them being able to stress it with SQL, unless they're day
>> traders!!!
>>
>> Anyways if the AS/400 is fast enough what is wrong with its record
>> level access? I've never tried to use Java for record level access
>> (but I very much bet it's possible via JTOpen) I've only it in RPG (if
>> you don't program in RPG see SETLL and CHAIN) and even then I'd rather
>> use sql than doing record level. The problem with it is if you
>> support more than one client. Different clients are going to have
>> different data needs and so an optimal set of CHAINs for one customer
>> is not going to be the same for another and if this simple group of
>> strings ever has a dependency with data elsewhere... it's just so much
>> harder to manage than simply using transactions.
>>
>> I haven't done much research on Express-C (isn't it free?), it might
>> have the same functionality and you'd probably be more comfortable
>> with a flavor of DB2.
>
> Our customers have AS/400s, some of them dating back to 2001. Either
> SQL or record level access would be more than fast enough if the code
> ran directly on the 400, but when you connect over a network the
> latency goes way up, and I wanted to be able to do a bunch of
> unrelated queries (roughly 10 to 200) with no perceptible delay to the
> user (this is part of displaying a screen).
>
> I'm beginning to think that the same problem would apply to anything I
> do over a network, so I'm currently rethinking my whole approach to
> the problem. The data is small enough I can cache it all locally, but
> I need to reload it when it changes.

Network latencies are pretty low these days. And even if you are
spending a few milliseconds the user won't probably notice.

In situations like these this is what I typically do: I create a toy
example that models important aspects of the original problem and
benchmark it. In your case I would create a simplified schema that
covers what you need and create a console only application that accesses
it in meaningful ways.

One solution to make JDBC with multiple queries fast is using the JDBC
driver's batch mode. That way you may reduce network roundtrips. The
efficiency of this of course depends on the driver and DB at hand.
Given that DB2 is a mature product and IBM is heavily involved in Java
for years I expect it to work pretty well.

Another solution would be to write stored procedures that smartly return
the data you need. Whether that is feasible depends of course on the
application logic.

In any case, if customers do have an AS/400 already and if they have DB2
on that machine already I'd try to use that. Advantage is that they do
not need to buy new licenses, install additional software, train admins
etc. and you can be sure that the DB is capable of handling the load
(and probably much higher load).

Depending on your data and access logic it may even be simple to hack
some client side caching together on top of JDBC.

Even if that approach fails or you have extreme requirements for
performance I would look into "non standard" solutions.

Kind regards

robert


--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/
From: Martin Gregorie on 3 Apr 2010 08:51
On Fri, 02 Apr 2010 16:23:02 -0600, Jim Janney wrote:

> I'm beginning to think that the same problem would apply to anything I
> do over a network, so I'm currently rethinking my whole approach to the
> problem. The data is small enough I can cache it all locally, but I
> need to reload it when it changes.
>
I seem to remember that AS/400 remote file replication via the
journalling facility worked pretty well, so if you do go for local
caching, remember to look at it. And, of course, its language-agnostic.


--
martin@ | Martin Gregorie
gregorie. | Essex, UK
org |
First  |  Prev  | 
Pages: 1 2 3
Prev: Buy Genuine Google Adsense Account only for Rs.200/- for indian people.
Next: Getting Windows ACLs in Java