Prev: Transferring an exception from one thread to an other.
Next: Drawing glyphs based on their index (NOT their character code)
From: Thomas Lehmann on 20 May 2010 06:58
>
> The question is:
> Is there a limit on the number of entries a dictionary can have i
> jython?
>
> I wrote a little app where my data is stored in a huge dictionary
> (11746 entries) generated with a python script.
> When I try to import the dictionary, jython complains with the
> following message:
>

1) You did not say what you have saved (content of your dictionary).
2) You did not say how you have saved.

From the callstack - also I have never used jython - it looks like
that
there is a try to create a class. It looks like - I may be wrong -
that
you have saved user objects in your dictionary - have you?

If so you might fail on loading those objects - especially when your
program
does not have the code for it.

More details are required...
From: keobox on 21 May 2010 11:54
On 20 Mag, 12:58, Thomas Lehmann <t.lehm...(a)rtsgroup.net> wrote:
> > The question is:
> > Is there a limit on the number of entries a dictionary can have i
> > jython?
>
> > I wrote a little app where my data is stored in a huge dictionary
> > (11746 entries) generated with a python script.
> > When I try to import the dictionary, jython complains with the
> > following message:
>
> 1) You did not say what you have saved (content of your dictionary).

- As you can see I received the error when I tried to import
jmoco_event_data.py module from the jython interpreter prompt.
- The "import" will cause the module compilation, jython will produce
a .class file instead of .pyc.
- jmoco_event_data.py contains 4 dictionaries:

typesValToName={
220:'EMS_EVENT_EMS_INTERNAL_BASE',
221:'EMS_EVENT_INTERNAL_TYPE_BASE',
222:'EMS_EVENT_INTERNAL_N_E_P_M_EVENT',
223:'EMS_EVENT_INTERNAL_N_E_ALARM_RCVD',
224:'EMS_EVENT_NE_SPECIFIC_BASE',
.... 11746 entries int key: string value
}

typesNameToVal={
'EMS_EVENT_EMS_INTERNAL_BASE':220,
'EMS_EVENT_INTERNAL_TYPE_BASE':221,
'EMS_EVENT_INTERNAL_N_E_P_M_EVENT':222,
'EMS_EVENT_INTERNAL_N_E_ALARM_RCVD':223,
'EMS_EVENT_NE_SPECIFIC_BASE':224,
.... total 11746 entries string key: int value
}

causesValToName={
0:'NOT_APPLICABLE_UNKNOWN',
1:'SOFTWARE_CAUSE_UNKNOWN',
2:'ABSENT_MODULE',
3:'FAULTY_MODULE',
.... 483 entries int key: string value
}

causesNameToVal={
'NOT_APPLICABLE_UNKNOWN':0,
'SOFTWARE_CAUSE_UNKNOWN':1,
'ABSENT_MODULE':2,
'FAULTY_MODULE':3,
.... 483 entries string key: int value
}

> 2) You did not say how you have saved.

The dictionaries are in the jmoco_event_data.py module.

> From the callstack - also I have never used jython - it looks like
> that
> there is a try to create a class. It looks like - I may be wrong -
> that
> you have saved user objects in your dictionary - have you?

Nope, the dictionaries are only int to string mappings and string to
int mappings

> If so you might fail on loading those objects - especially when your
> program
> does not have the code for it.

The failure happens during module's import, so my question is: Is
jython able to handle such big dictionaries?

> More details are required...

jmoco_event_data.py has a size of roughly 700KB.
From: Peter Otten on 22 May 2010 02:15
keobox wrote:

> On 20 Mag, 12:58, Thomas Lehmann <t.lehm...(a)rtsgroup.net> wrote:
>> > The question is:
>> > Is there a limit on the number of entries a dictionary can have i
>> > jython?
>>
>> > I wrote a little app where my data is stored in a huge dictionary
>> > (11746 entries) generated with a python script.
>> > When I try to import the dictionary, jython complains with the
>> > following message:
>>
>> 1) You did not say what you have saved (content of your dictionary).
>
> - As you can see I received the error when I tried to import
> jmoco_event_data.py module from the jython interpreter prompt.
> - The "import" will cause the module compilation, jython will produce
> a .class file instead of .pyc.
> - jmoco_event_data.py contains 4 dictionaries:
>
> typesValToName={
> 220:'EMS_EVENT_EMS_INTERNAL_BASE',
> 221:'EMS_EVENT_INTERNAL_TYPE_BASE',
> 222:'EMS_EVENT_INTERNAL_N_E_P_M_EVENT',
> 223:'EMS_EVENT_INTERNAL_N_E_ALARM_RCVD',
> 224:'EMS_EVENT_NE_SPECIFIC_BASE',
> ... 11746 entries int key: string value
> }
>
> typesNameToVal={
> 'EMS_EVENT_EMS_INTERNAL_BASE':220,
> 'EMS_EVENT_INTERNAL_TYPE_BASE':221,
> 'EMS_EVENT_INTERNAL_N_E_P_M_EVENT':222,
> 'EMS_EVENT_INTERNAL_N_E_ALARM_RCVD':223,
> 'EMS_EVENT_NE_SPECIFIC_BASE':224,
> ... total 11746 entries string key: int value
> }
>
> causesValToName={
> 0:'NOT_APPLICABLE_UNKNOWN',
> 1:'SOFTWARE_CAUSE_UNKNOWN',
> 2:'ABSENT_MODULE',
> 3:'FAULTY_MODULE',
> ... 483 entries int key: string value
> }
>
> causesNameToVal={
> 'NOT_APPLICABLE_UNKNOWN':0,
> 'SOFTWARE_CAUSE_UNKNOWN':1,
> 'ABSENT_MODULE':2,
> 'FAULTY_MODULE':3,
> ... 483 entries string key: int value
> }
>
>> 2) You did not say how you have saved.
>
> The dictionaries are in the jmoco_event_data.py module.
>
>> From the callstack - also I have never used jython - it looks like
>> that
>> there is a try to create a class. It looks like - I may be wrong -
>> that
>> you have saved user objects in your dictionary - have you?
>
> Nope, the dictionaries are only int to string mappings and string to
> int mappings
>
>> If so you might fail on loading those objects - especially when your
>> program
>> does not have the code for it.
>
> The failure happens during module's import, so my question is: Is
> jython able to handle such big dictionaries?

The problem seems to be the size of the code, i. e. the dict literal rather
than the size of the dictionary. With a slightly smaller dict I get

$ cat make_dict_source.py
from itertools import chain, count, islice

def main():
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("n", type=int, default=100)
parser.add_argument("py")

args = parser.parse_args()

prefix = ["data = {\n"]
suffix = ["}\n"]
pairs = ("'enum_whatever_{0}': {0},\n".format(i) for i in count())
pairs = islice(pairs, args.n)
with open(args.py, "w") as outstream:
lines = chain(prefix, pairs, suffix)
outstream.writelines(lines)

if __name__ == "__main__":
main()
$ python make_dict_source.py 4000 tmp.py
$ jython tmp.py
Traceback (innermost last):
(no code object) at line 0
java.lang.ClassFormatError: Invalid method Code length 71912 in class file
org/python/pycode/_pyx0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:632)
at org.python.core.BytecodeLoader2.loadClassFromBytes(Unknown
Source)
at org.python.core.BytecodeLoader.makeClass(Unknown Source)
at org.python.core.BytecodeLoader.makeCode(Unknown Source)
at org.python.core.Py.compile_flags(Unknown Source)
at org.python.core.Py.compile_flags(Unknown Source)
at org.python.core.__builtin__.execfile_flags(Unknown Source)
at org.python.util.PythonInterpreter.execfile(Unknown Source)
at org.python.util.jython.main(Unknown Source)

java.lang.ClassFormatError: java.lang.ClassFormatError: Invalid method Code
length 71912 in class file org/python/pycode/_pyx0
$ jython --version
Jython 2.2.1 on java1.6.0_0

Jython seems to generate a method that is bigger than the 64K allowed by
Java.

The simplest workaround is probably to let your module read the dict data
from a separate file in e. g. csv format.

Peter


 | 
Pages: 1
Prev: Transferring an exception from one thread to an other.
Next: Drawing glyphs based on their index (NOT their character code)