pickle unable to load collection
in [Python]
|
From: paragk on 17 May 2010 22:02 Hi, I am unable to figure out the cause of python pickle unable to find the collection module. I am getting __import__(module) ImportError: No module named collections when I try to load a pickled object. Details: Python version: Python 2.6 (r26:66721, Oct 2 2008, 11:35:03) [MSC v.1500 32 bit (Intel)] on win32 [CODE] import sys import glob import collections import pickle .... SomeListDict = collections.defaultdict(list) .... # Populate SomeListDict DictFile = open (options.saveDict, 'w') pickle.dump(SomeListDict , DictFile, -1) DictFile.close() .... # Try loading the dictionary LoadDictFile = open(options.loadDict, 'rb') SomeListDict = pickle.load(LoadDictFile) LoadDictFile.close() .... [CODE END] The pickle.dump(CallGraphDict, DictFile, -1) works fine. The dictionary is saved. The interesting thing is, the dump function is able to find the collections module. try: __import__(module) mod = sys.modules[module] klass = getattr(mod, name) The above try block succeeds. module str: collections mod module: <module 'collections' from 'C:\Python26\Lib\collections.pyc'> klass type: <type 'collections.defaultdict'> The SomeListDict = pickle.load(LoadDictFile) fails with __import__(module) ImportError: No module named collections The sequence of calls is: def load_global(self): module = self.readline()[:-1] name = self.readline()[:-1] klass = self.find_class(module, name) self.append(klass) dispatch[GLOBAL] = load_global Variables: module str: collections name str: defaultdict The code where the exception occurs is: def find_class(self, module, name): # Subclasses may override this __import__(module) mod = sys.modules[module] klass = getattr(mod, name) return klass From the above analysis, clearly the collection module exists, since the dump works. What am I missing?
From: Steven D'Aprano on 17 May 2010 22:07 On Mon, 17 May 2010 19:02:18 -0700, paragk wrote: > Hi, > > I am unable to figure out the cause of python pickle unable to find the > collection module. > > I am getting > > __import__(module) > ImportError: No module named collections What is the actual value of module? Here's an obvious failure mode: >>> __import__("collections ") Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named collections > module > str: collections Try printing repr(str) to see what's actually there. It may also help if you post a copy-and-paste of the complete traceback. -- Steven
From: Peter Otten on 18 May 2010 03:10 paragk wrote: > Hi, > > I am unable to figure out the cause of python pickle unable to find > the collection module. > > I am getting > > __import__(module) > ImportError: No module named collections > > when I try to load a pickled object. > > Details: > > Python version: > > Python 2.6 (r26:66721, Oct 2 2008, 11:35:03) [MSC v.1500 32 bit > (Intel)] on win32 > > > [CODE] > > import sys > import glob > import collections > import pickle > > ... > > SomeListDict = collections.defaultdict(list) > > ... # Populate SomeListDict > > DictFile = open (options.saveDict, 'w') > pickle.dump(SomeListDict , DictFile, -1) > DictFile.close() [...] > From the above analysis, clearly the collection module exists, since > the dump works. > > What am I missing? You have to open the file in binary mode "wb". >>> import collections >>> import pickle >>> data = pickle.dumps(collections.defaultdict()) >>> pickle.loads(data) defaultdict(None, {}) Now simulate the effect of writing in text mode and reading in binary mode: >>> garbled_data = data.replace("\n", "\r\n") >>> pickle.loads(garbled_data) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python2.6/pickle.py", line 1374, in loads return Unpickler(file).load() File "/usr/lib/python2.6/pickle.py", line 858, in load dispatch[key](self) File "/usr/lib/python2.6/pickle.py", line 1090, in load_global klass = self.find_class(module, name) File "/usr/lib/python2.6/pickle.py", line 1124, in find_class __import__(module) ImportError: No module named collections >>> try: pickle.loads(garbled_data) .... except ImportError as e: .... e .... ImportError('No module named collections\r',) So as Steven suspected there was a whitespace char in the module name: pickle.load() was looking for the "collections\r" module. Peter
From: paragk on 18 May 2010 04:26 On May 18, 12:10 am, Peter Otten <__pete...(a)web.de> wrote: > paragk wrote: > > Hi, > > > I am unable to figure out the cause of python pickle unable to find > > the collection module. > > > I am getting > > > __import__(module) > > ImportError: No module named collections > > > when I try to load a pickled object. > > > Details: > > > Python version: > > > Python 2.6 (r26:66721, Oct 2 2008, 11:35:03) [MSC v.1500 32 bit > > (Intel)] on win32 > > > [CODE] > > > import sys > > import glob > > import collections > > import pickle > > > ... > > > SomeListDict = collections.defaultdict(list) > > > ... # Populate SomeListDict > > > DictFile = open (options.saveDict, 'w') > > pickle.dump(SomeListDict , DictFile, -1) > > DictFile.close() > > [...] > > > From the above analysis, clearly the collection module exists, since > > the dump works. > > > What am I missing? > > You have to open the file in binary mode "wb". > > >>> import collections > >>> import pickle > >>> data = pickle.dumps(collections.defaultdict()) > >>> pickle.loads(data) > > defaultdict(None, {}) > > Now simulate the effect of writing in text mode and reading in binary mode: > > >>> garbled_data = data.replace("\n", "\r\n") > >>> pickle.loads(garbled_data) > > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > File "/usr/lib/python2.6/pickle.py", line 1374, in loads > return Unpickler(file).load() > File "/usr/lib/python2.6/pickle.py", line 858, in load > dispatch[key](self) > File "/usr/lib/python2.6/pickle.py", line 1090, in load_global > klass = self.find_class(module, name) > File "/usr/lib/python2.6/pickle.py", line 1124, in find_class > __import__(module) > ImportError: No module named collections > > >>> try: pickle.loads(garbled_data) > > ... except ImportError as e: > ... e > ... > ImportError('No module named collections\r',) > > So as Steven suspected there was a whitespace char in the module name: > pickle.load() was looking for the "collections\r" module. > > Peter Thank you for the suggestions. Opening the file in 'wb' mode worked. Thanks, Parag
From: Terry Reedy on 18 May 2010 13:07 On 5/18/2010 3:10 AM, Peter Otten wrote: > Now simulate the effect of writing in text mode and reading in binary mode: > >>>> >>> garbled_data = data.replace("\n", "\r\n") >>>> >>> pickle.loads(garbled_data) > Traceback (most recent call last): .... > ImportError: No module named collections > >>>> >>> try: pickle.loads(garbled_data) > ... except ImportError as e: > ... e > ... > ImportError('No module named collections\r',) Nice catch, Peter. I filed a request that the message quote the name it cannot import, so that the OP would have seen the more informative ImportError: No module named 'collections\r' http://bugs.python.org/issue8754 Terry Jan Reedy
|
Next
|
Last
Pages: 1 2 Prev: Can't find _sqlite3.so in lib-dynload Next: max time wait for a function |