Saving C structures to file -- best practices
in [Embedded]
|
Prev: 50 ohm PCB antenna track impedance
Next: BT earpieces
From: Dave Boland on 24 Mar 2010 11:16 I'm doing some work on a 16-bit micro and one of the things the software does is store data in a structure for easy access. Currently, I save the entire structure to file and get it back from file when the program is restarted. No problem so far. However, if I need to change the structure (add a member, remove one, change one, etc.), the data, when read in from file, gets corrupted as one may expect. In addition, this code may be ported to a 32-bit controller, which means the alignment of the data in the structure will be different. So even if the structure doesn't change, when reading data created by a 16-bit processor I would expect the 32-bit processor to corrupt the data. So here is my question -- what are the best practices for saving data from a C structure so it is universally readable -- even if the structure definition changes? I'm leaning toward a character array (buffer) that uses name-value pairs, but that is a lot more work, and may be over-kill. Any thoughts? Dave,
From: Vladimir Vassilevsky on 24 Mar 2010 11:57 Dave Boland wrote: > I'm doing some work on a 16-bit micro and one of the things the software > does is store data in a structure for easy access. Currently, I save > the entire structure to file and get it back from file when the program > is restarted. No problem so far. This way eventially leads to disaster because of platform - toolset - settings dependent order of the members, alignment, size, endian and padding. If you want to save structures, make explicit functions for conversion each struct member to array of bytes and back. > However, if I need to change the structure (add a member, remove one, > change one, etc.), the data, when read in from file, gets corrupted as > one may expect. In addition, this code may be ported to a 32-bit > controller, which means the alignment of the data in the structure will > be different. So even if the structure doesn't change, when reading > data created by a 16-bit processor I would expect the 32-bit processor > to corrupt the data. > > So here is my question -- what are the best practices for saving data > from a C structure so it is universally readable -- even if the > structure definition changes? I'm leaning toward a character array > (buffer) that uses name-value pairs, but that is a lot more work, and > may be over-kill. Any thoughts? The only portable way of saving data to files is in the text form. This also allows for easy editing of the files. I like Windows-style (like GetPrivateProfileString() does): [Setup_Data] foo = 12345; bar = 6789; [Test_Data] bla_bla_bla = 123.456; etc. etc. Vladimir Vassilevsky DSP and Mixed Signal Design Consultant http://www.abvolt.com
From: Spam on 24 Mar 2010 12:06 On Wed, 24 Mar 2010, Dave Boland wrote: > Date: Wed, 24 Mar 2010 11:16:13 -0400 > From: Dave Boland <dboland9(a)fastmail.fm> > Newsgroups: alt.comp.lang.c, comp.arch.embedded > Subject: Saving C structures to file -- best practices > > I'm doing some work on a 16-bit micro and one of the things the software does > is store data in a structure for easy access. Currently, I save the entire > structure to file and get it back from file when the program is restarted. > No problem so far. > > However, if I need to change the structure (add a member, remove one, change > one, etc.), the data, when read in from file, gets corrupted as one may > expect. In addition, this code may be ported to a 32-bit controller, which > means the alignment of the data in the structure will be different. So even > if the structure doesn't change, when reading data created by a 16-bit > processor I would expect the 32-bit processor to corrupt the data. > > So here is my question -- what are the best practices for saving data from a > C structure so it is universally readable -- even if the structure definition > changes? I'm leaning toward a character array (buffer) that uses name-value > pairs, but that is a lot more work, and may be over-kill. Any thoughts? > > Dave, > I'd suggest coding in XML (or similar) ... pretty much any arbitrary structure can be serialized, but you'll need a parser which can handle your variant to read the structures back in ...
From: Rich Webb on 24 Mar 2010 11:54 On Wed, 24 Mar 2010 11:16:13 -0400, Dave Boland <dboland9(a)fastmail.fm> wrote: >I'm doing some work on a 16-bit micro and one of the things the software >does is store data in a structure for easy access. Currently, I save >the entire structure to file and get it back from file when the program >is restarted. No problem so far. > >However, if I need to change the structure (add a member, remove one, >change one, etc.), the data, when read in from file, gets corrupted as >one may expect. In addition, this code may be ported to a 32-bit >controller, which means the alignment of the data in the structure will >be different. So even if the structure doesn't change, when reading >data created by a 16-bit processor I would expect the 32-bit processor >to corrupt the data. > >So here is my question -- what are the best practices for saving data >from a C structure so it is universally readable -- even if the >structure definition changes? I'm leaning toward a character array >(buffer) that uses name-value pairs, but that is a lot more work, and >may be over-kill. Any thoughts? Add to the structure a version and a size member in a fixed location (most easily at the top of the structure). That gives your application code at least some chance of doing the right thing if it knows, e.g., that member X is only present in version Y and later. Once the layout has mostly settled down, only add new members at the end. Never remove (or change) old members, just mark them as "reserved" or "optional" and fill them with benign values. For alignment, make the first element of the structure a 32-bit quantity. The standard requires that a pointer to the structure also points to the first element (and vice versa, of course). That should "square up" the overall alignment between 16- and 32-bit environments. -- Rich Webb Norfolk, VA
From: Dave Boland on 24 Mar 2010 15:13
Spam(a)ControlQ.com wrote: > On Wed, 24 Mar 2010, Dave Boland wrote: > >> Date: Wed, 24 Mar 2010 11:16:13 -0400 >> From: Dave Boland <dboland9(a)fastmail.fm> >> Newsgroups: alt.comp.lang.c, comp.arch.embedded >> Subject: Saving C structures to file -- best practices >> >> I'm doing some work on a 16-bit micro and one of the things the >> software does is store data in a structure for easy access. >> Currently, I save the entire structure to file and get it back from >> file when the program is restarted. No problem so far. >> >> However, if I need to change the structure (add a member, remove one, >> change one, etc.), the data, when read in from file, gets corrupted as >> one may expect. In addition, this code may be ported to a 32-bit >> controller, which means the alignment of the data in the structure >> will be different. So even if the structure doesn't change, when >> reading data created by a 16-bit processor I would expect the 32-bit >> processor to corrupt the data. >> >> So here is my question -- what are the best practices for saving data >> from a C structure so it is universally readable -- even if the >> structure definition changes? I'm leaning toward a character array >> (buffer) that uses name-value pairs, but that is a lot more work, and >> may be over-kill. Any thoughts? >> >> Dave, >> > > I'd suggest coding in XML (or similar) ... pretty much any arbitrary > structure can be serialized, but you'll need a parser which can handle > your variant to read the structures back in ... > > > I thought about XML, but that will eat up too much storage (I have 2 KB). I also thought about JASON, and the way web page data is encoded when a POST occurs (name1=data1&name2=data2a+data2b). I'm going to look at all of the suggestions to see what makes the best compromise between robust, small, flexible, easy to do (in order). Dave, |