Auto-update protocol
in [Embedded]
|
Prev: Microchip Introduces 16 New Enhanced Mid-Range 8-bit Core Microcontrollers
Next: New Rabbit 6000 Microcontroller has on-board WiFi
From: D Yuniskis on 9 Mar 2010 15:48 Hi Tim, Tim Watts wrote: > D Yuniskis <not.going.to.be(a)seen.com> > wibbled on Monday 08 March 2010 19:34 > >> I'm looking for ideas for a lightweight (yet robust) >> autoupdate protocol for a family of network clients >> I'm designing. >> >> Some criteria: >> >> - Unattended, persistent operation. A device shouldn't >> require hand-holding from a user to get itself into a >> working configuration. It is acceptable (though not >> *desireable*) to withold its normal functionality >> from the user while updating *or* recovering from a >> botched upgrade. But, it must be able to eventually >> "right itself". >> >> - I don't want to have to *tell* the devices that they >> need to be updated. The user shouldn't even be aware >> that this is happening. I.e., the devices should be >> able to examine/query the "current images" to see if >> they coincide with *their* images. >> >> - I should be able to tailor an image to a *particular* >> device (i.e., an *instance* of a device; not just a >> "model number"). >> >> - Images need to be signed so they can't be forged. >> >> - The protocol can't be spoofed by "unfriendlies". >> - The protocol should be as light as possible -- but >> no lighter! ;-) > > I would use TFTP for the actual image transfer - simple, standard and the > client initiates the connection so as long as it is happy it is talking to > the right IP it doesn't have to worry too much about connection based > attacks - code signing should sort the rest of the security out. The issue (I think) with TFTP is that it doesn't protect against the image file being updated "between packets". So, you run the risk of pulling down part of one image and part of another, flashing this combination -- only to discover the resulting image is corrupt (fails checksum). With a connection oriented protocol, the server side can open() the desired file and keep a link to it throughout the transaction (even if the original file is unlink(1)-ed, etc.) The client can try to *recover* from this mixup. But, it means starting the entire process over again. In a pathologic scenario, it could *never* recover completely (highly unlikely, but possible). >> - Minimize unnecessary network traffic as well as >> load on the server (the goal is for the user *not* >> to notice this activity -- though I am not trying >> to "keep it secret") > > Unless you go multicast/broadcast (with the extra complexity that will force > onto the clients) I don't think you could do much better. Would you really > need to reduce net traffic below one image transmission per device? My point was to avoid *needless* traffic. E.g., don't download the image ALL THE TIME if you don't need to do so. (imagine a RAM-based environment which *would* need that sort of support) >> (seems like I have forgotten something -- but I >> can't recall what! :< Too early in the day...) > > Now the interesting bit is advertising the availability of images to the > clients. TFTP of course cannot do directory listings, though the client > could periodically retrieve a manifest file. Or, it can just request a *specific* file name (e.g., related to it's MAC address -- since both sides of the link would need to know that) >> So, all I should have to do is put "current images" >> on the user's server and wait for each device to >> discover the image and update itself accordingly. >> >> For example, the devices could check the server at IPL >> (and periodically at run time -- though updates at >> run time can be more of a challenge as they will >> probably interfere with normal operation :< ) >> and "fingerprint" the current image located there >> to see if it looks like it differs from its own >> image. >> >> One way of doing this is to store the image for >> each device instance in a R/O file bearing the MAC >> of the device in question. But, this would >> require the device to perform a "trial download" >> of the image for the sole purpose of computing >> the fingerprint (why not just do a bytewise compare >> if you are going to this extreme?!). > > Do the devices have a notion of "type ID" anywhere in their firmware? In > which case, you could serve the files up thusly named: Well, audio clients differ from video clients differ from... But, I don't want to have to "presonalize" each instance of an "audio client" from each other instance. They configure themselves at runtime. However, I may want to try a different image on a particular client (e.g., during development or when testing new features). So, I would want to be able to say: "device having MAX xx:xx:xx:xx:xx:xx please use *this* image" As such, if you can support this, then you might as well specify the image for *each* device in this same way. For example, I manage my X Terminal clients with a file hierarchy of: ModelA/ Version1/ ConfigurationX ConfigurationY Version2/ ConfigurationW Version3/ ConfigurationQ ModelB/ Version2/ ConfigurationP Version47/ ... Devices/ MACxxxxxxxxxxxx -> ../ModelA/Version1/ConfigurationY MACyyyyyyyyyyyy -> ../ModelA/Version3/ConfigurationQ MACzzzzzzzzzzzz -> ../ModelB/Version2/ConfigurationP but this is very "manual" -- not the sort of thing I want to impose on others. > TypeID-Version > or > MacAddress > > eg > > 23FE-010A > 23FE-010B > FFEECCBB1177 > 2401-0200 > > etc > > Any device of type 23FE will sort the list and discover that version 010B is > the highest version and will update to that. > > Device with MAC address FFEECCBB1177 will find a specific image and will see > if it i the one it is currently running and if necessary will update to > that. > > Images should contain a header also containing TypeID and Version so the > device can make sanity checks that it has runnable code. Already considered. Fingerprinting the image (secure hash). And, if the image crashes, you remember that and don't try it again! :> > Simple, controllable. > >> This will hammer the network pretty hard. Granted, >> the individual segments for each device will see >> modest traffic -- but the server's segment will >> quickly max out if multiple devices do this >> simultaneously (e.g., powering up together). >> This would necessitate a second layer of the >> protocol to randomize/defer such competition. :< > > Random hold off for server polling. Or don't poll, but send a magic packet > (contents unimportant) to a specific device to start the poll operation, > then the server can manage timings. I don't want the "image server" to be aware of this stuff. I want to burden the devices with it all. After all, *they* are the things "benefiting" from this... :> >> Another approach is to *store* the fingerprint on the >> server in a uniquely accessible manner (e.g., use >> a file name like MAC.fingerprint). But, that represents >> a duplication of data (pet peeve of mine) which makes >> it at risk for getting out of sync. >> >> For example, updating the image and forgetting to update >> the fingerprint; or, a run-time race -- the device >> examines the fingerprint, sees that it differs, starts >> to download image but the image hasn't been updated >> yet. Or, the image is updated but the fingerprint >> is stale when the device examines it. As a result, >> the image is NOT seen as "new". >> >> [you could get around this if you implemented network >> file locking -- but that is more complexity and leaves >> open the possibility of stale locks, etc.] >> >> This could be worked around by forcing the server to >> recompute the fingerprint each time an image is added. >> But, that still leaves a race window *and* requires >> the server to be aware of the introduction of the >> new image file(s). >> >> The simplest compromise would seem to be having the >> device track the timestamp of the image file and >> check *that*. If changed, then *assume* the image >> actually has changed (of course, touch(1)-ing the >> image file would then force the device to consider >> the image file as changed -- this could be an >> advantage?). The device could then either >> fingerprint the image itself *or* naively assume >> it to be a new image and begin the (secure) update >> procedure. > > I'd just go for Type and Version numbers, it's all you need if I understand > your problem correctly. Timestamps are not really necessary. Timestamp is a way around explicit version numbers. Their advantage is that they are viewable "outside" the image itself. >> Security then is another facet to be addressed. E.g., >> given that the devices *don't* have enough resources to >> store an entire image before flashing, the protocol >> would have to be interruptible. E.g., encrypted >> packets so they can't be spoofed. But, accepting >> the possibility that the entire image might not >> become available "when needed" for the reflash. >> I.e., fall back to a secure boot loader that >> can do the update without the rest of the application >> being available. >> >> Note that this loader should *not* restart the upload >> but, instead, *continue* where it apparently left off, >> previously -- even in light of any intervening power >> cycles. This saves time, reduces flash wear and is >> just "smarter", in general. :> >> >> But, it seems like this *still* requires server side >> locking so the file isn't changed *while* it is being >> doled out. E.g., something like TFTP would be >> inappropriate as it doesn't reliably lock the file >> from one packet to the next. > > Put a management system in place that people do not manage the TFTP > directory directly, but give new firmware to a script that gracefully puts > it in place, atomically updating the manifest file (hint rename() under > linux is atomic) Again, this forces changes on the server. I can build something into the makefile (e.g., "make release") that automates some of this. But, that will be happening on a different host so that adds more things that can go wrong. And, it doesn't address "production deployments" -- build an image and distribute it to others for *them* to deploy locally. > I think the hardest part is managing encryption and/or code signing in a way > that doesn't overload the client if they are lightweight. The basic approach That's another reason why I want to avoid unnecessary updates. (network traffic, slower boot times, more windows of vulnerability, etc.). The "easy" fix is something that *pushes* updates to each device. But, that also requires the most "support" (on the servers as well as "by the developer") :< I want a push deployment in a pull implementation! :> > however of offering a manifest of files and allowing the client to make a > simple informed choice is the easy bit IMO.
From: D Yuniskis on 10 Mar 2010 16:16 Hi Tim, Tim Watts wrote: > D Yuniskis <not.going.to.be(a)seen.com> > wibbled on Wednesday 10 March 2010 19:48 > >>> If you were to open() with O_WRONLY or O_RDWR then you stand to modify >>> the file in place with unpredictable results unless coordination >>> mechanisms are in place. >> <frown> > > If it didn't, how could you write any system with shared access to a set of > files, eg an RDBMS? Yes, I was assuming that FTPd et al. would take pains *not* to do this (i.e., to effectively lock the file *in* the filesystem instead of having to mmap it completely *or* risk it being changed midstream). As I said, I'll have to start perusing sources. >> Yes. I was hoping I could rely on particular (existing) services >> to do that "consequentially" for me. E.g., using connection >> oriented protocols in the hope that they would "lock" files >> they were actively transfering (vs. connectionless protocols >> which can opt to forget they were even involved in a transfer >> "moments ago") > > Can I ask for some parameters? How big are these images likely > to be in kB with any header overhead? For the audio and video clients, small -- probably 200K. But, I'm trying to learn something from this exercise that I can later use on other applications. E.g., if I come up with a scheme that scales nicely, then all-the-better. If it only works for small images, then I have to design bigger applications with a "piece-meal" update policy inherent in the design (i.e., plan on only updating a portion at a time) > Just because we've considered the obvious, doesn't mean that there might not > be something already existing - such as one particular instance of an FTP > server that does this. Understood. I need to make time to see how the ones I have access to work. That might require "empirical testing" for those whose sources I can't inspect (so, I will need to find a way to throttle an FTP client so I can start a transfer and then have time to "play" on the FTP host while the transfer is still active) >>> If you are at the mercy of customer server hardware, I suspect the only >>> sane way is to provide a daemon of your own for them to install. Note >>> however, this could be Apache with your custom config and service tree of >>> Perl/PHP/whatever. Whether you write a server from scratch or wrap the >>> protocol in HTTP and bundle an Apache blob, at least you give the user >>> something to install which you do control. >>> >>> I don't think there is any other way around it. >> <frown> > > Do you have any expectations of your customer's systems - are you contrained > to one OS, do you supply the update server - or do they, will they insist on In the short term (e.g., all the home automation stuff) I really don't *care* about the "customer" :> It will all be open source so, hopefully, *someone* will be willing to pick up the mantle and carry it forward (*I* Shirley won't! :> ) > using any random OS they happen to like? Would it be less of a headache to > sell them a little embedded update server with an FTP interface that they > dump images onto (which of course are auto managed by the box in a correct > fashion)? They might like that - little black box to plug n go and you get This could be a win for the "bigger" projects down the road. Which would suggest it might be smarter for me just to bite the bullet and come up with a *good* protocol, now. Though that would force "someone" to port it to <whatever> OS folks opted to use for this "short term" project. :-/ > to own it. Would one of your existing devices be powerful enough to just re- > badge for the job, with perhaps additional flash bolted on. Just thinking > out loud here... Yes, this is worth looking into. I could even build a small "system" (disk, etc.) to act as an "on-site" agent for this express purpose. >>> Personally I'd probably write a perl server based off FTP or TFTP using >>> the many available CPAN modules to do a forking server (or a multiplexed >>> one) and also to implement the protocol. There's be very little actual >>> code to write so it shouldn't take long. But of course that's just me. >> I suspect I am just going to leave others deal with *their* >> problems as best *they* see fit! :> I can leave myself a note >> as to how *I* would (in the future) update my devices and >> "not worry" about whether this works for other people. >> (I am amazed at how often people fail to read/understand >> "simple instructions". Particularly when it comes to updating >> firmware -- e.g., why are there so many "bricked" devices >> out there? couldn't people *read* how NOT to brick their >> device??? :< ) > > Always a difficult one. Are your users technical? Most sysadmins would be > expected to flash all manner of devices using anything from TFTP to > XModem/RS232 and not break the box (otherwise their boss chops their nuts <grin> In a quality shop, that's not a problem. But, you can't always be sure that you'll be dealing with one. In my case, worst possible outcome is a dead device (whether that is permanent or temporary) as it can "cost" the user. Which means it will cost *me*! :< And, even if the device runs, if it isn't running the firmware that the user *thinks* it SHOULD be running ("I updated the firmware and its not working the way the release notes claim it should!" -- "No, you *think* you updated the firmware but it actually hasn't happened, yet"). Service calls cost money/time. :< I think the biggest headache will be the opensource stuff as folks there tend to be "cheap" -- wanting something for nothing -- in terms of the time and effort they are willing to invest. I can (will?) simply chose to ignore the whining -- "You must be THIS TALL to ride this ride" -- or come up with something that minimizes the chance of folks having problems. (while it's my nature to *try* to be patient and helpful, as I get older, I find myself having less time available for "hand-holding" :< Or, maybe I'm just getting cranky and irritable!! ;-) ) > off). If you sold the devices to me, i would not consider running a little > TFTP server a problem, provided clear instructions were given on how to load > the new images. > > But if your users are AV dudes, that might be too much to expect - their > expertise lies in different places. Not sure where your (Linux, etc.) lies. But, look at the chatter frequently accompanying things like MythTV. How many of those questions are unnecessary? Why bother writing release notes, How-To's, etc. if people are *still* going to whine: "Mine doesn't work..." -- "OK, go *buy* a DVR!!" >> Correct. The TEXT is considerably larger than the RAM available >> in the devices (which is often the case!). It is going to be >> a significant challenge to arrange the image so that I can >> do the update while the device is still "operating" (something >> I've opted not to mention in this thread as its just extra >> complexity that doesn't, IMO, impact the update protocol >> itself) > > I don't suppose you have enough flash for 2 images with a pivot in the > bootloader? I expect that is obvious to you, so I apologise for wibbling > eggs to granny ;-> <grin> No. What I plan on doing is "partitioning" the application so that the minimum amount of it *required* to fulfill its function can remain active. E.g., sort of like exec()-ing a pared down version of itself. Then, while that continues to provide the service intended (e.g., play music), update the rest of the application. Finally, once that portion has been reflashed, update the "active player" portion (while executing out of RAM or the *updated* "pared down" version of this code). It will require some juggling to ensure I can reuse data structures from one version to the next "on the fly" but I can do that if I design with that in mind. >> Reliability is of paramount importance. Can never have corrupted >> images. Must (also) always be able to *complete* an update >> (subject to the service being available, of course). > > OK > >> Efficiency -- in terms of how it impacts other things on the >> network and the server(s) -- is next most important. Since >> updates are (should be!) infrequent, don't want them to bring the >> rest of the "system" to its knees when they occur. > > I think randomised polling will deal with that adequately. Yes. Though the recent idea of having one client tell the others about an available update adds a complication (i.e., the recipients will have to deliberately hold off on trying to update) >> Simple is not important. However, I am trying to push the >> complexity into the *device* instead of the server. :< >> >>> Having written a fair few mini servers in perl, it is a small cost to >>> make other stuff work. >>> >>> As you say, you basically want a naming scheme and a server that caches >>> the file to RAM when asked for a download and verifies the file before >> You also want to make that server multithreaded *and* smart -- so it >> doesn't try to cache 5 copies of the same image for five different >> clients (and not service them "sequentially"). :-/ > > Again - step back. How many devices and how big is the image. If the > probability leans towards 5 concurrent connections for a few minutes for a > single 64kB image, although you might like to be clever, it's not really > worth it if the server has 2GB RAM! Sure. For an audio/video/home automation client, no big deal. But, imagine updating 5 "Linux servers" (e.g., new file system images) concurrently. :> I.e., I have to think about how the protocol/process will scale. >>> Does the client have a list of all the other clients - otherwise such an >>> announcement is unlikely to escape the VLAN segment. >> Part of ongoing network discovery. E.g., the devices all need >> to synchronize their clocks, converse with each other when >> something goes wrong (i.e., to determine if the reason "I" >> am not getting data from the audio server is because of something >> specific to *me* or common to "all of us", etc.), etc. > > The only reliable way unless you know UDP broadcasts are guaranteed to get > between devices would be to coordinate through the server - at least as far > as each device is given the IPs of one device on each other VLAN or > broadcast domain. > > If the expectation is that many devices will likely be within the same > broadcast domain, and it is not important that they can talk to *all* their > friends, broadcasting should be good enough. I don't have enough > information. I am thinking more in terms of using "dedicated" TCP connections between devices. The server can inform each device -- as it comes on-line (initial connection to server) -- of its peers. Since we're not talking about thousands of peers, it's not a huge demand to place on a device (i.e., remember a list of a couple dozen peers). The device could also update its list "as needed" from the audio/video server (but, must take some pains to keep *a* list locally lest the server crash or lose connectivity) >> Yes. Any protocol has to take into account the fact that there >> might be switches, routers, etc. "in the way" and how to bridge >> those -- or, how *not* to! > > Your unlikely to have issues with Layer 2 switches IME even over trunked > connections to the same VLAN on other switches, but once routing is > involved, it's mostly game over for broadcasts. Yes. Here it's a problem -- but that's just because of my current network topology (speaking in terms of the A/V/HA devices). I can fix that. Or, just use the directed TCP connections. >>>> Wow, that sounds cool! ;-) >>> Yes it would. >> Progress! :> > > Call me devil's advocate(!) I'm a newbie with tiny systems but I've > run some fairly decent linux installations. I think you have to actively question *all* of your design decisions to prove they are robust -- or, at least, be able to identify how they *will*/can break! It's like doing a backup/restore cycle "needlessly" -- just to be sure it *does* work! :>
From: George Neuner on 11 Mar 2010 00:06 On Mon, 08 Mar 2010 12:34:53 -0700, D Yuniskis <not.going.to.be(a)seen.com> wrote: >I'm looking for ideas for a lightweight (yet robust) >autoupdate protocol for a family of network clients >I'm designing. Hi, I've seen at least some of the ongoing discussion, but it might help to have a little more information (see below). >Some criteria: > >- Unattended, persistent operation. A device shouldn't >require hand-holding from a user to get itself into a >working configuration. It is acceptable (though not >*desireable*) to withold its normal functionality >from the user while updating *or* recovering from a >botched upgrade. But, it must be able to eventually >"right itself". How large are the client images? How many devices? How much bandwidth? What is the overall structure? Does the new image reset the device or is it a hot patch that needs to continue using existing data structures? Is it feasible to have the client load a new image concurrently with the old and switch at some well defined point(s)? >- The protocol can't be spoofed by "unfriendlies". There's only one real solution to spoofing and/or "man in the middle" attacks and that is to encrypt the image binaries. You want it to be as hard as possible for someone to decompile your application and figure out how to spoof it. Building on the idea of embedding the signature in a header, you could also embed a decryption key for the image. The header itself should be encrypted with a public key algorithm so that the client can verify that _you_ created the image. (Obviously the whole image could be public key encrypted, but it isn't necessary ... symmetric key algorithms are more performant. Successfully completing the process of decrypting the header, extracting the image decryption key, decrypting and verifying the image signature proves that the whole thing is legitimate.) And don't store the decrypted image on the device ... design the update process so that the local image can be decrypted into RAM and reuse it to do the startup. >- The protocol should be as light as possible -- but >no lighter! ;-) This feeds back to the size of the images. TFTP is about as simple as you can get using a generic farm server, but TFTP does not have any way to request a partial transfer (e.g., a "head" command). Of course, for version checking, you can hack your client to start the transfer and then abort after fetching the predetermined image header. >- Minimize unnecessary network traffic as well as >load on the server (the goal is for the user *not* >to notice this activity -- though I am not trying >to "keep it secret") How often do you need to check? How often do you anticipate a planned update (as opposed to an emergency bug fix)? >This will hammer the network pretty hard. Granted, >the individual segments for each device will see >modest traffic -- but the server's segment will >quickly max out if multiple devices do this >simultaneously (e.g., powering up together). >This would necessitate a second layer of the >protocol to randomize/defer such competition. :< Again, how many devices and how much bandwidth? >But, it seems like this *still* requires server side >locking so the file isn't changed *while* it is being >doled out. E.g., something like TFTP would be >inappropriate as it doesn't reliably lock the file >from one packet to the next. Use a "well known" public name and hide the file updates behind hard links. Changing a hard link is an atomic process so there will be no race conditions with the update - a process trying to open the file through the link will either get some version of the file or fail if the open call sneaks between unlinking the old file and linking the new. So updating the server then is a 3 step (scriptable) process. You upload the new version alongside the old, change the well known hard link to point to the new version, then delete the old version. Of course, if the old version happens to be in use it will persist until all handles to it are closed. Both Unix/Linux and Windows (NTFS) support hard links, so you have a wide choice of server platforms. >Shirley, this sort of thing has been done before (?). Don't call me "Shirley". >Thanks! >--don George
From: D Yuniskis on 11 Mar 2010 01:56 Hi George, George Neuner wrote: > On Mon, 08 Mar 2010 12:34:53 -0700, D Yuniskis > <not.going.to.be(a)seen.com> wrote: > > I've seen at least some of the ongoing discussion, but it might help > to have a little more information (see below). > >> Some criteria: >> >> - Unattended, persistent operation. A device shouldn't >> require hand-holding from a user to get itself into a >> working configuration. It is acceptable (though not >> *desireable*) to withold its normal functionality >>from the user while updating *or* recovering from a >> botched upgrade. But, it must be able to eventually >> "right itself". > > How large are the client images? How many devices? How much > bandwidth? They will vary. The audio client, e.g., is about 200K. There are a couple dozen of them (here... I doubt anyone else would use that many :> ). The video client is a bit larger but only 4 or 5 of those. The other "Home Automation" clients are much smaller -- maybe 50-100KB (depending on how I implement the file system) -- but there are several "flavors". But, this is all just a "make one to throw away" exercise. The projects that I am "practicing for" have much larger images (1 - 4G) -- though generally have more resources to call on at run time, so... <shrug> I'm currently playing on a 100Mb wired network. But, I am looking for a solution that will scale up to Gb and down to "wireless" rates. > What is the overall structure? Does the new image reset the device or > is it a hot patch that needs to continue using existing data > structures? Everything tries to run hot if at all possible. E.g., if conditions coincidentally arise that would cause some piece of code to rely on resources that aren't available during the update, then things might not quite work as they would if the update process was not active (but this would be unlikely). > Is it feasible to have the client load a new image concurrently with > the old and switch at some well defined point(s)? Not the whole image. I am structuring the application so I can swap out pieces of it and update it incrementally. E.g., update the RTOS, then switch it in; update parts of the library, then switch them in; etc. >> - The protocol can't be spoofed by "unfriendlies". > > There's only one real solution to spoofing and/or "man in the middle" > attacks and that is to encrypt the image binaries. You want it to be > as hard as possible for someone to decompile your application and > figure out how to spoof it. The sources for the application will be available. So, shared secret/public key is the only way to do the encryption. The problem there is it forces all devices to have the same keys (or, I have to build special images for each device). The more realistic problem is to guard against something messing with the image or process and effectively leaving the device(s) in "half updated" states. So, it is even more important that updates be supported "in pieces" (otherwise the window of opportunity for screwing things up is just too big) > Building on the idea of embedding the signature in a header, you could > also embed a decryption key for the image. The header itself should > be encrypted with a public key algorithm so that the client can verify > that _you_ created the image. (Obviously the whole image could be > public key encrypted, but it isn't necessary ... symmetric key > algorithms are more performant. Successfully completing the process > of decrypting the header, extracting the image decryption key, > decrypting and verifying the image signature proves that the whole > thing is legitimate.) Yes, but it would have to be done for each "piece". > And don't store the decrypted image on the device ... design the > update process so that the local image can be decrypted into RAM and > reuse it to do the startup. Huh? I *think* you are assuming there is enough RAM to support the decrypted image *and* my run-time data requirements? If so, that's not the case -- far more code than data. I'm not too worried about people reverse engineering devices. Rather, I am more concerned with someone having access to the medium and screwing with (spoofing or otherwise) the update "undetected" (consider the wireless case as well as how easy it is to have a hostile client sitting "somewhere" on a network...) >> - The protocol should be as light as possible -- but >> no lighter! ;-) > > This feeds back to the size of the images. TFTP is about as simple as > you can get using a generic farm server, but TFTP does not have any > way to request a partial transfer (e.g., a "head" command). Of > course, for version checking, you can hack your client to start the > transfer and then abort after fetching the predetermined image header. I was figuring I could just have the client request whatever blocks it wants in whatever order, etc. E.g., TFTP doesn't *force* me to request blocks in sequential order... >> - Minimize unnecessary network traffic as well as >> load on the server (the goal is for the user *not* >> to notice this activity -- though I am not trying >> to "keep it secret") > > How often do you need to check? How often do you anticipate a planned > update (as opposed to an emergency bug fix)? For this "throwaway" application, I doubt I will update the code more than once or twice (beyond testing). I don't plan on bugs :> and think I have put the features I want/need into the initial release. What I am more concerned with are the apps that follow. Those are so big that updates can be much more frequent. The more pressing issue for those apps is turnaround time. I.e., the update would want to start as soon as it was available (on the server). So, checking "once a day" would be inappropriate. Once minute might be nice as the expected delay to update start would only he half that (recall that those devices would need much longer to perform their entire update). The idea (mentioned elsewhere in this thread) of having devices collaborate to: - reduce the polling rate "per device" to keep the overall polling rate (seen by the update server) constant - share discovery of valid updates with other devices >> This will hammer the network pretty hard. Granted, >> the individual segments for each device will see >> modest traffic -- but the server's segment will >> quickly max out if multiple devices do this >> simultaneously (e.g., powering up together). >> This would necessitate a second layer of the >> protocol to randomize/defer such competition. :< > > Again, how many devices and how much bandwidth? > >> But, it seems like this *still* requires server side >> locking so the file isn't changed *while* it is being >> doled out. E.g., something like TFTP would be >> inappropriate as it doesn't reliably lock the file >>from one packet to the next. > > Use a "well known" public name and hide the file updates behind hard > links. > > Changing a hard link is an atomic process so there will be no race > conditions with the update - a process trying to open the file through > the link will either get some version of the file or fail if the open > call sneaks between unlinking the old file and linking the new. > > So updating the server then is a 3 step (scriptable) process. You > upload the new version alongside the old, change the well known hard > link to point to the new version, then delete the old version. <frown> Requires someone do something "special" on the server. When that someone (IT) screws up, The Boss doesn't see it as *his* problem. The IT guy *claims* he did everything right and there must be something wrong with the update software or the devices. So, *vendor* gets a frantic call from an angry customer complaining that the system has been "down" for "over an hour now"... It has been suggested that a fix for this might be to deploy a vendor supplied "update server" with the installation just so this gets done "right" (regardless of whatever OS's the customer has on site) > Of course, if the old version happens to be in use it will persist > until all handles to it are closed. > > Both Unix/Linux and Windows (NTFS) support hard links, so you have a > wide choice of server platforms. > >> Shirley, this sort of thing has been done before (?). > > Don't call me "Shirley". Roger!
From: George Neuner on 11 Mar 2010 19:50
On Wed, 10 Mar 2010 23:56:44 -0700, D Yuniskis <not.going.to.be(a)seen.com> wrote: >George Neuner wrote: >> On Mon, 08 Mar 2010 12:34:53 -0700, D Yuniskis >> <not.going.to.be(a)seen.com> wrote: >> >The projects that I am "practicing for" have much larger images >(1 - 4G) -- though generally have more resources to call on >at run time, so... <shrug> > >I'm currently playing on a 100Mb wired network. But, I am >looking for a solution that will scale up to Gb and down to >"wireless" rates. > >> What is the overall structure? Does the new image reset the device or >> is it a hot patch that needs to continue using existing data >> structures? > >Everything tries to run hot if at all possible. E.g., if >conditions coincidentally arise that would cause some piece >of code to rely on resources that aren't available during the >update, then things might not quite work as they would if the >update process was not active (but this would be unlikely). > >> Is it feasible to have the client load a new image concurrently with >> the old and switch at some well defined point(s)? > >Not the whole image. I am structuring the application so I >can swap out pieces of it and update it incrementally. E.g., >update the RTOS, then switch it in; update parts of the >library, then switch them in; etc. That's definitely the right approach, particularly if you want to scale to low wireless rates ... on 100Mb WiFi a 1GB file takes minutes to transfer even with a single client and no interference. You can just forget about using something like 3-G cellular. >>> - The protocol can't be spoofed by "unfriendlies". >> >> There's only one real solution to spoofing and/or "man in the middle" >> attacks and that is to encrypt the image binaries. You want it to be >> as hard as possible for someone to decompile your application and >> figure out how to spoof it. > >The sources for the application will be available. > >So, shared secret/public key is the only way to do the encryption. >The problem there is it forces all devices to have the same >keys (or, I have to build special images for each device). Multiple devices sharing the same key is not a problem. Although one of the keys is "public" and the other "private", it actually doesn't matter which is which: public(private(text)) == private(public(text)). The point is to authenticate the origin of the image - the client's "public" key can only decrypt a file encrypted by your private key - successfully decryption is the authentication. (Of course you need to verify by signature that the decryption was successful.) You only need to worry about more complicated key handling if you are creating programming tools for end users that themselves will need to worry about spoofing. AFAIHS, your previous comments didn't indicate that. Regardless, it neither desirable nor necessary to use public key encryption for the whole image (or its parts). Symmetric single key encryption (like DES, AES, etc.) requires fewer resources and runs faster. The only parts that require public key for secure communication and authentication are the symmetric key(s) and image signature(s). >The more realistic problem is to guard against something >messing with the image or process and effectively leaving >the device(s) in "half updated" states. So, it is even >more important that updates be supported "in pieces" >(otherwise the window of opportunity for screwing things up >is just too big) > >> Building on the idea of embedding the signature in a header, you could >> also embed a decryption key for the image. The header itself should >> be encrypted with a public key algorithm so that the client can verify >> that _you_ created the image. (Obviously the whole image could be >> public key encrypted, but it isn't necessary ... symmetric key >> algorithms are more performant. Successfully completing the process >> of decrypting the header, extracting the image decryption key, >> decrypting and verifying the image signature proves that the whole >> thing is legitimate.) > >Yes, but it would have to be done for each "piece". True, but signatures for all the pieces can be bundled for en-mass checking. If the pieces need to be encrypted, you only need to provide one decryption key for all of them. >> And don't store the decrypted image on the device ... design the >> update process so that the local image can be decrypted into RAM and >> reuse it to do the startup. > >Huh? I *think* you are assuming there is enough RAM to >support the decrypted image *and* my run-time data requirements? >If so, that's not the case -- far more code than data. I only assumed that the device stored the program for cold start and that you wanted to keep the code secret. However, since the source will be available there is no particular reason to encrypt images at all - runtime image encryption is typically only done to protect trade secrets. What I was saying was that the decryption mechanism could be reused in the general program loader, enabling the image(s) to remain encrypted on storage. That way, it would be much harder to gain access to and reverse engineer the code. >I'm not too worried about people reverse engineering devices. >Rather, I am more concerned with someone having access to >the medium and screwing with (spoofing or otherwise) the >update "undetected" (consider the wireless case as well as >how easy it is to have a hostile client sitting "somewhere" >on a network...) So then the only things that require authentication are the image signatures. You don't need to encrypt the image binaries at all - just the image signature bundle. >>> - The protocol should be as light as possible -- but >>> no lighter! ;-) >> >> This feeds back to the size of the images. TFTP is about as simple as >> you can get using a generic farm server, but TFTP does not have any >> way to request a partial transfer (e.g., a "head" command). Of >> course, for version checking, you can hack your client to start the >> transfer and then abort after fetching the predetermined image header. > >I was figuring I could just have the client request whatever >blocks it wants in whatever order, etc. E.g., TFTP doesn't >*force* me to request blocks in sequential order... It has been a long time since I've read any of the relevant RFCs, so I may be completely wrong here (and please feel free to correct me 8), but AFAIK, none of the popular file copy/transfer protocols allow random access. My understanding is that protocols like FTP, TFTP, Kermit, etc. just send file blocks in order and wait for the receiver to ACK/NAK them. Some of the sliding window implementations allow OoO within the window but, AFAIK, none allows arbitrary positioning of the window or advancing the window beyond a missing block. The so-called "restartable" protocols periodically checkpoint the file/window position and restart from the last checkpoint if the same client re-attaches and requests the same file within some timeout period. If you want/need essentially random access without needing your own server application you might be forced into a remote file system protocol (like SMB, NFS, etc.). I have no experience implementing such protocols ... I've only used them. >>> - Minimize unnecessary network traffic as well as >>> load on the server ... > >What I am more concerned with are the apps that follow. >Those are so big that updates can be much more frequent. True, but that situation is mitigated by piece wise updates. >The more pressing issue for those apps is turnaround time. >I.e., the update would want to start as soon as it was >available (on the server). So, checking "once a day" would >be inappropriate. Once minute might be nice as the expected >delay to update start would only he half that (recall that >those devices would need much longer to perform their entire >update). Downloading a small header (up to 1 UDP block) once every couple of minutes would not be too bad - at least for a limited number of devices and images (I'm presuming here that "like" devices can use the same image?). Obviously it would be far better to broadcast/multicast that updates are available and forgo client polling at all ... but that requires an intelligent server. >The idea (mentioned elsewhere in this thread) of having >devices collaborate to: >- reduce the polling rate "per device" to keep the overall > polling rate (seen by the update server) constant >- share discovery of valid updates with other devices Couple of possible solutions: - have all the "like" devices on a network segment (i.e. those that can share an image) cooperate. Use a non-routable protocol to elect one to do the polling and have it tell the others when an update is available. - run the devices in promiscuous mode and have them "sniff" update checks. This way when any device on the net polls, all devices on the segment see the result. - combine election and sniffing so only one device actually talks to the server and the others eavesdrop on the conversation. Sniffing might be more complicated than it's worth - ideally you'd like to extend it to cover the whole update process. But it may be worth doing for update checks because the file name, block number, etc. could be fixed and so easy to search for in sniffed packets. >> Use a "well known" public name and hide the file updates behind hard >> links. >> >> Changing a hard link is an atomic process so there will be no race >> conditions with the update - a process trying to open the file through >> the link will either get some version of the file or fail if the open >> call sneaks between unlinking the old file and linking the new. >> >> So updating the server then is a 3 step (scriptable) process. You >> upload the new version alongside the old, change the well known hard >> link to point to the new version, then delete the old version. > ><frown> Requires someone do something "special" on the server. >When that someone (IT) screws up, The Boss doesn't see it as *his* >problem. The IT guy *claims* he did everything right and there >must be something wrong with the update software or the devices. >So, *vendor* gets a frantic call from an angry customer complaining >that the system has been "down" for "over an hour now"... Yes. But the key is that it is scriptable. Bleary eyed, overworked IT person or dumbass noob makes no difference ... it's hard to get much simpler than "<script> filename". If you must have a GUI you can use Javascript in a web browser so the (l)user only needs to specify the file (or better yet drop it on the browser page). In any event, the server update process can be stupid friendly. George |