API for aggregator of asynchronous calls
in [JavaScript]
|
From: Scott Sauyet on 29 Jun 2010 16:57 I needed a technique to aggregate the results of several asynchronous calls. Obviously there are many ways of doing this. I wrote an Aggregator constructor function and was wondering if the API this presents makes sense to people here. Here is some demo code that uses this function: var agg = new Aggregator(); var myHandler = function(data, statuses, errors) { // statuses = {"blue": "completed", "red": "completed", // "yellow": "completed"} // errors = {} // // data = {"red": { /* ... */}, "blue": { /* ... */ }, // "yellow": { /* ... */ }} // do something with data.blue, data.red, data.yellow } agg.onComplete(myHandler); // Or var agg = new Aggregator(myHandler); agg.start("blue"); agg.start("red"); agg.start("yellow"); getJsonAjax({ url: "blue/", success: function(data, status) { agg.complete("blue", data); }, failure: function(status, err) { agg.error("blue", status); } }); getJsonAjax({ url: "red/", success: function(data, status) { agg.complete("red", data); }, failure: function(status, err) { agg.error("red", status); } }); getJsonAjax({ url: "yellow/", success: function(data, status) { agg.complete("yellow", data); }, failure: function(status, err) { agg.error("yellow", status); } }); (I don't actually have a getJsonAjax() function, but it should be obvious enough what it's meant to do.) The user associates each asynchronous process with a String (above "red", "blue", and "yellow"). When each process completes, the relevant returned data is supplied with the complete() call. The completion handlers accept three parameters. The first represents the actual results of the various asynchronous calls, the second, the status flags of each call ("started", "completed", "error", possibly "aborted"), and the third, any error messages generated by these calls. Any actual combination of the results is left to the caller, presumably in the completion handlers. You can pass such handlers to the onComplete() method or to the constructor. I do have an implementation of this, but at the moment, I'm mostly concerned with whether the API makes sense. Is there a cleaner way to do this? Is this general enough? Are the String identifiers for the calls robust enough? Should the constructor also accept the identifiers? What suggestions do you have? -- Scott
From: Asen Bozhilov on 29 Jun 2010 17:58 Scott Sauyet wrote: > I needed a technique to aggregate the results of several asynchronous > calls. Obviously there are many ways of doing this. I wrote an > Aggregator constructor function and was wondering if the API this > presents makes sense to people here. > Here is some demo code that uses this function: > > var agg = new Aggregator(); > > var myHandler = function(data, statuses, errors) { > // statuses = {"blue": "completed", "red": "completed", > // "yellow": "completed"} > // errors = {} // > // data = {"red": { /* ... */}, "blue": { /* ... */ }, > // "yellow": { /* ... */ }} > > // do something with data.blue, data.red, data.yellow > } > > agg.onComplete(myHandler); > > // Or var agg = new Aggregator(myHandler); > > agg.start("blue"); > agg.start("red"); > agg.start("yellow"); > getJsonAjax({ > url: "blue/", > success: function(data, status) { > agg.complete("blue", data); > }, > failure: function(status, err) { > agg.error("blue", status); > } > }); Leave `Aggregator' to care about your requests. I do not see any reasons for: agg.start("yellow"); First I should push in aggregated request and after that I should explicitly inform `Aggregator' for completion of that request: agg.complete("blue", data); That is a little bit inconsistency. The following is my point of view of the problem. function Aggregator() { var that = this; this._aggregated = []; this._count = 0; this._handler = function (req, data) { that._count--; if (that._count == 0) { that.onComplete(); } }; } Aggregator.prototype = { onComplete : function () {}, push : function (reqWrapper) { reqWrapper.success = this._handler; this._count++; this._aggregated.push(reqWraper); }, start : function () { for (var i = 0, len = this._aggregated.length; i < len; i++) { this._aggregated[i].send(); } } }; function RequestWrapper() {} RequestWrapper.prototype.send = function () { //... }; var agg = new Aggregator(); agg.onComplete = function () { //... }; agg.push(new RequestWrapper()); agg.push(new RequestWrapper()); agg.push(new RequestWrapper()); agg.start();
From: Stefan Weiss on 29 Jun 2010 19:00 On 29/06/10 22:57, Scott Sauyet wrote: > I needed a technique to aggregate the results of several asynchronous > calls. Obviously there are many ways of doing this. I wrote an > Aggregator constructor function and was wondering if the API this > presents makes sense to people here. (snip) > I do have an implementation of this, but at the moment, I'm mostly > concerned with whether the API makes sense. Is there a cleaner way to > do this? Is this general enough? Are the String identifiers for the > calls robust enough? Should the constructor also accept the > identifiers? What suggestions do you have? I don't see any major problems with the API you suggested. If you ask 10 people to design an Aggregator object, you'll very likely end up with 10 different designs, and the one you come up with yourself will always feel the most natural. But since you asked for feedback, here are some random unsorted thoughts that came to my mind: - If "agg = new Aggregator(handler)" and "agg.onComplete(handler)" are equivalent, then the onComplete method is misnamed, because the handler will also get called on failures (and possibly other status changes). "setHandler" might be a better name. - One thing I've learned in a project where we had very complex and highly automated forms with more than one layer of "aggregators" before the actual Ajax transport layer, is that the handler function won't always be interested in the complete current state of all requests, but in the changes. From the handler's perspective, it would ask itself - why am I being called now? - has a job completed? - has a job failed? - is there a new job to consider? etc. To answer these questions it would have to keep an internal copy of the previous state for comparison, but that would only duplicate what's already available in the Aggregator object. You could find a way to pass delta information to the handler. It could still get at the complete state information if it has a reference to the Aggregator (in any of several possible ways, or maybe even passed as an argument in the callback), and the Aggregator exposes methods to query its state. - I don't know how general you want to keep the Aggregator. Code reuse was never my strong suit; I tend to copy and adjust instead of writing one perfect class/object/function and leaving it untouched afterwards. From that standpoint, I would probably integrate the Ajax requests into the Aggregator. In your example, they all look similar, so with an internal getJsonAjax function in the Aggregator, the calls could be changed from agg.start("blue"); agg.start("red"); ... getJsonAjax({ url: "blue/", success: function(data, status) { agg.complete("blue", data); }, failure: function(status, err) { agg.error("blue", status); } }); getJsonAjax({ url: "red/", success: function(data, status) { agg.complete("red", data); }, failure: function(status, err) { agg.error("red", status); } }); ... to agg.start("blue", "blue/"); agg.start("red", "red/"); - I guess there are parts of the API you haven't posted. From what I've seen, the Aggregator could almost be considered as semantic sugar for a simple hash-like registry object: // create "Aggregator" var agg = {}; // start "red" job agg.red = { status: "pending" }; myHandler(agg); getJsonAjax({ url: "red/", success: function(data, status) { agg.red = { status: "completed", data: data }; myHandler(agg); }, failure: function(status, err) { agg.red = { status: "failed", error: err }; myHandler(agg); } }); Okay, that's a little too simplistic, and not very pretty, either ;-) At the other end of the spectrum, you could go in a more object oriented direction and create Job objects which would know how they should be transported, and which an Aggregator could then run and manage. This could also help to eliminate the "stringiness" of your job identifiers. PS: I just saw Asen's reply. It looks like his RequestWrapper objects are similar to what I was trying to say in my last paragraph. -- stefan
From: Scott Sauyet on 30 Jun 2010 10:06 Asen Bozhilov wrote: > Scott Sauyet wrote: >> I needed a technique to aggregate the results of several asynchronous >> calls. Obviously there are many ways of doing this. I wrote an >> Aggregator constructor function and was wondering if the API this >> presents makes sense to people here. [ ... ] > Leave `Aggregator' to care about your requests. I do not see any > reasons for: > > agg.start("yellow"); > > First I should push in aggregated request and after that I should > explicitly inform `Aggregator' for completion of that request: > > agg.complete("blue", data); > > That is a little bit inconsistency. First off, thank you Asen for taking the time to reply. I'm not sure I seen an inconsistency here, but it certainly might seem awkward. > The following is my point of view of the problem. > > function Aggregator() { > var that = this; > this._aggregated = []; > this._count = 0; > > this._handler = function (req, data) { > that._count--; > if (that._count == 0) { > that.onComplete(); > } > }; > > } > > Aggregator.prototype = { > onComplete : function () {}, > > push : function (reqWrapper) { > reqWrapper.success = this._handler; > this._count++; > this._aggregated.push(reqWraper); > }, > > start : function () { > for (var i = 0, len = this._aggregated.length; i < len; i++) { > this._aggregated[i].send(); > } > } > > }; > > function RequestWrapper() {} > RequestWrapper.prototype.send = function () { > //... > > }; > > var agg = new Aggregator(); > > agg.onComplete = function () { > //... > > }; > > agg.push(new RequestWrapper()); > agg.push(new RequestWrapper()); > agg.push(new RequestWrapper()); > > agg.start(); (Sorry for the long quote, can't find any way to trim it without removing something essential.) This would certainly be clearer and cleaner to use for the problem I presented in my demo. It probably is enough for my current needs, too. But there is a real possibility that I will need some additional features I didn't mention in my initial message, but which did prompt the API I used. First of all, sometimes the asynchronous process I'm waiting for might be user input rather than an AJAX call. Second, some calls might lead me to make additional ones, and I want my wrap-up function to run only after all the results are in. I might still be able to get away without listing the calls, though, and only counting, but I don't think I could use the push-push-push-start system, but rather one in which each process is started and the aggregator checks after each one completes whether there are still any processes still running. I will look into whether I need the labels at all. I might be able to avoid them. Thanks again, -- Scott
From: Scott Sauyet on 30 Jun 2010 10:38
Stefan Weiss wrote: > On 29/06/10 22:57, Scott Sauyet wrote: > >> I needed a technique to aggregate the results of several asynchronous >> calls. [ ... ] >> I do have an implementation of this, but at the moment, I'm mostly >> concerned with whether the API makes sense. Is there a cleaner way to >> do this? Is this general enough? Are the String identifiers for the >> calls robust enough? Should the constructor also accept the >> identifiers? What suggestions do you have? > > I don't see any major problems with the API you suggested. If you ask 10 > people to design an Aggregator object, you'll very likely end up with 10 > different designs, and the one you come up with yourself will always > feel the most natural. Of course. That's why when I don't have others to immediately work with an API, I like to ask knowledgeable people if the API makes sense. > But since you asked for feedback, here are some random unsorted thoughts > that came to my mind: > > If "agg = new Aggregator(handler)" and "agg.onComplete(handler)" are > equivalent, then the onComplete method is misnamed, because the handler > will also get called on failures (and possibly other status changes). > "setHandler" might be a better name. Absolutely right. I changed my stop(name, data) method to complete(name, data) at the last minute because I was using the status values of "started", "completed", or "errored" for the various tasks and it made sense to make the verbs match these statuses. I didn't consider that this is too close to the onComplete() method. I'm not sure I like setHandler() because I'm allowing multiple callback functions (quite possibly for no good reason.) I think I'll keep the onComplete() and rename complete(name, data), but I'm not sure to what. Perhaps I should use startTask() and stopTask(), although "errorTask()" does not roll off the tongue properly. > One thing I've learned in a project where we had very complex and highly > automated forms with more than one layer of "aggregators" before the > actual Ajax transport layer, is that the handler function won't always > be interested in the complete current state of all requests, but in the > changes. From the handler's perspective, it would ask itself > > - why am I being called now? > - has a job completed? > - has a job failed? > - is there a new job to consider? > etc. > > To answer these questions it would have to keep an internal copy of the > previous state for comparison, but that would only duplicate what's > already available in the Aggregator object. You could find a way to pass > delta information to the handler. It could still get at the complete > state information if it has a reference to the Aggregator (in any of > several possible ways, or maybe even passed as an argument in the > callback), and the Aggregator exposes methods to query its state. That is a fascinating concept. I've had systems where writing this in a general way would have been very useful. For my current needs and those anticipated relatively soon, this would be overkill, but I can almost see how it could be done elegantly with a fairly simple API. I think if I find a little spare time, I might try that just for the fun of it. > I don't know how general you want to keep the Aggregator. Code reuse was > never my strong suit; I tend to copy and adjust instead of writing one > perfect class/object/function and leaving it untouched afterwards. From > that standpoint, I would probably integrate the Ajax requests into the > Aggregator. Oh, I definitely want it more general than that. One AJAX call might actually add a new task to the aggregator. Other asynchronous processes for the aggregator might involve waiting for user input. The main thing is that I want a simple way to know when all the data needed for processing, however it's gathered from disparate sources, is available. So it should be fairly general. > I guess there are parts of the API you haven't posted. From what I've > seen, the Aggregator could almost be considered as semantic sugar for a > simple hash-like registry object: > > // create "Aggregator" > var agg = {}; > > // start "red" job > agg.red = { status: "pending" }; > myHandler(agg); > > getJsonAjax({ > url: "red/", > success: function(data, status) { > agg.red = { status: "completed", data: data }; > myHandler(agg); > }, > failure: function(status, err) { > agg.red = { status: "failed", error: err }; > myHandler(agg); > } > }); > > Okay, that's a little too simplistic, and not very pretty, either ;-) To some extent that's right. But the trouble with such a hash is the lack of knowledge of overall completion. In such a system, that calculation would have to be pushed down into the handler function. Avoiding that is really the motivator for this. I've done such things fairly often, but the scenarios are getting more complicated, and if certain expected requirements come through, they will get far more complicated. > At the other end of the spectrum, you could go in a more object oriented > direction and create Job objects which would know how they should be > transported, and which an Aggregator could then run and manage. This > could also help to eliminate the "stringiness" of your job identifiers. > > PS: I just saw Asen's reply. It looks like his RequestWrapper objects > are similar to what I was trying to say in my last paragraph. Yes, I hadn't really considered going that far, but it might really be useful. I'm not quite sure of the API, though. Are you thinking something like this?: var agg = new Aggregator(myHandler); var job1 = new Job(function() { getJsonAjax({ url: "red/", success: function(data, status) {job1.stop(data);}, failure: function(status, err) {job1.error(status);} }); }); agg.add(job1); Thank you for your thoughtful and detailed reply, -- Scott |