Spring 2010 MATLAB Contest, April 28 - May 5
in [Matlab]
|
Prev: Single number into vector elements
Next: neural network
From: Alan Chalker on 11 May 2010 23:09 "Jan" <thisis(a)notanemail.com> wrote in message <hsb2vn$hjs$1(a)fred.mathworks.com>... > So, I could upload the result from running the submissions against the test suite also on the FileExchange: http://www.mathworks.com/matlabcentral/fileexchange/27554-matlab-sensor-contest-data-set-run-on-test-data > > > Cheers > Jan I used Jan's code as a 'module' for my comment submission code and inserted all the result data he calculated into the individual entry comments. You can now just browse to a specific entry on the website and see what the test suite results were. For example, the comment for Hannes winning entry is: "This entry gets a result of 19556985 on the test suite (5678335 more than the contest suite). It has a ranking of 1698 compared to all other entries run against the test suite according to the data set provided by Jan Keller. " Likewise, I also ran a module that commented all the leaders. Here's an example comment from one of the cyclist's final entries (#4515): "This was the 278th entry to take the overall contest scoring lead. It improved upon entry #4512 by 0.015% (4.3 points). It stayed in the lead for 2 mins, 35 secs before being replaced by entry #4526. " I'm interested in hearing feedback on this. Is this of value? Potentially I could be adding these types of comments in near real-time during future contests. Obviously the ultimate goal is to handle comments and code legacy, but unless the comment length issue is resolved that's not really feasible right now.
From: Hannes Naudé on 12 May 2010 02:35 Alan:"I'm interested in hearing feedback on this. Is this of value? Potentially I could be adding these types of comments in near real-time during future contests. Obviously the ultimate goal is to handle comments and code legacy, but unless the comment length issue is resolved that's not really feasible right now." I definitely think this is of value. If you're doing this in near real-time, then you'll run into problems with statements like "It has a ranking of 1698 compared to all other entries..." since these rankings will be changing all the time and you don't want to repost all the comments each time this happens. I do wish that it was possible to do things like "sort by test suite result". Since people don't comonly overfit to the test suite (allthough public visibility of test suite scores may change this) sorting by test suite score should give a better indication of where algorithmic improvements took place. These improvements (that would otherwise have been lost) can then be picked up and tweaked until they are competitive. Regarding the comment length issue. What do you think of the "URL comment" solution that I suggested? Obviously this requires access to hosting resources, but I'm sure this can be resolved (TMW might even be willing to host it for us). I don't think the bandwidth required will be excessive. Hannes
From: Yi Cao on 12 May 2010 04:05 "Alan Chalker" <alancNOSPAM(a)osc.edu> wrote in message > I used Jan's code as a 'module' for my comment submission code and inserted all the result data he calculated into the individual entry comments. You can now just browse to a specific entry on the website and see what the test suite results were. For example, the comment for Hannes winning entry is: > Thank Alan for doing this. It is valuable. From these comments, I noticed an error in the statistics page, hence in the comments: My entry (ID 4475) was marked as the highest rank 1st, but it was not included in the All the Leaders list. This was the entry cloned by 27 entries in the final hour. So the highest rank record is correct. This indicates that the code to produce the statistics page has some bugs. The contest team may wish to have a look. Yi
From: Alan Chalker on 12 May 2010 11:39 "Hannes Naudé" <naude.jj+matlab(a)gmail.com> wrote in message <hsdi79$5bo$1(a)fred.mathworks.com>... > Regarding the comment length issue. What do you think of the "URL comment" solution that I suggested? Obviously this requires access to hosting resources, but I'm sure this can be resolved (TMW might even be willing to host it for us). I don't think the bandwidth required will be excessive. > Hannes: Regarding the URL comments, it's an interesting idea. I did some testing and unfortunately the comments field currently does NOT accept html tags. Thus while you can put a URL in the comment, it won't be clickable, somewhat diminishing the value of that.
From: Alan Chalker on 12 May 2010 11:49
"Yi Cao" <y.cao(a)cranfield.ac.uk> wrote in message <hsdng1$992$1(a)fred.mathworks.com>... > > My entry (ID 4475) was marked as the highest rank 1st, but it was not included in the All the Leaders list. > > This was the entry cloned by 27 entries in the final hour. So the highest rank record is correct. This indicates that the code to produce the statistics page has some bugs. The contest team may wish to have a look. > Yi: Actually the stats page is correct, the highest rank is wrong. I encountered this previously during the longevity phase with entry 4012 from Amitabh and mentioned it in a posting above. Here's what the details of the issue are: #4369 got a score of 27799.8 and took the lead #4475 also got a score of 27799.8, but since it didn't BEAT #4369 it didn't take the lead. However the submission list page for some reason showed it in first place #4512 later got a score of 27799.7, overtaking both. I suspect there is a minor coding error in the submission listing / highest rank code that uses a '>=' comparison instead of the '>' comparison it should be using. Alternatively, it's using more decimal points in the calculation, which aren't available to the statistics code. Regardless, an entry needs to overtake, not equal another in order to take the lead. This situation happened several times during the contest. |