Spring 2010 MATLAB Contest, April 28 - May 5
in [Matlab]
|
Prev: Single number into vector elements
Next: neural network
From: the cyclist on 6 May 2010 13:56 "Hannes Naudé" <naude.jj+matlab(a)gmail.com> wrote in message <hruurp$e65$1(a)fred.mathworks.com>... > "the cyclist" <thecyclist(a)gmail.com> wrote in message > > Congratulations on a great victory, Hannes, and on your long-overdue grand prize win. Maybe Cobus was dragging you down before? ;-) > > Errm, if you'd like to know who's been dragging me down I'd refer you to the Blockbuster contest archive ;-) > Yes, I continue to watch my back since reading your commentary in that contest!
From: Sergey Y. on 6 May 2010 14:45 The problem of development of the code which works in general is the definition of “general”. For the contest “general” is the test set. One may say that set is not general enough. Then I will ask: what will make it more general? Adding more pictures? Then what about orientation preferences in natural scene pictures? Adding pictures of absolutely random pixels? Then do we need to make sure that statistics of random pixel is uniform in all possible ways? Any finite set of test boards will have distortions in one way or another. And we will not know those distortions until contest will uncover them. I do not think that we can use second test set to fix it. It will be just checking one bias set using another bias set. However what we may try to enforce is “uniform” overfitting of the test set. I suspect that currently tweaking results in overfitting of the most valuable examples at the expense of less valuable. Maybe we have to normalize each example score (one way is to normalize to the current best result for each picture) Obviously the Amy Ant contest practically resolves those issues. (Practically because result still depended on field configuration. Theoretically it should be tested on all possible fields). If we have that kind of contest in the future then my only recommendation would be to run new entries against all previous Kings of the Hill to make best solution more “universal”. Probably we even can reduce number of submissions to 1 per 1 hour. P.S. In addition, Amy Ant type contest will easily allow changing boards every day. And previous day battles could be shown every day for general public entertainment.
From: Hannes Naudé on 6 May 2010 14:48 "Alan Chalker" <alancNOSPAM(a)osc.edu> wrote in message >As a general comment to everyone asking for various changes to put the focus more >on the algorithm development side of things, I'd like to point out that that would >have the effect of reducing the overall number of participants in the contest. Most >of the user community who could potentially be involved doesn't have the time or >skill set to dive into the literature and try to develop new algorithms. >There is a need for that obviously, but if the overall purpose of the contest is to >increase awareness and excitement about using MATLAB, then making the contest >as accessible as possible to anybody is vital. Allowing tweaking is one way of doing >that. Yes Alan, we've heard this argument before, but the numbers aren't with you on this one. The stats page clearly indicates that darkness and twilight are more popular than daylight by a factor ~3 (or more if sock puppet accounts are taken into account). While its impossible to determine to what extent this trend is caused by excessive parameter tuning in daylight it would be incredibly naive to imagine that the two are wholly unconnected. On the other side of the equation, there is the "most of the user community" group that would supposedly abandon the contest if they were expected to do more than parameter tweaks. Looking at the same stats page we see a total of 5 participants with more than 200 entries. For each of these 5 participants I can point to a productive change they've applied to the code that improved (as opposed to just overfitted) it. So I don't know who these people who are unable or unwilling to do more than change 0.38 to 0.37 are. Maybe you'd like to give a few examples? >The biggest issue I had to deal with obviously involved the logging in to your >account in order to submit an entry. I spent hours trying to get urlread to work >with the new system. However in the end I couldn't because some of the login >pages utilize SSL, which urlread doesn't support. (NOTE to MathWorks.. could you >please consider adding that support as a feature request? I'd love to continue to do >this all in MATLAB) Similar experience here. I also devoted a couple of days to developing an autosubmitter. In previous contests I developed a mechanism for copy protection that would harm performance (typically by dropping out of the local minimum) if the code was edited or even if it was resubmitted identically. As an unintended side effect it would seriously screw with the contest diff. See http://www.mathworks.com/contest/splicing.cgi/diff.html?id1=45516&id2=45323 for an example. However using this mechanism requires that my code contains non-printing characters that get corrupted in the process of a copy-paste. So I could not submit code manually but had to go via an autosubmitter. Anyway, back to the point. I also realised that urlread wouldn't do the job, partly because of SSL, but also because of lack of support for cookies, which are used to keep track o you once you're logged in. I developed an autosubmitter using WaTiR in C# (which I don't actually know) only to realise that because WaTiR actually just automates IE, all the complexity of dealing with SSL, cookies, and malformed pages are handled for you, but on the other hand you are still unable to send hidden characters. :-(( So I started over using the System.Web assembly in C#. I did not do the login at all but just grabbed the cookies from IE's cache. This still did not work so I resorted to using a packet sniffer to see what I was doing different from IE. Turns out IE had some more cookies that weren't in its cookiefile as far as I could see. (If some network guru here could tell me where IE squirelled these away I would be eternally grateful) So I just copy pasted these into my code and voilla, I could submit. Only to learn that something in the contest machinery change has caused this to stop working. Even when sending the hidden characters without the urlencoding, they would somehow get corrupted/translated at the contest machine before the code got to execute. So that was the end of my autosubmitter venture. > Finally, to everyone who dismisses 'auto-code' as trivial and not within the spirit of >the competition, I respectfully disagree. I've ended up with hundreds of lines of >code spread out across dozens of functions that took me hours and hours to develop >over the past week. Dealing with all the data handling and error conditions and >whatnot requires a lot of effort. Multiple times I'd start it running and come back >later to find the program errored out due to some weird occurrence I hadn't taken >into account. It's a different sort of challenge from the underlying algorithm of the >contest problem, but it still requires a good understanding of MATLAB and thinking >through different approaches to a problem. Having coded an autosubmitter that is nowhere near as sophisticated as yours, I fully agree that what you did is not trivial. But it IS disruptive and it DOES change the nature of the game. And most of us agree that it changes the nature of the game for the worst.
From: Sergey Y. on 6 May 2010 15:17 "Hannes Naudé" <naude.jj+matlab(a)gmail.com> wrote in message <hrv2t4$d42$1(a)fred.mathworks.com>... > In previous contests I developed a mechanism for copy protection that would harm performance (typically by dropping out of the local minimum) if the code was edited or even if it was resubmitted identically. As an unintended side effect it would seriously screw with the contest diff. See http://www.mathworks.com/contest/splicing.cgi/diff.html?id1=45516&id2=45323 > for an example. However using this mechanism requires that my code contains non-printing characters that get corrupted in the process of a copy-paste. So I could not submit code manually but had to go via an autosubmitter. Anyway, back to the point. Brilliant !!!! Sergey
From: Yi Cao on 6 May 2010 15:52
"Hannes Naudé" <naude.jj+matlab(a)gmail.com> wrote in message <hru0tk$75i$1(a)fred.mathworks.com>... > > rand('seed',SomeParameterThatIdentifiesTheProblem); > lookup=[ 0.5 1;1 0]; > i=find(rand(1)<lookup(1,:),1,'first'); > blockSize = ExistingFormula+lookup(2,i)*DELTA; > Hanners, Thanks for sharing this great idea. By revising your 20 'Random Randomness' entries, I got the same loopup table your implemented in your winning code and the predicted result is exactly the same as the actual one show on the website. However, onething still puzzels me: in two solvers, you used two independent rand calls to determined the corresponding column of lookup table. In theory, these two calls should give two different columns, shouldn't they? If so, the above calculation cannot be right, can it? Yes, from this approach, almost any parameters can be overfitted to get the minimum results. We have to consider a workable scheme to make future contest more general. Contesters against contesters is a good idea. For example, the testsuit can contain a lager number of cases, say, 1000. Select 100 cases for which the current leading entry has the best results and calculate the results of these 100 cases. The new leading code have to beat the current leader on score based on these 100 cases. Once new leader determined, another set of 100 cases will be determined based on the new leading code. In this way we can ensure the leading score is monotonically decrease. However, since the actual suit is dynamically chaning, it will make overfitting difficult. Another idea regarding submission limit, we may introduce an average-waiting time scheme to punish multiple submission simutanously. Each submission has a waiting time counted from when it is submitted. If the same contester submit several entries, then the waiting time is the acutal time divided by the number of entries in the queue. Then, if there is another contester has an entry in the queue. Even the single entry is submitted later, the waiting time of this entry grows quicker then the first contester who has multiple entries in the queue. Hence, the entry of the second contester may get evaluated earlier. This may encourage contesters to keep the queue as short as possible. Yi |