Recently, one of my students found a very nice example of how Evolution can be tricky, and why we should never rely on numerical indicators alone when doing research.
This student is working on Evolutionary Computation for soft robots. The idea is to use Island Models to improve the reliability of virtual robots that were evolved.
Before implementing the complex Island Model, I asked the student to start with a simple task using standard Genetic ALgorithms. When the student completed the task, they sent me the following results graph:
This graph, which shows the fitness of the best individual over the generations, looks exactly what we would expect from a well working GA: At first there is a low value of fitness, which increases as the generation progresses, eventually stabilizing over a higher value.
However, I knew that this particular problem was tricky. So I asked the student to produce a video of the best individual found by the GA:
Oops! As you can see in the animation above, GA actually found a bug in the physics simulation, and evolved a robot that could exploit this bug to fly away! That is very cool, but it does not tell us whether the robot is adapting to this specific task, as we wanted.
This is actually so common in Evolutionary Computation, that a few years ago a bunch of researchers got together to publish a paper with several remarkable cases of this kind of glitch exploiting. "The Surprising Creativity of Digital Evolution" in the Artificial Life Journal. Highly recommended read!
Anyway, this case highlights the problems of what I call, "Kaggle Science" - i.e., focusing your research efforts so much in raising an specific measure, that you end up with an actually bad result that hides behind a "high score".
For problems like virtual robot evolution, it is usually easy to find these cases, because the problem has such good visualization. However, the same could occur in more complex and opaque optimization problems.
This shows, how important it is to have ways to visualize the results of optimizers, classifiers, etc, and sanity check them.
But for now, enjoy the flying robots!
Alberto Santos DuBot.