Goodhart’s law:
When a measure becomes a target, it ceases to be a good measure.
Much like IQ tests for humans are flawed too. Figuring out series of numbers or relations in a graphic representation, only tells how good you are at these specific tasks, and doesn’t provide a reliable picture of “general” intelligence.
Looks quite satisfying to me, otherwise, we can still create new tests … :
The tests cover an astounding range of knowledge, such as eighth-grade math, world history, and pop culture. Many are multiple choice, others take free-form answers. Some purport to measure knowledge of advanced fields like law, medicine and science. Others are more abstract, asking AI systems to choose the next logical step in a sequence of events, or to review “moral scenarios” and decide what actions would be considered acceptable behavior in society today.
This is the way:
