There’s something deeply human about the way we learn to play video games. We look at the screen, press the buttons, and maybe—if we’re lucky—pick up a trick or two by watching someone else play first. Now a developer known as [tryfonaskam] has been trying to teach an artificial intelligence to learn the exact same way.
The result is PILA—short for Polytrack Imitation Learning Agent—an AI built to play a simple racing game called PolyTrack. What makes this different from the usual approach is that nobody hard-coded the AI’s driving skills.
Instead, PILA trains itself through supervised learning. It watches the game state by capturing the screen, and it monitors the keyboard inputs made by human players as they navigate the tracks. From that data, it learns to mimic the behavior, and eventually to play the game entirely on its own.
When the model is actually playing, it receives live frames from the graphics engine, predicts the appropriate actions, and makes the right keyboard inputs to steer the car through the course. Behind the breakthrough is a lineage of similar efforts.
The project reminds us of earlier attempts to teach a raw AI how to play Trackmania, and of the Drivatar technology that has appeared in the Forza series of racing games. “Never did I ever think that Polytrack would show up on Hackaday!” the developer wrote. “Amazing stuff!” For now, PILA is focused on one simple racing game.
But the approach, learning from screen capture and keyboard inputs, points toward a future where AI might learn to interact with any software interface the same way a person would. By watching. By imitating.
By trying again.
