When a developer known as [tryfonaskam] set out to teach an AI to play a simple racing game called PolyTrack, the method was as much about how it learns as what it learns. The result is PILA—Polytrack Imitation Learning Agent—and it trains itself the way a human might: by watching the screen. PILA does not rely on hard-coded driving skills.
It uses supervised learning. The AI captures the game state by taking screenshots.
It monitors the keyboard inputs of human players as they steer through the tracks. From that data, it learns to mimic the behavior. Eventually, it plays the game entirely on its own.
When the model is actually playing, it receives live frames from the graphics engine, predicts the appropriate actions, and makes the right keyboard inputs to steer the car through the course. This is not a new concept in the field.
The project draws on earlier attempts to teach an AI how to play Trackmania. It also echoes the Drivatar technology that has appeared in the Forza series of racing games. For [tryfonaskam], the response was personal.
“Never did I ever think that Polytrack would show up on Hackaday!” the developer wrote. “Amazing stuff!” What matters to the wider community is the approach. PILA currently focuses on one simple racing game.
But the technique—learning from screen capture and keyboard inputs—points toward a future where AI might learn to interact with any software interface the same way a person would. By watching.
By imitating. By trying again. That future is not theoretical.
Institutions and researchers are paying attention. The lineage of this work includes academic and industry efforts to build AI that learns from raw screen data rather than from hand-crafted sensor feeds.
The Trackmania efforts were a proof of concept. The Drivatar system in Forza is a commercial application. PILA sits between them—a demonstration that imitation learning can work with minimal engineering overhead.
The developer’s approach is deliberately simple. No complex reward functions. No reinforcement learning loops.
Just screen grabs and key presses. The AI builds a model of what works by copying humans.
Then it runs on its own. That simplicity is the point. If an AI can learn to drive a car in a game by watching the screen, the same method could apply to other software.
A customer service interface. A spreadsheet.
A flight control system. The AI would not need special APIs or custom integrations. It would just need to see the screen and know what keys to press.
This is not a finished product. PILA is a project, not a product. But it is a concrete example of a direction the field is taking.
The developer did not announce a company or a funding round. They just showed the work.
The response from the community has been positive. The developer’s own surprise at the attention suggests this was a personal project, not a corporate initiative. That matters.
It means the tools for building imitation-learning agents are becoming accessible to individual developers, not just large labs. For now, PILA stays in its lane—literally.
It plays PolyTrack. But the method is general. That is what makes it worth watching.
