An AI named PILA learned to drive a car in a video game the same way a human would: by watching the screen and pressing the right keys. The developer [tryfonaskam] built it. The method is called supervised learning.
No one taught it the physics of steering or braking. It just watched.
PILA stands for Polytrack Imitation Learning Agent. It plays a simple racing game called PolyTrack. The AI captures the screen image in real time.
It records what keys a human player presses. From that data, it builds a model.
When the model runs, it looks at live frames from the graphics engine, predicts the correct action, and sends the right keyboard inputs. The car drives itself. The developer’s reaction was surprise.
“Never did I ever think that Polytrack would show up on Hackaday!” [tryfonaskam] wrote. “Amazing stuff!” This is not the first time someone has tried this. The report notes earlier attempts to teach a raw AI how to play Trackmania.
There is also the Drivatar technology used in the Forza series of racing games. Those systems let AI drivers learn from player behavior.
PILA follows that same lineage but works from pure screen capture and keyboard monitoring. What matters here is the interface. Most AI systems in games are built into the game engine.
They have direct access to data like car position, track layout, and speed. PILA does not have that.
It sees what a person sees. It presses what a person presses. That constraint is the point.
If an AI can learn to play a racing game by watching a screen, the same approach could be applied to any software. A word processor. A spreadsheet.
A flight simulator. The AI would not need special access or an API.
It would just need to see the screen and know what keys do what. The project is small. It works on one simple game.
But the logic scales. Teaching an AI to interact with software by imitation rather than by programming opens a door.
That door leads to systems that can learn new tools without being rebuilt. The developer did not hard-code driving skills. The AI learned them.
There is a difference between knowing a rule and learning a behavior. PILA learned the behavior. It watched a human drive a car around a track.
It copied the inputs. It got better.
That is how people learn too. We watch. We imitate.
We fail. We try again.
The AI does the same thing, only faster and with no frustration. The report frames this as a breakthrough. It is a small one.
A racing game is a controlled environment. The track does not change. The car handles the same way every time.
Real-world driving is not like that. Neither is most software.
But the principle holds. Screen capture plus input monitoring plus supervised learning equals an AI that can act like a human operator. For now, PILA stays in PolyTrack.
The developer may expand it. Other developers may copy the method.
The idea is out there. Teach an AI by showing it, not by coding it. That is the real story.
