1/ If you want a preview of how Tesla's RoboTaxis may behave on the road look no further than DeepMind's AlphaStar and its domination of Starcraft— hint: its beating 99.8% of all players.

2/ AlphaStar doesn't perceive the game through vision, instead it receives raw game data as an input and thus in effect has perfect "perception" of the environment.

3/ The neural net only has to output game decisions (limited to a human bandwidth mind you, RoboTaxi's won't have this limitation). The goal being: win the game while also not letting the opponent win.

4/ Imagine what will happen when Tesla's AutoPilot solves perception? Having a neural net output velocity and yaw changes in order to "win" (not crash) while also not being "beaten" (navigating to destination) is relatively straight forward given enough curated training data.

5/ This is also where reinforcement learning can play a large part in designing the driving policy... Simulate common environments from fleet weak spots, iterate with varying reward functions until emergent behaviour is better than human.

6/ The problem of course is that Tesla have not solved perception. However could they be getting close? The PlaidNet rewrite may provide important clues. By using 4 dimensions (3D space and time) any apparent signal in the input data should be easier to extract.

7/ The car will have an easier time understanding occluded objects, estimating intent, predicting drivable space and more. In essence the car is able to build a prediction of the future environment and work backwards to give accurate data at current time step.

8/ (This also lends itself to self supervision for learning, but that is another topic)

9/ Why is this important? Because the AutoPilot "vision" output getting very close to AlphaStar's input for both data quality and quantity.

10/ Once perception improvement reaches a critical mass, the automated driving characteristics will be as impressive as the long term strategic decision making displayed by Alphastar that many thought not possible.

@jpr007 thoughts? ;)