Former Tesla AI Head Explains Why They Removed Sensors; Others differ

In a recent interview, Andrej Karpathy, who used to be the head of AI for Tesla’s Autopilot and FSD products, explained his reasons for removing both radar and ultrasound from Tesla cars and never using LIDAR or maps. While Elon Musk is best known for making such statements, Karpathy has been his go-to person to back up that argument. However, Karpathy raised eyebrows when he took time off work earlier this year and eventually announced he was leaving her.

The main points of Karpathy:

1. Additional sensors add cost to the system and, more importantly, complexity. They complicate the software task and increase the cost of all data pipelines. They add risk and complexity to the supply chain and manufacturing.
2. Elon Musk has a philosophy of “the best is not a part” which is reflected throughout the car, e.g. B. when everything is done via the touchscreen. This is an expression of this philosophy.
3. Vision is necessary for the task (which almost everyone agrees on) and should be sufficient. When it’s enough, the cost of additional sensors and tools outweighs their benefits.
4. Sensors change as parts change or become available and unavailable. They have to be maintained and the software adapted to these changes. They also need to be calibrated for the fusion to work properly.
5. Having a fleet collecting more data is more important than having more sensors.
6. Processing lidar and radar creates a lot of bloat in the code and data pipelines. He predicts that other companies will also drop these sensors in time.
7. Mapping the world and keeping it up to date is far too expensive. You will not change the world with this limitation, you must focus on seeing, which is the most important thing. The roads are designed to be interpreted with foresight.

Complexity of sensor fusion

In my recent interview with Jesse Levinson, CEO and co-founder of Zoox, I asked him the same question. While he agreed that more sensors definitely mean more work and more noise, these problems are not insurmountable and are worth the benefit. He believes that if you’re smart and get your sensor fusion right, you can ensure that new data from sensors and conflicting data isn’t a disadvantage. While every input has noise, if you’re good you can pull the true signal out of it and win.

In general, other teams won’t necessarily disagree with too many points from Karpathy. Having multiple sensors and fusion adds significant complexity and cost. Many will even agree that one day visibility may be sufficient and these other sensors may be dropped. However, everyone (including probably Karpathy and Musk) would agree that vision is not enough today. Also, the others would find that it’s not at all clear when visibility will be sufficient. Karpathy and many others point out that humans drive primarily with sight, so it’s clearly possible, but the reality is that computers have nowhere near the power that human brains have to do this. Very few technologies work quite like the human mind—just because birds flap their wings doesn’t mean airplane designers follow those routes. It is more common to use different, sometimes superhuman abilities of machines to compensate for the lack of brain power of machines.

Tesla’s approach would be quite rare in the AI ​​world to intentionally limit a system to only the capabilities of human sensors and hope to adapt the human brain to work with those limited sensors.

Cost as driver or time to market?

This disagreement stems in part from the fact that Tesla is an automaker, and further from their goal of getting their system to work on their already-delivered cars, or at worst, a minor upgrade of their already-delivered cars. (That upgrade is already underway, and owners of old cars have seen a main processor upgrade with a second pending, and in some cases, camera replacements — and possibly a new camera system.)

Car manufacturers are very, very cost conscious. Anything they add to a vehicle increases the vehicle’s list price by 2x to 5x its cost. Anything they can take out contributes to their bottom line. The philosophy of removing parts makes sense here and has served Tesla well, although many drivers complain that in some cases they overdid it a bit.

However, this is less clear when a part is removed when the system will not work without that part. After Tesla removed radar support, they downgraded a number of features in Tesla Autopilot, and even a year later it hasn’t returned to the speeds it was capable of. Many Tesla owners complain that the no-radar system has much more frequent “phantom braking” events, where the vehicle sometimes brakes hard for obstacles that aren’t there or aren’t a problem.

Tesla’s new cars that ship without ultrasonics have removed almost all of the features of ultrasonics, such as: B. Park Assist, Auto Parking, Summon and more. They are promised, says Tesla, to return in the near future.

Most self-driving teams believe that the shortest route to deployable self-driving is using LIDAR, radar, and in some cases other sensors. They consider it the shortest and safest way, not the easiest and cheapest. Since they do not sell vehicles, these restrictions are not a priority for them. Zoox’s Jesse Levinson says the added cost of specialized sensors isn’t a barrier to a car being sold to consumers, since their custom robotaxi is heavily used and charges good fees.

But while cost is one factor, speed of development is the biggest. LIDAR today detects a large class of obstacles with absolute reliability, and at a level of reliability that you can stake your life on. Camera doesn’t, and while they likely will one day, the date when they will do so is unknown by both Tesla and other teams. The date when they will have low cost is much better known.

This question of when influences the software complexity. Getting cameras to deliver the required reliability is more work these days — so much more work that no one can do it yet. That a simpler system could make it possible in the future is not considered by most teams today. The leading teams are all investing billions of dollars and taking the cost of the added complexity. A theoretically simpler solution that doesn’t work yet is no simpler than a more complex but working one.

Of course, it should be noted that none of the other self-driving teams have a production deployment, although several pilot projects are being conducted in complex cities with no safety drivers on board. Previously, I’ve published a number of articles and videos on what the major remaining challenges teams are seeing, and by and large, reliable perception is not one of the major roadblocks for the LIDAR and map-using teams. Rather, the challenge lies in the immense amount of detail required to ensure that the vehicle can handle all unusual cases, especially cases that have never been seen before.

mapping and fleet

The question of the usefulness of maps is another one where Tesla/Karpathy and other teams differ. While Karpathy hoped to build a car that could fully understand the road and where to go without a detailed map, such a car is also a car that can remember what it learned and create one Map can use to support the next version of this car to drive this road. Ironically, Karpathy’s own statement about the tremendous value of a large fleet applies well here – having a large fleet makes it possible to create complete, detailed maps of the whole world and keep them fresh, and it’s foolish to throw away useful ones Information learned from this fleet.

These issues were discussed in more detail in my article and video on Tesla’s mapping choices:

The way to the future

Karpathy is right that at some point there will probably be a breakthrough that will enable computer vision to perform the driving task with a high level of safety. Most other teams don’t disagree with him. He may be right in predicting that they will eventually get rid of their LIDARs to cut costs. But they think they will after they’re in production, after they’ve taken the lead in the robotaxi business, while Tesla is still only doing driver assistance. You could be wrong – this breakthrough could come sooner, in which case Tesla will have great success. But they don’t think that’s the way to bet.

It’s also that over time and as all tools improve, the additional sensors may not cost as much or add complexity. LIDAR, radar and thermal imaging cameras offer superhuman perception. They can see things cameras can’t. Even if that benefit dwindles, it won’t go to zero – the debate is on as to whether their costs are justified. But when it comes to digital technology, those costs have historically been known to fall. The immense complexity of a modern mobile phone would amaze anyone not long ago, and its cost would shock them even more. People who bet that technology is expensive have rarely won the technology race. Tesla is actually a great example of a company that won by betting that technology would get better and cheaper.

Karpathy’s view of this future is difficult to discern. His position at Tesla was highly sought after and lucrative in his field. For someone who believes in Tesla’s path, it’s a particularly important place to change the world. However, he didn’t leave Tesla to start another project, at least as far as public announcements go. His departure suggests (but doesn’t guarantee) that he had some issues – possibly with the project or his notoriously difficult job for the boss. Possibly something else or something personal of course – that’s just speculation.

True, the bet Tesla made based on these principles is a big one — with a big payoff, or a big risk of falling behind. Fortunately for Tesla, however, it has so many resources that it can afford to change directions even if its internal research fails. Had it wanted to, it probably would have wanted to buy Argo.AI last week — but Argo’s assets don’t align with Tesla’s current plan. If the plan changes, another player may be available to take over.