Tesla Autopilot and Full Self Driving FSD Features and Potentials

Tesla Autopilot is an advanced driver assistance system (ADAS) developed from ground up by Tesla that enhances safety and convenience behind the wheel featuring FSD Full Self-Driving Capability. It is a suit that amounts to SAE International Level 2 vehicle automation. When used properly, Autopilot reduces the overall workload for a driver. Each new Tesla vehicle is equipped with eight external cameras and powerful vision processing to provide an additional layer of safety with all vehicles built for the North American market now using the camera-based Tesla Vision to deliver Autopilot features, rather than radar.

Tesla Autopilot comes in three multiple packages; Autopilot, Enhanced Autopilot and Full Self-Driving Capability all available for purchase depending on when the car model was built and can be installed on vehicles that were delivered without the autopilot. Its features are lane centering, traffic-aware cruise control, automatic lane changes, semi-autonomous navigation on limited access freeways, self-parking, and the ability to summon the car from a garage or parking spot.

With all of these features, the driver is responsible and the car requires constant supervision. Autopilot, Enhanced Autopilot and Full Self-Driving Capability are intended for use with a fully attentive driver, who has their hands on the wheel and is prepared to take over at any moment. While these features are designed to become more capable over time, the currently enabled features do not make the vehicle autonomous.

Features of Tesla Autopilot, Enhanced Autopilot and Full Self-Driving Capability

three multiple packages; Autopilot, Enhanced Autopilot and Full Self-Driving Capability all available for purchase depending on when the car model was built, their features are as follows

Features of Autopilot

Autopilot includes the following functionality and features:

1. Traffic-Aware Cruise Control: Matches the speed of the car to that of the surrounding traffic
2. Autosteer: Assists in steering within a clearly marked lane, and uses traffic-aware cruise control.

Features of Enhanced Autopilot

In addition to the functionality and features of Autopilot, Enhanced Autopilot also includes:

1. Navigate on Autopilot: Actively guides the car from a highway’s on-ramp to off-ramp, including suggesting lane changes, navigating interchanges, automatically engaging the turn signal and taking the correct exit.
2. Auto Lane Change: Assists in moving to an adjacent lane on the highway when Autosteer is engaged.
3. Autopark: Helps automatically parallel or perpendicularly park the car with a single touch.
4. Summon: Moves the car in and out of a tight space using the mobile app or key.
5. Smart Summon: the function that allows the car navigate more complex environments and parking spaces, maneuvering around objects as necessary to come find the owner in a parking lot.

Features of Full Self-Driving Capability

In addition to the functionality and features of Autopilot and Enhanced Autopilot, Full Self-Driving Capability also includes:

1. Traffic and Stop Sign Control (Beta): Identifies stop signs and traffic lights and automatically slows the car to a stop on approach, with the driver’s active supervision.
2. Upcoming: Autosteer on city streets.

Design and Developmental Approach of the Tesla Autopilot

Elon Musk first discussed the Tesla Autopilot system publicly in 2013, noting that “Autopilot is a good thing to have in planes, and we should have it in cars.” Over the past decade, Tesla Autopilot went through a series of hardware and software enhancements, gradually approaching the goal of full autonomy, which as of May 2022, remained a work in progress.

In October 2014, Tesla offered customers the ability to pre-purchase Autopilot that was not designed for self-driving. Initial versions were built in partnership with Mobileye, but Mobileye ended the partnership in July 2016 because Tesla “was pushing the envelope in terms of safety”

Tesla cars manufactured after September 2014 had the initial hardware (hardware version 1 or HW1) that supported Autopilot. The first Autopilot software release came in October 2015 as part of Tesla software version 7.0. Version 7.1 removed some features to discourage risky driving.

Version 8.0 processed radar signals to create a point cloud similar to lidar to help navigate in low visibility. In November 2016, Autopilot 8.0 was updated to encourage drivers to grip the steering wheel. By November 2016, Tesla Autopilot had operated for 300 million miles (500 million km).

In October 2016, Autopilot sensors and computing hardware transitioned to hardware version 2 (HW2) for new cars. Tesla used the term Enhanced Autopilot (EA) to refer to novel HW2 capabilities. In February 2017 Autopilot gained the ability to navigate freeways, to change lanes without driver input, to transition from one freeway to another, and to exit the freeway. It included traffic-aware cruise control, autosteer on divided highways, and autosteer on ‘local roads’ up to a speed of 45 mph. Software version 8.1 for HW2 arrived in March 2017, providing HW2 cars software parity with HW1 cars. The following August, Tesla announced hardware version 2.5 (HW2.5).

In March 2019, Tesla transitioned to hardware version 3 (HW3) for new cars. To comply with the new United Nations Economic Commission for Europe regulation related to automatically commanded steering function, Tesla provided an updated Autopilot in May, limited to Europe. In September, Tesla released software version 10 to Early Access Program (EAP) testers, citing improvements in driving visualization and automatic lane changes.

In September 2020, Tesla reintroduced the term Enhanced Autopilot to designate the subset of features applying to highway travel, parking, and summoning, whereas the Full-Self Driving option included navigation on city roads. Tesla released a “beta” version of its Full Self-Driving software in the United States in October 2020 to EAP testers

Tesla’s approach to try to achieve SAE Level 5 is to train a neural network using the behavior of hundreds of thousands of Tesla drivers using chiefly visible light cameras and information from components used for other purposes in the car (the coarse-grained two-dimensional maps used for navigation; the ultrasonic sensors used for parking, etc.)

Tesla has made a deliberate decision to not use LiDAR, which Elon Musk has called “stupid, expensive and unnecessary”. This makes Tesla’s approach markedly different from that of other companies like Waymo and Cruise which train their neural networks using the behavior of highly trained drivers and are additionally relying on highly detailed (centimeter-scale) three-dimensional maps and lidar in their autonomous vehicles.

According to Elon Musk, full autonomy is “really a software limitation: The hardware exists to create full autonomy, so it’s really about developing advanced, narrow AI for the car to operate on.” The Tesla Autopilot development focus is on “increasingly sophisticated neural nets that can operate in reasonably sized computers in the car”. According to Musk, “the car will learn over time”, including from other cars.

Tesla’s software has been trained based on 3 billion miles driven by Tesla vehicles on public roads, as of April 2020. Alongside tens of millions of miles on public roads, competitors have trained their software on tens of billions of miles in computer simulations, as of January 2020. In terms of computing hardware, Tesla designed a self-driving computer chip that has been installed in its cars since March 2019 and also developed a neural network training supercomputer (“Tesla Dojo”); other vehicle automation companies such as Waymo regularly use custom chipsets and neural networks as well.

Tesla Dojo

Tesla Dojo (Project Dojo) is an artificial intelligence (AI) neural network training supercomputer announced by Musk on Tesla’s AI Day on August 19, 2021. It had previously been mentioned by Musk in April 2019 and August 2020. In September 2021, a Tesla Dojo whitepaper was released. According to Musk, Project Dojo will be operational in 2022.

The Dojo supercomputer will use Tesla D1 chips, designed and produced by Tesla. According to Tesla’s senior director of Autopilot hardware, Ganesh Venkataramanan, the chip uses a “7-nanometer manufacturing process, with 362 teraflops of processing power”, and “Tesla places 25 of these chips onto a single ‘training tile’, and 120 of these tiles come together… amounting to over an exaflop [a million teraflops] of power”. Tesla claims that Dojo will be the fastest AI-training computer among competing offerings from Intel and Nvidia. As of August 2021, Nvidia says the current Tesla AI-training center uses 720 nodes of eight Nvidia A100 Tensor Core GPUs (5,760 GPUs in total) for up to 1.8 exaflops of performance.

Gartner research vice president Chirag Dekate said, “The Tesla Dojo is an AI-specific supercomputer designed to accelerate machine learning and deep learning activities. Its lower precision focus limits applicability to a broader high-performance computer (HPC) context.” He also argued that Dojo’s reported capabilities don’t grant it true HPC status, largely because it hasn’t been tested using the same standards as Fugaku and other supercomputers. Dylan Patel from Semi Analysis suggests that while the input/output is impressive, the amount of memory is inadequate, and the two most difficult issues (the software compiler and tile-to-tile interconnects) remain to be solved.

Tesla Autopilot Hardware

Hardware 1

Vehicles manufactured after late September 2014 are equipped with a camera mounted at the top of the windshield, forward looking radar in the lower grille and ultrasonic acoustic location sensors in the front and rear bumpers that provide a 360-degree view around the car. The computer is the Mobileye EyeQ3. This equipment allows the Tesla Model S to detect road signs, lane markings, obstacles, and other vehicles.

Auto lane change can be initiated by the driver turning on the lane changing signal when safe (due to ultrasonic 16-foot limited range capability), and then the system completes the lane change. In 2016 the HW1 did not detect pedestrians or cyclists, and while Autopilot detects motorcycles, there have been two instances of HW1 cars rear-ending motorcycles.

Upgrading from Hardware 1 to Hardware 2 is not offered as it would require substantial work and cost.

Hardware 2

HW2, included in all vehicles manufactured after October 2016, includes an Nvidia Drive PX 2 GPU for CUDA based GPGPU computation. Tesla claimed that HW2 provided the necessary equipment to allow FSD Full Self-Driving capability at SAE Level 5. The hardware includes eight surround cameras and 12 ultrasonic sensors, in addition to forward-facing radar with enhanced processing capabilities. The Autopilot computer is replaceable to allow for future upgrades. The radar is able to observe beneath and ahead of the vehicle in front of the Tesla; the radar can see vehicles through heavy rain, fog or dust. Tesla claimed that the hardware was capable of processing 200 frames per second.

When “Enhanced Autopilot” was enabled in February 2017 by the v8.0 (17.5.36) software update, testing showed the system was limited to using one of the eight onboard cameras—the main forward-facing camera. The v8.1 software update released a month later enabled a second camera, the narrow-angle forward-facing camera.

Hardware 2.5

In August 2017, Tesla announced that HW2.5 included a secondary processor node to provide more computing power and additional wiring redundancy to slightly improve reliability; it also enabled dashcam and sentry mode capabilities. During this time, the supplier for the system’s radar components was changed from Bosch to Continental.

Hardware 3

According to Tesla’s director of Artificial Intelligence Andrej Karpathy, Tesla had as of Q3 2018 trained large neural networks that work but which could not be deployed to Tesla vehicles built up to that time due to their insufficient computational resources. HW3 provides the necessary resources to run these neural networks.

HW3 includes a custom Tesla-designed system on a chip fabricated using 14 nm process by Samsung. Jim Keller and Pete Bannon among other architects have led the project since February 2016 and took over the course of 18 months. Tesla claimed that the new system processes 2,300 frames per second (fps), which is a 21× improvement over the 110 fps image processing capability of HW2.5. The firm described it as a “neural network accelerator”. Each chip is capable of 36 trillion operations per second, and there are two chips for redundancy. The company claimed that HW3 was necessary for Full Self-Driving FSD, but not for “enhanced Autopilot” functions.

The first availability of HW3 was April 2019. Customers with HW2 or HW2.5 who purchased the FSD package are eligible for an upgrade to HW3 without cost.

Tesla claims HW3 has 2.5× improved performance over HW2.5 with 1.25× higher power and 0.2× lower cost. HW3 features twelve ARM Cortex-A72 CPUs operating at 2.6 GHz, two Neural Network Accelerators operating at 2 GHz and a Mali GPU operating at 1 GHz.

Tesla Vision

In late May 2021, Elon Musk posted to Twitter that “Pure Vision Autopilot” was starting to be implemented. The system, which Tesla brands “Tesla Vision”, eliminates the forward-facing radar from the Autopilot hardware package on Model 3 and Model Y vehicles built for the North American market and delivered in and after May 2021. For vehicles without the forward radar, temporary limitations were applied to certain features such as Autosteer, and other features (Smart Summon and Emergency Lane Departure Avoidance) were disabled, but Tesla promised to restore the features “in the weeks ahead … via a series of over-the-air software updates”.

In response, the U.S. National Highway Traffic Safety Administration (NHTSA) rescinded the agency’s check marks for forward collision warning, automatic emergency braking, lane departure warning, and dynamic brake support, applicable to Model 3 and Model Y vehicles built on or after April 27, 2021. Consumer Reports delisted the Model 3 from its Top Picks, and the Insurance Institute for Highway Safety (IIHS) announced plans to delist the Model 3 as a Top Safety Pick+, but after further testing, both organizations restored those designations.

In December 2021, the New York Times reported that Musk was the decision maker behind the camera-only approach and had “repeatedly told members of the Autopilot team that humans could drive with only two eyes and that this meant cars should be able to drive with cameras alone.” Several autonomous vehicle experts were quoted denouncing the analogy.

Autopilot, Enhanced Autopilot and Full Self-Driving Capability User Guide.

Before using Autopilot, please read your Owner’s Manual for instructions and more safety information. While using Autopilot, it is your responsibility to stay alert, keep your hands on the steering wheel at all times and maintain control of your car. Many of our Autopilot features, like Autosteer, Navigate on Autopilot and Summon, are disabled by default. To enable them, you must go to the Autopilot Controls menu within the Settings tab and turn them on.

Before enabling Autopilot, the driver first needs to agree to “keep your hands on the steering wheel at all times” and to always “maintain control and responsibility for your vehicle.” Subsequently, every time the driver engages Autopilot, they are shown a visual reminder to “keep your hands on the wheel.”

Traffic-Aware Cruise Control

To engage Traffic-Aware Cruise Control in Model S and Model X, pull down once on the cruise control stalk on the left of the steering column. In Model 3 and Model Y, pull down once on the gear selector stalk on the right of the steering column.

Autosteer

To engage Autosteer in Model S and Model X, pull towards you twice on the cruise control stalk on the left of the steering column. In Model 3 and Model Y, pull down twice on the gear selector stalk on the right of the steering column. A grey steering wheel icon will appear on your display, next to the speedometer, when the system is available to engage. A blue steering wheel icon will appear on your display, next to the speedometer, when Autosteer is engaged.

When Autosteer is in use, it measures the amount of torque that you apply to the steering wheel and, if insufficient torque is applied, an escalating series of audible and visual alerts again reminds you to place your hands on the wheel. This helps ensure you are attentive and trains good driving habits. If you repeatedly ignore these warnings, then you will be locked out from using Autopilot for the duration of that trip.

Navigate on Autopilot

Navigate on Autopilot is designed to get you to your destination more efficiently by actively guiding your car from on-ramp to off-ramp, including suggesting and making lane changes, navigating highway interchanges, and taking exits. It’s designed to make finding and following the most efficient path to your destination even easier on the highway when Autopilot is in use.

To engage this feature, you must first enable Autosteer by going to Controls > Autopilot > Autosteer – then enable the Navigate on Autopilot feature. Camera calibration will be required and the latest version of Navigation maps must be downloaded via Wi-Fi.

For each route where Navigate on Autopilot is available, you will have the option to activate it by pressing the NAVIGATE ON AUTOPILOT button located in your Navigation Turn List. You can also activate Navigate on Autopilot automatically any time a destination is entered and Autopilot is engaged by adjusting your Navigate on Autopilot settings from Controls > Autopilot > Customize Navigate on Autopilot.

On Model 3 and Model Y, Navigate on Autopilot can be engaged on most highways by moving the gear lever twice downwards, in quick succession.

On Model S and Model X built between 2012-2021, Navigate on Autopilot can be engaged on most highways by pulling the cruise stalk toward you, twice in quick succession.

Auto Lane Change

To initiate an automated lane change, you must first enable Auto Lane Changes through the Autopilot Controls menu within the Settings tab. Then when the car is in Autosteer, a driver must engage the turn signal in the direction that they would like to move. In some markets depending on local regulations, lane change confirmation can be turned off by accessing Controls > Autopilot > Customize Navigate on Autopilot and toggle ‘Lane Change Confirmation’ off.

Autopark

If your car sees a parking spot, a grey ‘P’ icon will appear on the left or right hand side of your instrument panel, depending on the location of the space. Autopark detects parallel parking locations when driving below 15mph and perpendicular parking locations when driving below 10mph.

To use Autopark at that time, press on the brake and shift the gear selector into ‘Reverse.’ Keep your foot on the brake. ‘Start Autopark’ will appear in blue text on your touchscreen – press this to start the feature and release the brake and steering wheel. Autopark will then begin to maneuver the vehicle into the parking space by controlling your cars speed, gear changes and steering angle, but you should remember to be alert and monitor the backup camera view to check for obstacles.

Once Autopark is complete, the car will let you know it is complete and shift into ‘Park.’ You can override Autopark at any time by taking control of the steering wheel.

Summon

To use Summon, open the Tesla app. Press Summon and then press the forward or reverse buttons. Model S and Model X owners can use Summon with their key fob by holding the center of the key fob for three seconds until the car’s hazard lights come on, and then pressing either the frunk or trunk button on key fob to Summon forwards and backwards respectively. Summon also integrates with HomeLink and will open your garage door to pull your car out of the garage.

Smart Summon

Smart Summon is designed to allow your car to drive to you or a location of your choosing, maneuvering around and stopping for objects as necessary. Like Summon, Smart Summon is only intended for use in private parking lots and driveways. You are still responsible for your car and must monitor it and its surroundings at all times and be within your line of sight because it may not detect all obstacles. Be especially careful around quick moving people, bicycles and cars.

To use Smart Summon, open your Tesla app, tap Summon and then select the Smart Summon icon. To activate the feature, press and hold the COME TO ME button. Alternatively, tap the target icon, set the target destination of your choice by adjusting the map, and then press and hold the GO TO TARGET button. You can stop your car from driving at any time by releasing the button.

Smart Summon works with your Tesla app and your phone’s GPS to operate. You must be within approximately 200 feet of your car to use. Smart Summon requires the latest version of the Tesla mobile app (3.10.0 or later). Please refer to the Owner’s Manual for additional details about this feature.

Traffic Light and Stop Sign Control (Beta)

Traffic Light and Stop Sign Control (Beta) identifies stop signs and traffic lights and automatically slows your car to a stop upon approach while using Autopilot, with your active supervision. To enable, shift your car into PARK and tap Controls > Autopilot > Traffic Light and Stop Sign Control (Beta), then engage Traffic-Aware Cruise Control or Autosteer.

When Traffic Light and Stop Sign Control (Beta) is enabled, the driving visualization displays upcoming traffic lights, stop signs or road markings at intersections where your car may need to stop. As you approach an intersection, even if the traffic light is green, your car will display a red line to indicate where the vehicle will stop and the vehicle will begin to slow. To continue through the stop line, pull the Autopilot stalk or briefly tap the accelerator pedal to confirm that it is safe to proceed. At this time, Traffic Light and Stop Sign Control does not turn the vehicle, whether you are in a turn lane or have your blinker on.

Notifications on the touchscreen will provide a stop reason (stop sign or traffic light) and will also provide an estimated distance to when your vehicle will stop. If the Traffic Light notification does not indicate a light color, your vehicle has not confirmed the traffic control state.

As with all Autopilot features, you must be in control of your vehicle, pay attention to its surroundings and be ready to take immediate action including braking. This feature is in Beta and may not stop for all traffic controls. While Traffic and Stop Sign Control is enabled on surface streets with Autosteer active, your speed will be confined to the noted limit. Please review the Owner’s Manual for additional information, instructions for use and warnings.

Tesla Autopilot and Full Self-Driving Safety Features

Active safety features come standard on all Tesla vehicles made after September 2014 for elevated protection at all times. These features are made possible by our Autopilot hardware and software system and include:

• Automatic Emergency Braking: Detects cars or obstacles that the car may impact and applies the brakes accordingly
• Forward Collision Warning: Warns of impending collisions with slower moving or stationary cars
• Side Collision Warning: Warns of potential collisions with obstacles alongside the car
• Obstacle Aware Acceleration: Automatically reduces acceleration when an obstacle is detected in front of your car while driving at low speeds
• Blind Spot Monitoring: Warns when a car or obstacle is detected when changing lanes
• Lane Departure Avoidance: Applies corrective steering to keep your car in the intended lane
• Emergency Lane Departure Avoidance: Steers your car back into the driving lane when it detects that your car is departing its lane and there could be a collision

Active safety features are designed to assist drivers, but cannot respond in every situation. It is your responsibility to stay alert, drive safely and be in control of your car at all times.

Reviews and Critics for Tesla Autopilot and Full Self-Driving

In October 2020, Consumer Reports called Tesla Autopilot “a distant second” in driver assistance systems (behind Cadillac’s Super Cruise), although it was ranked first in the “Capabilities and Performance” and “Ease of Use” category. Collisions and deaths involving Tesla cars with Autopilot engaged have drawn the attention of the press and government agencies.

As an upgrade to the Autopilot capabilities, the company’s stated intent is to offer full autonomous driving (SAE Level 5) at a future time, acknowledging that regulatory and technical hurdles must be overcome to achieve this goal. From 2016 through 2022, Elon Musk, Tesla’s CEO, has repeatedly made inaccurate predictions as to when Tesla would be able to achieve SAE Level 5 autonomy. As of April 2019, some experts criticized Tesla vehicles’ lack of lidar, which is used by other companies working on autonomous driving.

In October 2020, Tesla initiated, and commissioned customers for, a Full Self-Driving beta program in the United States; as of April 2022, Tesla has over 100,000 people in this program. Some industry observers criticized Tesla’s decision to use untrained consumers to validate the beta software as dangerous and irresponsible. In May 2021, Tesla was ranked last for both strategy and execution in the autonomous driving sector by Guidehouse Insights.

In September 2021, legal scholars William Widen and Philip Koopman argued that Tesla has misrepresented FSD as SAE Level 2 to “avoid regulatory oversight and permitting processes required of more highly automated vehicles”. Instead, they argued FSD should be considered a SAE Level 4 technology and urged state Departments of Transportation in the U.S. to classify it as such since publicly available videos show that “beta test drivers operate their vehicles as if to validate SAE Level 4 (high driving automation) features, often revealing dramatically risky situations created by use of the vehicles in this manner.”