Upgrading the Firmware

From Unofficial Tesla Tech
Revision as of 20:32, 22 August 2020 by Carl (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Introduction

Congratulations, you now have root. Chances are, if you're on an older version of firmware you'll want to upgrade to the latest 8.1, or 9, whatever. I'm going to assume here that you're booted to a version lower than 2018.24.

You cannot install anything beyond 18.24 into a pre-18.24 CID because they'd changed the firmware signing key (so you must do 18.24 first), and then there are the maps changes around 18.28.5. Now the impending change in version 10 from OpenVPN to websocket (aka 'herpes', er I mean 'hermes') for comms security. So you must either have these versions, or find someone who has access to the firmware repository. I've been asking for volunteers who have access but so far no one has enough time. (or is willing)

So let's assume that your car boots fine and is functioning reasonably well (or not so), but well enough to log in.

You may want to dd a new firmware image to mmcblk0p1 and/or p2, but don't. Just don't. Build your eMMC image as recommended and be sure to use the version which matches that on the Spansion chip. Some day I'll do an article on how to find that out if you don't already know. To upgrade or downgrade this way if you do not dd the exact firmware version that the car was running, it means a mismatch with code in the Spansion chip, and thus code-signing failure, therefore boot failure and black screen.

We are going to do this firmware install the Tesla way. Except we are not going to fetch firmware from the Mothership, we are going to serve it using our own http server. There is a complete firmware repository, and if you know specifically the versions you need maybe someone with access will provide it to you. You can put the firmware file in the root of any webserver you have, as long as the CID can see it. Do -not- serve it through SSL; you must use http protocol.

Presenting the Firmware File

Error creating thumbnail: File missing
CAUTION - Science Content

~~ Trim Firmware Images for Partitions 1 and 2 ~~

It is imperative that you use only trimmed firmware images in this process.

Firmware for the OS (partitions 1 and 2) in the 21st century, is 'signed'. This means that an algorithm has been run on the whole file which develops a 'hash' (numeric result), that is then appended to the end of the file. When it's time to install the image this hash is checked to make sure the firmware hasn't been corrupted or molested, and this check must pass or the upgrade doesn't go through.

cid-updater -expects- this hash to be at the end of the image. If you dd an image from partitions 1 or 2, it won't be because the image is smaller than storage so zeroes are added to it. Therefore you must trim these last zeroes off.

This is where a utility called hexedit comes in handy.
# hexedit 18.24.image

Error creating thumbnail: File missing

We are now editing a binary file. Left column is address, center is the hex (machine) code, and right is the ASCII representation of the machine code. You can edit the machine or ASCII code (toggle between columns with {Tab}), although that's usually not advisable. For Help, {F1}.

We're interested in what's at the bottom of this file. Hold down {Shift}, and hit >. Ziiip, you're at the bottom of the file.

Error creating thumbnail: File missing

Ok, in this case there is gibberish at the end of the file. This is in fact the 64 byte hash, which means that this is a trimmed file.

If you see a bunch of zeroes at the end instead of gibberish, you need to delete them all, up to the hash and then stop. Don't fsck it up!

When ready to save, {F2}. To exit {Ctrl}z.

You might edit it once more to make sure you have it right.

If you got a firmware version from the firmware repository, it will already be trimmed.

Set up a Webserver

This is necessary as it's what cid-updater demands. I know that we now have lots of space on the eMMC in /home to lay down a firmware image (since we've upgraded to the Swissbit chip), but referencing that from cid-updater does not work -- it must be an http-served file. And you can't put it in Dropbox or Megaupload, etc, as by default they all use SSL. And even if you can find a non-SSL server out there on The Internets™, when you point the CID at it it will go through the parrot interface (external GSM) and the download usually fails. (Don't ask me how I know, but try it if you must)

And it can not be uhttpd or other embedded webserver. (Don't ask how I know) In my case I have an Arduino Yun installed in the car and on the same switch as the CID and IC, so I installed uhttpd on the Yun. Thirty-one failures trying every-which-a-way until I changed tack.

Maybe you're already running Apache. Like most others my websites automatically redirect from port 80 to SSL on 443, but we can't use SSL here because the CID isn't smart enough. So for whichever is your default Apache virtual host (usually the first one), comment out that redirect. Make it look something like this:

<VirtualHost *:80>
        ServerName example.com
#       Redirect permanent / https://example.com/
        DocumentRoot /srv/html/example.com/
</VirtualHost>

Then put your firmware image in that root directory. Be sure to set the firmware's permissions such that Apache can read it. Restart Apache.


Now if you're like me, your webserver is a virtual machine on your LAN, and you have a nanocomputer in the car which is on a switch with the diag port. Of course my webserver is in a DMZ and the nanocomputer is just a citizen on the LAN. Can't get the firmware by reaching out on The Internets™ as above-mentioned, so what's a po' fella to do?

Error creating thumbnail: File missing
CAUTION - Science Content

~~ About Reverse SSH Tunnels ~~

In Linux a daemon ('demon') is a service which is running all the time on a machine to serve some purpose, such as providing the correct time to other machines, a webserver, DNS service, etc. Most daemons listen on a 'port', which is an artificial location in the machine that other processes can look to for a service.

In my case the Apache server daemon is listening on 10.2.10.19, port 80, which is a location in my DMZ. Problem is the CID in my car can't reach that because it is in a different IP domain. (CIDR) The CID must get the firmware, which must (appear to) be on the nanocomputer.

So we use a wonderful mechanism known as a reverse SSH tunnel. This allows a server's port to be extended to a client, to look as if the server is -local-. (127.0.0.1) On the client any time you need the server you point it at 127.0.0.1, the client reaches into its own bellybutton, and puls out the server's response, all through an military-grade encrypted tunnel on the LAN (or WAN)

I use reverse SSH tunnels everywhere in my LAN, like on my laptop I can just look at localhost:8090 to see all of my (remote) security cameras. My laptop reaches into its bellybutton and pulls out the server for all my security cameras. Very nice.

So the nanocomputer reaches out to the Apache server using SSH, authenticates, and is allowed to bring port 80 from the webserver to itself, as if it were the webserver itself. For every purpose it thinks it is the webserver, which satisfies the CID. Why? Because we -must- use Apache (or nginx) to serve the firmware or the upgrade will fail.

The first thing to know is that on the Apache webserver, the SSH daemon must allow root logins. (temporarily) Certs don't work, at least with the Arduino Yun. So edit /etc//ssh/sshd_config to make sure that root can log in, and restart sshd.

Try it, from your nanocomputer:
# ssh 10.1.10.12 (or wherever your server is)
... if you can't log in this way using root's password for the server, fix that problem first.

Now on the nanocomputer set up the (temporary) reverse SSH tunnel:
# ssh -N -L 192.168.90.125:80:127.0.0.1:80 {webserver} sleep 365d
... it will appear to freeze, but the tunnel is set up. That remote webserver's port 80 is now presented on the nanocomputer for the CID as if local.

If you have further interest in reverse SSH tunnels see Bonus - Setting up Reverse SSH Tunnels Permanently.


Invoking the Firmware Update

Now log in as your suser to the CID:
# ssh carl@cid
carl@cid's password:
Linux cid 4.4.35-release-03mar2017-g352318b-dirty #see_/etc/commit SMP PREEMPT 1202798460 armv7l armv7l armv7l GNU/Linux

Welcome to Ubuntu! * Documentation: https://help.ubuntu.com/
Last login: Sun Feb 23 09:11:04 2020 from 192.168.90.125

carl@cid$ sudo -i
[sudo] password for carl:
root@cid#


So if your car is in an indeterminate state or you're unsure what condition its ECMs are in, it's always best to refresh the current version of firmware first. Put that version on your http server. Then we'll set cid-updater to download the firmware from the webserver and write it to the unused partition, either 1 or 2. It'll then determine the configuration of your car and load the appropriate ECM binaries for your car into /var/spool/cid-updater/staged, and install those binaries into the car's ECMs one-by-one.

cid-updater and ic-updater can be in an indeterminate state, maybe busy with previous attempts or stopped, so let's start fresh with a master reboot of the CID, gateway, IC, and all ECMs in the car. On the CID:
root@cid# emit-reboot-gateway

When it comes back to consciousness again log in to the CID as root. We are about to invoke the cid-updater daemon's shell to install the firmware, which does many things, one of which is passing control to the IC to do the kernel upgrade, CID upgrade, etc:

root@cid# nc localhost 25956
Welcome to Model S cid-updater ONLINE Built for Package Version: develop-2018.24-7330-baf6ebc570 (b243baac3e2a2ccb @ baf6ebc5700c61275e90f8341266c992ccf2793c) up 136.641545000s!

> install http://192.168.90.125:80/18.24.image
That file is not being served

Now see; this is its funny little trick. If you take it seriously and believe it you will be wrong.

Monitoring Progress

It actually did work, and we want to monitor it. Open another terminal window and ssh to the CID and root up. Then:

root@cid# tail -f /var/log/syslog |grep cid-updater
...
2020-03-22T11:20:38.119609-07:00 cid : cid-updater:10138: cid-updater Transferred 966367680/973008960 bytes
2020-03-22T11:20:44.900398-07:00 cid : cid-updater:30655: sid 156 has just asked to be kept awake
2020-03-22T11:20:44.901584-07:00 cid : cid-updater:29316: download filename=/dev/mmcblk0p2 pass=0.0 status=complete got_bytes=973008960 expected_bytes=973008960 offset_bytes=0
2020-03-22T11:20:44.903225-07:00 cid : cid-updater:29317: pending_command for retry: install http://192.168.90.125:80/18.28.5.image -o 966367680
2020-03-22T11:20:44.918062-07:00 cid : cid-updater:10032: uinotify /update_downloaded
2020-03-22T11:20:44.921541-07:00 cid : cid-updater:29334: download status=complete
2020-03-22T11:20:44.923060-07:00 cid : cid-updater:26675: flush_relays sid=156 filename=/dev/mmcblk0p2 abs_filename=/dev/mmcblk0p2
2020-03-22T11:20:44.923522-07:00 cid : cid-updater:26693: relay status=relaying monitor_sid=156 monitor_name=download relay_sid=162 relay_name=handle_http_request filename=/dev/mmcblk0p2 sent_bytes=972705025 expected_bytes=973008960 idle_seconds=0
2020-03-22T11:20:45.130245-07:00 cid : cid-updater: 7308: handle_http_request filename=/dev/mmcblk0p2 pass=UNK status=continuing sent_bytes=973008960 expected_bytes=973008960
2020-03-22T11:20:45.130651-07:00 cid : cid-updater: 7312: queue_file status=pre_eof sid=162 fd=18 read_this_much=1024 fbq=973008960 xbe=973008960 pending_output=queue_file /dev/mmcblk0p2 973007936
2020-03-22T11:20:45.130870-07:00 cid : cid-updater: 7316: queue_file status=post_eof sid=162 fd=18 read_this_much=1024 fbq=973008960 xbe=973008960 pending_output=queue_file /dev/mmcblk0p2 973007936
2020-03-22T11:20:45.131565-07:00 cid : cid-updater: 7337: queue_file status=post_onclose sid=162 fd=18 read_this_much=1024 fbq=973008960 xbe=973008960 pending_output=
2020-03-22T11:20:45.133166-07:00 cid : cid-updater: 9180: flush_output_then_close sid=162 circ_buffer_used=1024
2020-03-22T11:20:46.140954-07:00 cid : cid-updater:10149: Remote says: ic-updater:30655: sid 178 has just asked to be kept awake
2020-03-22T11:20:46.141625-07:00 cid : cid-updater:10138: cid-updater Transferred 973008960/973008960 bytes
2020-03-22T11:20:46.141823-07:00 cid : cid-updater:10149: Remote says: ic-updater:29317: pending_command for retry: install http://192.168.90.100:20564/dev/mmcblk0p2 -o 966367680
2020-03-22T11:20:46.142014-07:00 cid : cid-updater:10149: Remote says: ic-updater:10032: uinotify /update_downloaded
2020-03-22T11:20:46.146694-07:00 cid : cid-updater: 9684: Remote says: ic-updater: 7703: unexpected write error on sid 181 fd 15: Connection refused (errno 111)
2020-03-22T11:20:46.147041-07:00 cid : cid-updater: 9684: Remote says: ic-updater: 9870: sid 181 in onclose_free_on_OK
2020-03-22T11:20:46.147221-07:00 cid : cid-updater: 9684: Remote says: ic-updater: 9885: sid 181 has valid esc
2020-03-22T11:20:46.147385-07:00 cid : cid-updater: 9684: Remote says: ic-updater: 9887: sid 181 http client termination code -1
2020-03-22T11:20:46.147553-07:00 cid : cid-updater: 9684: Remote says: ic-updater: 9889: Retrying with session->pending_command get http://localhost:4070/update_downloaded -o 0
2020-03-22T11:20:46.147735-07:00 cid : cid-updater: 9684: Remote says: ic-updater: 4703: FOO RETRY: /update_downloaded: SOCK_STREAM parent: 178 fd:15 addr:127.0.0.1:36007 dtx:15770 rx:0 tx:0 px:0/0 inb:0 outb:128 READY:ox pending: get http://localhost:4070/update_downloaded -o 0
2020-03-22T11:20:46.148489-07:00 cid : cid-updater: 9684: Remote says: ic-updater:28576: do_HTTP_GET offset=0 download_retries=0 check_signature=0 reporting_name=doget argv_localpath=http://localhost:4070/update_downloaded
2020-03-22T11:20:46.148976-07:00 cid : cid-updater: 9684: Remote says: ic-updater:28266: HTTP Client sid 183 using session->xfer_filename = http://localhost:4070/update_downloaded 2020-03-22T11:20:46.155497-07:00 cid : cid-updater: 9684: Remote says: ic-updater: 7703: unexpected write error on sid 183 fd 17: Connection refused (errno 111)
2020-03-22T11:20:46.155739-07:00 cid : cid-updater: 9684: Remote says: ic-updater:11834: handle_http_client_completion sid=183 session_name=doget session_termination_code=-1 esc.valid=1 success_required=1 xfer_filename=http://localhost:4070/update_downloaded
2020-03-22T11:20:46.156015-07:00 cid : cid-updater: 9684: Remote says: ic-updater:11839: sid 183 http client termination code -1
2020-03-22T11:20:46.157001-07:00 cid : cid-updater: 9684: Remote says: ic-updater:11852: handle_http_client_completion status=giving_up session->esc.retries=0
2020-03-22T11:20:49.924703-07:00 cid : cid-updater:26713: relay status=flushed monitor_sid=156 monitor_name=download filename=/dev/mmcblk0p2
2020-03-22T11:20:49.925540-07:00 cid : cid-updater:29338: complete_download status=flushed retval=0
2020-03-22T11:20:50.926028-07:00 cid : cid-updater: 4703: TIMER: do after: timerfd OD parent: 0 fd:16 dtr:-1 READY:i pending: stage install http://192.168.90.125:80/18.28.5.image
2020-03-22T11:20:50.926582-07:00 cid : cid-updater:30655: sid 214 has just asked to be kept awake
2020-03-22T11:20:50.926846-07:00 cid : cid-updater:29940: verify_offline_and_stage sid=214 retry command=install http://192.168.90.125:80/18.28.5.image
2020-03-22T11:20:50.938294-07:00 cid : cid-updater:29977: signature status=starting
2020-03-22T11:20:51.045563-07:00 cid : cid-updater:11049: send_gwcmd line=11261 sid=214 status=success latency=0.10 command=0xa retval=0a01
2020-03-22T11:20:51.072707-07:00 cid : cid-updater:11429: verifysig status=in_progress fused=1

It gets down to this point and stops -- don't worry, you're in good shape, it's checking that the signature is correct for the firmware. Be patient, it will continue forthwith.

When it starts giving you this jazz:
2020-03-22T16:18:14.600284-07:00 cid : cid-updater: 4873: TIMER: MCU_test_touchscreen_keepawake: P timerfd RA parent: 0 fd:08 dtr:14 LC:system AUTH READY:i pending: system sdv POWER_touchState 1
2020-03-22T16:18:14.683165-07:00 cid : cid-updater: 4873: TIMER: test_gateway_keepawake: P timerfd RA parent: 0 fd:09 dtr:14 LC:pet AUTH READY:i pending: pet 10 2020-03-22T16:18:14.790459-07:00 cid : cid-updater:11364: send_gwcmd line=11576 sid=7 status=success latency=0.10 command=0xa retval=0a00
2020-03-22T16:18:29.600876-07:00 cid : cid-updater: 4873: TIMER: MCU_test_touchscreen_keepawake: P timerfd RA parent: 0 fd:08 dtr:14 LC:system AUTH READY:i pending: system sdv POWER_touchState 1
2020-03-22T16:18:29.682660-07:00 cid : cid-updater: 4873: TIMER: test_gateway_keepawake: P timerfd RA parent: 0 fd:09 dtr:14 LC:pet AUTH READY:i pending: pet 10 2020-03-22T16:18:29.789235-07:00 cid : cid-updater:11364: send_gwcmd line=11576 sid=7 status=success latency=0.10 command=0xa retval=0a00

... what it really means is get yer red arse out to the car and authorize that upgrade.

Carl A. Cook