Letting an agent drive the TV

The work in decluttering our TCL Google TV was done by a coding agent running on my laptop, with ADB open to the TV, over one evening and the next morning. ADB on a TV is close to having the keys to the house: it can switch off any package, change any setting and restart the TV whenever it likes. The TV was in use at the same time, and some of the restarts had to wait for an advert break. What kept the two compatible was a short rules file the agent reads at the start of every session. Some of the rules were there from the start. The rest were added after something went wrong, which is the more useful half. This is what’s in it.

Only changes that can be undone

The first rule is that packages are disabled, never uninstalled. A disabled package stays on the TV, and pm enable puts it back. Every package change goes through one script, tv-batch.sh, which writes each change to a log with a timestamp, so there’s never a change made by hand that the log doesn’t know about. Settings go in the same log, with the old value, the new one and the command to undo it. It has about a hundred lines now.

The second is the keep-list: packages the agent must never disable, each with its reason. The script refuses them, so the rule doesn’t depend on the agent remembering it. The packages I kept on a Google TV goes through that list.

Small batches, with me in between

Packages went in batches of about ten. After each batch I checked the things a broken package would break: the HDMI inputs, the voice button, casting from a phone, the remote and the apps we use. Only then did the next batch go on.

Anything that needed a hand or a router went into a separate file of steps for me: the one real press of the Disney+ button that Key Mapper needed, the CEC switch in the Fire Stick’s own settings, a fixed address for the TV on the router. The first restarts, which interrupt whatever is on, waited for a Fire Stick advert break and my yes.

Rules that came from mistakes

The rest of the file is scar tissue.

A restart with adb reboot lost a batch of packages and a permission set half an hour earlier, because Android saves both a while after the change, and that command skips the shutdown that would have saved them. The rule now spells out the safe sequence, and a script, tv-restart.sh, does it: save the app-ops, wait for the package settings, restart with a normal shutdown, then run the health check.

Reading the screen as text with uiautomator dump suspends other accessibility services while it runs, which is why Projectivy’s home-screen override wouldn’t stay on while the agent watched it being set up. The rule says not to use it during that setup, and screenshots replaced it.

The agent once reported that pressing Back on the TV unplugged the laptop’s extra screen, without having tried it. It didn’t: Sunshine keeps the session open in case the TV reconnects, so the screen stayed. Now every changelog entry says what was checked afterwards and what wasn’t, and “not checked” is an allowed answer.

Privacy

Every screenshot is deleted once it has been read, on the laptop and on the TV. Then the TV became a screen for the laptop, and a screenshot of the TV became a screenshot of whatever I had on that screen, messages and all. That rule is absolute: no TV screenshot while a Moonlight session is running. The agent checks Sunshine’s log for a connected client, or reads Moonlight’s menus as text, before taking any image. It held this morning, when the screenshots for these posts had to wait until I’d finished working on the TV screen.

There’s a gentler version for the remote. If I might be at the TV, the agent checks first, so we don’t both press buttons at once.

Asking less

Early on, the agent stopped to ask me about decisions it could have made itself. On the first day I told it to stop asking me for answers, get on with it, iterate and keep going. That went into the file too, with a condition attached: make the sensible call, then name exactly which packages or settings changed, in the changelog and in the report back. The things only I can do still go in my file of steps.

What it misses

Most of these rules are text. The agent follows them because it reads them, and nothing stops it ignoring one. The rules that matter most are the ones a script enforces: the keep-list refusal, the safe restart, the health check that fails when something has moved. If I did this again, I’d move more of the file into scripts, and sooner.

The file also only grows after a mistake, so it’s no help with the first one of each kind. The adb reboot that lost a batch was a restart I had agreed to, during an advert break, and it still cost a batch and a permission.

Using it

The scripts the rules point at are in the repository. The rules file itself is specific to this TV and this house, but its shape carries over to any device an agent can reach:

• Only make changes that can be undone, and make them through one script that logs them.
• Keep a list of things never to touch, and have the script refuse them.
• Change things in small batches, with a person checking in between.
• Write down what was checked after each change, and what wasn’t.
• Put anything that needs a hand or a password in a separate list for the person.