Control a 3D avatar with language instead of buttons

I built a 3D character you can control with language: https://programasweights.com/avatar

Traditionally, 3D avatars are controlled through predefined buttons or scripts. Here you just describe what you want in plain English - including sequences and combinations you'd never wire to buttons, like "wave while walking, then jump a couple times."

How it works: it's built on programasweights, which we made earlier that compiles neural programs from plain-English descriptions. This avatar's "director" is one such program - at runtime it turns your sentence into a tiny action program (loops, holds, and parallel tracks) that runs locally in the browser. The exact program behind this avatar: https://programasweights.com/hub/9c2309c0c9019b180adc (and you can easily build your own).

Using a compiled program locally is just a few lines (pip install programasweights):

import programasweights as paw director = paw.function("9c2309c0c9019b180adc") # the avatar's compiled program print(director("jump twice")) # -> repeat 2 { jump } 

(First call downloads the tiny program + base model, then runs offline.)

Debugging panel: add ?dbg=1 to the URL to open a debug panel and watch the exact action program it writes for each sentence.

I'm quite interested in applying this to games. Instead of NPCs following fixed, hand-authored recipes, they could improvise behavior from user chats and emotions - the model writes the action program on the fly. I think AI should give us better games.

Code + paper: The inference/runtime code is already released at https://github.com/programasweights, and more background about the approach is here: https://x.com/yuntiandeng/status/2044086557330579851. If you really want the full code right now, the uncleaned version we used for the submission is at https://anonymous.4open.science/r/programasweights, but we'll clean it up and release a better version.