John's World: Jetson Edition
Virtual worlds are crazy.
I mentioned in a prior article that I’d be working on a way to put local AI capabilities into a 24x7 loop on the Jetson. The question I wrestled with was what. The Jetson’s coding capabilities are not quite useful, but the text generation is decent. Further, I hadn’t yet solved whether I could get tool use to work on the Jetson, with any model. Tool use would allow an agentic loop to do questionably useful things like call a vision model to generate an image or fetch a webpage. The trick? How do you solve for agentic loop of a model that demands vram, swap it out for yet another model that demands vram, and then swap it back? It could be a leetcode puzzle.
The bigger challenge: what’s the UI metaphor?
What would I do to test all this out, and to really use that loop as both test loop and also the place where I could get the generations in place?
That’s when it hit me: simulated IRC chat. I could port over ideas that I’ve written about regarding John’s World, and then bring those over to the Jetson in a new way. The irc chat would have rpg character like mechanics (handy that we’ve had a few small RPG projects during the Summer into AI competition) and I’d just use those mechanics as personas connected to the model. Then I’d just have to solve for tool-use.
I don’t think the mechanical implementation matters quite as much, you could just point a frontier model at this article and get a plan that works for your hardware. The bigger thought is just the importance of framing the simulated IRC chat as something where multiple personas existed, each with a lightweight agentic implementation including tool use, and then a timer plus compaction techniques to ensure the chat context didn’t overflow.
What surprised me wasn’t that this setup worked-if you read John’s World related posts, then you have a pretty clear line into my thinking and why this was likely to work. What surprised me was the variability of the images. I’ve tried having simpler loops of having an ai model iterate and just chew threw image generation tasks. What surprised me was just how much uniqueness that there was to each image, while maintaining a kind of curious retro-glaze.
And it isn’t clear to me what these are. They all certainly feel like they’re from the same place, and if you take all the images together it feels like there’s some other universe trying to message us with key details of what life is like on their planet. And a few images I couldn’t quite figure out-I’ll leave those as an exercise for the reader.
I find all these images quite nice. While the Simulated IRC chat can get pretty busy, I store all generations in a gallery for me to review more easily later on. I feel like I should use these images somewhere, but for now you’ll only find them here in this article.
Speaking of ideas, one of the 10/10 feature ideas you’ll want to consider for any local AI work is both a usage tracking system and also a gallery. There’s a lot of times a downstream system goes wrong. The gallery is a great place, too, to just see what the system has been up to. The left nav you can see some of the ideas I’m exploring-benchmarks, model curation, workflows, etc. The lived experience is that nearly all my time of using this is in “playground”. Simulated irc is definitely a fun one though. Too expensive to do with frontier API-but with local AI just humming along? Slow? Yes. Fun? Also, yes.
Toss the above image into a coding model and just have it replicate the structure for you. The modern world is a fun and weird place.
This is the article that I wrote which got me thinking about these longer runs. If you didn’t catch it, and this sort of topic interests you-I’d recommend checking it out. It’s a fast read.
