Local AI: Vision Pipeline
Evolving from a concentrated vision analysis to full on pipeline
I’ve written quite a few times on how I’m using local vision for different projects. I find it extends token usage considerably, and at the same time I have a sense that the outputs off the coding harnesses are better. I believe the reason for this is the ruthless interrogation of the image. I’m not just getting a prompt, I’m doing a full on briefing on what the image contains.
An agent (or me, by using my web UI) gets back a very detailed briefing of the image. This is a complex image I’m doing an analysis on. What is it? In words?
I’m using several different vision systems not so much a factory orchestration, but in a concert. During the Summer into AI competition I bumped into OpenCV5 and also added on SAM2, a segmentation software that I’ll show examples of here in a moment. OpenCV5 is great, it’s like this great toolbox of vision capabilities. Why not wrap them in an API and then just tie it all into a pipeline that the vision system could work with? So, that’s what I did.
The important caveat here is that I’ve not yet put the full extent of the pipeline onto a new project. In my own interpretation of how things are going, I can see quite a few things that need adjusted. For example, it’s leaning a little too hard in the direction of citation and evidence flows to justify the pipeline and not quite enough on the prompting side.
The way I’m tackling this is with a control panel, which looks like the following image. This panel also gives some sense of what this suite of tools is capable of. It’s funny to me in a sad way: I can recall very expensive imaging solutions that did 0.001% of what this system is doing on my Surface Pro-and it’s largely all written by Codex.
The other vision API capabilities remain present. I bolted on a full vision pipeline because I just wanted to put the entire set of capabilities to work in one long set of exchanges. It’s incredible to me that one can prompt out a software factory that handles complex vision analysis and handoffs. It might be worthwhile to feed this article to your AI tool of choice and collect some ideas on how that might work.
Here’s another article on local vision. It might get you thinking about other possible use cases.
