2026Collaborator
llm-exe
A provider-agnostic framework promises that swapping an embedding model is a config change. That promise quietly breaks the moment images are involved, because multimodal embedders do not agree on what they hand back.
The result
Authored text-and-image embedding input for llm-exe, merged upstream by the maintainer on July 28, 2026 and published in v3.0.3 on npm the same day: 17 commits, two rounds of requested changes, CI green on Node 18, 20, 22, and 24.
llm-exe is a provider-agnostic TypeScript framework for building LLM applications, MIT licensed and published on npm. It gives you one interface for prompts, functions, and embeddings that works across providers, so switching models or supporting a new one does not mean rewriting your app. I am its second-highest contributor and a Collaborator on the repository, working across the embedding layer, provider correctness, the release pipeline, CI hardening, and the agent workflows the project runs on itself.
- Release
- Shipped in v3.0.3
- Commits
- 17
- CI
- Node 18 to 24
- Merged PRs
- 45+
What I found first
Before writing the input type I compared five embedding providers against their official documentation: Cohere Embed v4, Voyage multimodal-3, Amazon Titan Multimodal, Google Vertex multimodal, and OpenAI. There are effectively three request families and four different fusion behaviors.
| Provider | What you get back |
|---|---|
| Cohere Embed v4, Voyage multimodal-3 | One fused vector. |
| Amazon Titan Multimodal | The average of the text and image vectors. |
| Google Vertex multimodal | Two separate vectors, one per modality. |
| OpenAI text-embedding-3 | No image embedding model exists at all. |
That table is the whole finding. Changing a multimodal embedding provider can change the kind of result you get, not just the wire format. Someone who swaps Cohere for Titan does not get a slightly different vector, they get an average of two vectors, and nothing in the type system tells them. Swap in Vertex and the shape of the response changes entirely. So a design that only normalized the request would have made provider-swapping look safe while leaving a silent correctness trap underneath. Everything below follows from that.
Architecture
One embedding call
text, images, or both
Text only
texts, no inputs
Multimodal call input
inputs, texts dropped
Text plus the imageInputs option
inputs from the option, texts dropped
Both at once
the call input wins
Image sent to a text-only provider
typed error at the boundary
Unrecognized response envelope
typed invalid_response_shape
The public surface is one union: a plain string, an array of strings, or an array of content items, where an image part carries a full data URI. That is deliberately the same content convention OpenAI's chat vision API uses and Cohere Embed v4 accepts natively, so the shape is familiar rather than bespoke. From there the adapter decides where input goes instead of forwarding it: on the Bedrock Cohere provider, multimodal input routes to Cohere Embed v4's inputs field and plain text routes to texts, and because those two fields are mutually exclusive the four cases above have to land exactly right. Classification is what makes that safe. Three helpers decide what an input actually is, and the subtle case is OpenAI's pre-tokenized input, which arrives as an array of arrays and could easily be misread as multimodal content. Detection is narrow enough that pre-tokenized input is never misclassified, empty arrays and mixed batches are excluded, and a mixed batch fails loudly rather than returning vectors misaligned with their inputs, which is the failure you would not notice until your search results were quietly wrong. The new union also flows through the call factory into the request-wrapper generic, so a caller passes multimodal input without a cast: in a framework whose value proposition is typed calls, keeping inference intact was a requirement rather than a nicety.
Key decisions and trade-offs
Design for the difference instead of hiding it
The easy version of this feature normalizes the request bodies and presents one smooth surface over all five providers. That is the version I did not build. A uniform facade over four different fusion behaviors is worse than no abstraction, because nothing looks broken: vectors from two providers stop being comparable and a retrieval system quietly gets worse rather than failing. So the input type is provider-neutral while the routing stays deliberate, and a provider is never handed a request it cannot answer honestly.
Fail at the boundary, not at someone else's API
Two of the supported providers cannot take an image at all. OpenAI's embeddings endpoint has no image field, and Amazon's Titan embedding config sends a single inputText string: Titan's multimodal model is a separate model with a different request shape that this config does not target, which is exactly why the comparison above and this rejection are both true. Sending either one image content would post an object where a string is expected and come back as an opaque HTTP 400 from an API the caller did not write. So both reject in the input transform instead, with a typed LlmExeError carrying embedding.unsupported_input, the provider named, and a resolution pointer. The Cohere response resolver got the same treatment, throwing a typed embedding.invalid_response_shape when the envelope is unrecognized. If a provider cannot embed an image, the caller should learn that in their own stack trace.
Additive, or it does not ship
Every existing string and string array embedding call had to keep working untouched, and end-to-end type inference had to survive so callers pass multimodal input without casting. A widened input type that forces a cast at the call site is a broken abstraction wearing a new feature, and in a framework whose whole value is the shared interface, that cost lands on everyone downstream.
Deferring to the maintainer's release process
Review requested changes twice, and both rounds changed the work. The first blocked on a version bump I had applied myself. The project's convention is that the maintainer decides when to cut a release, and merging a self-applied bump would have landed an unreleased version number and pre-empted that, so I reverted it in both the manifest and the lockfile and left the version alone, while flagging that an unrelated dependency happened to share the same version string and was deliberately untouched. Worth saying plainly: the convenient move was to leave it in.
Cutting my own work to keep the pull request focused
I had also built a provider capability-descriptor layer, and review pushed back that it was a second concern inside a single-purpose change. I made the case for keeping it, since it had been requested as evidence the design generalizes past one provider, and then dropped it anyway and sequenced it as a separate follow-up. Arguing a position and still choosing the smaller change is the part I would want a reviewer to see.
A public API is a contract, so its numbers have to be real
Review caught that my descriptor values were tuned to the consumer I had in mind rather than the providers themselves. The OpenAI embedding descriptor claimed a maximum of one item per request when the endpoint accepts up to 2048, and the surrounding transform already forwarded batches, so the value contradicted both the API and the code next to it. I corrected it against provider documentation, verified live that a batch returns one vector per input, re-derived Titan's limit of one and Cohere's 96 from their docs, and removed references to a system that would mean nothing to anyone else reading it.
The reviewer that held me to this is one I help maintain
The project reviews its own pull requests with an automated reviewer, and I hold most of the commits on that workflow, along with the coder and docs-sync agents and the bot that answers on threads. So the review that blocked this change came from tooling I help build, which is a strange position to be in and a useful one: it caught a descriptor of mine that was quietly wrong, and I do not get to argue with it any differently than anyone else does. The maintainer still merged it. Building the thing that holds your own work to the standard is worth more than being trusted to hold yourself to it.
Documenting the limitation instead of papering over it
One static descriptor cannot be accurate for a provider key that serves both Cohere Embed v3 (text-only, fixed at 1024 dimensions) and v4. Rather than pick a value that is wrong half the time, I wrote the constraint down and identified splitting the Cohere key as the actual prerequisite, tracked separately. A known, documented gap is safe to build on. A confidently wrong one is not.
Results
- Merged upstream by the maintainer on July 28, 2026, and published in v3.0.3 on npm 33 minutes later.
- 17 commits, and CI green across the test matrix on Node 18, 20, 22, and 24.
- Two rounds of requested changes from the project's automated reviewer, each resolved and approved.
- One input type covers a string, an array of strings, or content items where an image part carries a full data URI.
- Multimodal input routes to AWS Bedrock Cohere Embed v4, plain text to the text field, mutually exclusive across all four call shapes.
- Text-only providers reject image content at the boundary with a typed, diagnosable error instead of an opaque provider 400.
- The embedding and error suites pass (298 tests at the merged state), covering all four routing cases through the real body-mapping path, the error codes and their context, the text-only rejection message, and that a multimodal response still resolves.
- Additive and non-breaking: existing string and string-array calls are untouched, and type inference is preserved end to end.
- I hold most of the commits on the project's agent workflows: the automated PR reviewer, the coder and docs-sync agents, and the bot that answers on threads.
- 45+ merged pull requests on the project overall, as second-highest contributor and a repository Collaborator, across provider-correctness fixes and release and CI hardening.
Repository
llm-exe is public and MIT licensed, so none of this has to be taken on trust: the feature pull request, the upstream repository, and a filtered view of my merged pull requests are all linked below.