AI Horde

serviceinfo, a new .well-known URI for API providers

db0

db0

— 2 min read
serviceinfo, a new .well-known URI for API providers

The text generation on the AI Horde supports multiple inference back-ends, unlike the image generation which is reliant on then reGen worker. As the AI Horde initially spawned utilizing the KoboldAI API, we kept using that API schema as the primary one for LLM inference. The existing inference back-ends we were using in the early days on-boarded the KoboldAI API so that they can be used by the AI Horde.

However in recent days, a lot of LLM inference back-ends such as new versions of Aphrodite, and TabbyAPI only provide APIs replicating the OpenAI API. As such people who run those back-ends and wanted to serve on the AI Horde, needed to install a completely different software with all the complexities that involves.

To fix this, I adjusted the AI Horde Worker which is the primary tool used to bridge between the Horde and the LLM inference backend so that it can talk to an OpenAI API as well. I also adjusted the AI Horde so that it now knows that there can be different back-ends behind the same bridge with different capabilities. This way someone knows and can also filter between something being generated between KoboldCPP, Aphrodite with koboldai API, or TabbyAPI etc.

However it was still a bit tricky for the horde bridge to know what API it was talking to, so that it can, in turn, inform the AI Horde. We needed a way for the bridge to inquire the API itself and get information about it, such as the software name and version, and the available API names and versions. So in collaboration with 2 other LLM developers AlpinDale (Aphrodite Engine), and @LostRuins (KoboldCPP) we came up with a new .well-known standard we named serviceinfo.

Already AI Horde, Aphrodite and KoboldCPP have onboarded it, and we've opened PRs and requests for other popular backends. You can take a look how it looks in action here: https://aihorde.net/.well-known/serviceinfo.

We hope that this can become a standard for inference back-ends and a way to make the life easier for integrators to support more of them.

Read more