Proxies
We just developed and deployed a new service which by itself caused us to also develop a new feature on the AI Horde!
The service itself is a small pilot for an OpenAI-compatible API to the AI Horde, working as a proxy service. The new service can be found on https://oai.aihorde.net and should hopefully provide an API you can plug any frontend and integration which is only compatible with only OpenAI.
You can find it at this URL: https://oai.stablehorde.net along with full API documentation here: https://oai.stablehorde.net/docs
(with many thanks to Luke100000 who provided the initial code which I forked to develop this)
The current OAI API Proxy supports only Large Language Models currently (i.e. no image generation) and is fairly limited in the amount of generation parameters one can send to it since the OpenAI API itself doesn't support a lot of the advanced features the Horde does in regards to customizations. Also due to the expectation that the models are always available behind this API, it doesn't take into account the realities of a crowdsourced model where workers can drop in and out at any time at varying capacities. Therefore we've had to make some creative so payloads sent to the API so as to ensure people don't get rejected without understanding why. For users practically, it means all request contexts and tokens will be automatically scaled to fit the highest available worker's capacity, when higher.
This service is running as a pilot at the moment and we might need to adjust access to it or otherwise its capacity based on how much it's utilized. We still hope that AI Horde integrators will opt to use our full REST API but hopefully this service will allow more end users to be able to hook onto the AI Horde.
However running such a service as a proxy, opened up some questions on how to ensure proper operations without enabling abuse of our service. Not just for this service, but for any service which wants to operate a proxying operation on the AI Horde. Since all requests are coming through the server running the API, it's not possible to separate a malicious proxy from a malicious user. If we don't accept forwarded IPs from the proxy, we severely limit their capacity to go through our rate limiter as a busy proxying service could easily appear to be spamming our endpoints. However if we trust them to send us their origin user's IPs, we simply run the risk of a malicious proxy operator sending us fake IP address to bypass all rate limiting restrictions.
In the past, existing services were mainly whitelisted on the code or the environment variables of the AI Horde, which made it very annoying to onboard new proxies, as it required a restart of the whole service.
To solve this we developed a new approach which allows accounts marked as "service accounts" to generate a proxy_passkey which they can then use to authenticate with the AI Horde as a proxying service. Afterwards all one such service needs to do is add the Proxied-For header with the original user's IP along with their Proxy-Authorization containing their proxy_passkey. When these are received, the AI Horde will consider the Proxied-For IP as the one sending the request, and apply rate limits on it.
This should allow a proxying service to serve a lot of users without worrying about hitting rate limits and of knowing that any sanctions those users get for malicious or unethical behaviour, will not affect the proxy service itself. And since the service account role is only ever assigned manually on request by our AI Horde moderation team, we can ensure that anyone trying to abuse this privilege can be blocked from operating such a proxying service.
We already have a bunch of such proxying services, such as our very own Discord bot, and of course the new OpenAI API proxy. Both of these are hosted by Haidra, but they can also be hosted by anyone else for their own communities as they're Free Open Source Software. All that remains is for those services to adjust to send the new headers and they can now function much more smoothly.
As always let us know what you think of our development efforts and how you like the new service.
