Fastly has launched Fastly AI Accelerator, the company’s first AI solution designed to improve performance and reduce costs across the use of similar prompts for LLM apps.
AI Accelerator is designed to reduce API calls and costs with intelligent, semantic caching. Built on Fastly’s Edge Cloud Platform and leveraging industry leading caching technology, it uses a specialised API gateway to improve performance for apps using popular LLMs.
When using Fastly AI Accelerator, developers only need to update their app to use a new API endpoint, which typically only requires changing a single line of code. Fastly AI Accelerator will then transparently implement semantic caching for OpenAI compatible APIs. This approach goes beyond traditional caching as Fastly AI Accelerator can understand the context of the requests and queries and will send a similar response if two or more requests are alike.
Anil Dash, Vice President of Developer Experience, Fastly, said: “Fastly AI Accelerator gives developers exactly what they want, by making the experience of their favourite LLMs a lot faster and more efficient, so they can focus on what makes their app or site unique – and what keeps their users happy.”
Stephen O’Grady, Principal Analyst, RedMonk, said: “Developers and enterprises alike are turning in large numbers to medium and smaller models. Whether it’s to lower costs, to shorten training cycles or to run on more limited hardware profiles, they’re an increasingly important option.”
