I benchmarked six AI Gateways, including ours
"That's great what you're doing with Edgee, but how does it affect performance?"
This is one of the questions I get asked most often in meetings.
Until now, I knew we weren't bad. Rather good, actually. Our engineering team loves spending time optimizing all sorts of things here and there, our infrastructure is ultra robust, and I have that in my DNA. I've built dozens of scalable, high performance infrastructures throughout my career.
But when it comes to our gateway, a simple "yeah, we're super fast" wasn't good enough. So I spent quite a bit of time building a benchmark to measure ourselves against our competitors, so I could say precisely where we're strong and where we're not.
I benchmarked six AI gateways against each other, and against calling the provider directly. Same region, same second, same prompt.
Turns out the gap between them is much bigger than I expected.
The benchmark
Everything runs from gateway-bench. Feel free to make it yours, run it, and tell me if there are things to fix.
The repo is a simple NodeJS job that calls inference endpoints, measures Time to First Token (TTFT) and a few other things, and stores the results in a BigQuery table.
- The job is deployed to 43 Google Cloud regions, firing several times a day
- Every gateway called in the same execution, one second apart
- Two models:
gpt-5.4andclaude-sonnet-4-6 - OpenAI and Anthropic called directly as the reference line
- 16,344 measurements across 1,173 runs, 28 countries, 6 continents
- Window: 5 to 6 August 2026
We measured time to first token: the wait before text starts appearing. Not headers, not total time. The pause the user actually feels.
The gateways: OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, Kong AI Gateway, TrueFoundry, and Edgee.
Edgee is first on GPT-5.4
Median time to first token:
| Gateway | Median TTFT | Overhead |
|---|---|---|
| Edgee | 798 ms | +24 ms |
| Cloudflare | 811 ms | +37 ms |
| TrueFoundry | 869 ms | +95 ms |
| Vercel | 938 ms | +164 ms |
| OpenRouter | 1,020 ms | +246 ms |
| Kong | 1,718 ms | +944 ms |
| OpenAI, called directly | 774 ms | reference |
24 milliseconds. That's what routing through Edgee costs you compared to calling OpenAI directly, and in exchange you get failover, observability, compression, and cost control. I'll take that trade.
Now look at the bottom of that column. Between the fastest gateway and the slowest, there's 920 ms. Between the fastest gateway and no gateway at all, there's 24 ms.
In other words, picking the wrong gateway costs you forty times more latency than adding a gateway in the first place.
Cloudflare takes Claude
Same test, claude-sonnet-4-6:
| Gateway | Median TTFT | Overhead |
|---|---|---|
| Cloudflare | 1,140 ms | +26 ms |
| Edgee | 1,215 ms | +101 ms |
| TrueFoundry | 1,232 ms | +118 ms |
| Vercel | 1,307 ms | +193 ms |
| OpenRouter | 1,321 ms | +207 ms |
| Kong | 5,101 ms | +3,987 ms |
| Anthropic, called directly | 1,114 ms | reference |
Cloudflare wins this one. We're second, 75 ms behind.
I'm not thrilled about it. But that's exactly what these benchmarks are for: knowing what to fix. So we're going to dig into Edgee's Rust code and find a way to shave off at least 75 ms ;)
That said, we're ahead of everyone else on this model too, and that part makes us pretty happy 😁.
Two out of three
Medians across a two day window are easy to get wrong. Gateways aren't always exercised in the same runs, and a median computed over a different set of regions isn't comparable.
So we also compared inside each run. Same batch, same region, seconds apart, difference in milliseconds.
Across 11,600 head to head comparisons against all five competitors, on both models, Edgee delivered the first token first in 66.7% of them. Two out of three 🚀
Per opponent:
| Against | Edgee first in |
|---|---|
| Kong | 96.7% |
| OpenRouter | 67.9% |
| Vercel | 64.9% |
| TrueFoundry | 54.4% |
| Cloudflare | 49.4% |
Cloudflare is a coin flip. The other four are not.
The method really matters here. In Asia, pooled medians put Cloudflare 41 ms ahead of us. Compared run by run, the exact same data puts us 6 ms ahead. The two views disagree on the direction, and only one of them is comparing like with like.
The most installed gateway is fifth
OpenRouter is the gateway most teams reach for first. It comes fifth of six on both models: +246 ms on gpt-5.4, +207 ms on claude-sonnet-4-6.
That's roughly ten times the overhead of the leaders, on every single request, forever.
Popularity and latency have nothing to do with each other 😏
Kong doesn't stream Anthropic
One result stopped me. On claude-sonnet-4-6, Kong's numbers look like this:
- Response headers: 5,100 ms
- First token: 5,101 ms
- Complete answer: 5,102 ms
Two milliseconds separate the three. Nothing is being streamed, the response only opens once the answer is finished.
We inspected the raw stream to be sure. For an answer of the same length, Kong emits 17 stream frames on the Anthropic path against 230 on the OpenAI path.
That obviously explains the numbers. We did try to enable streaming and couldn't find a way. If anyone knows how, we'd genuinely love to hear from you ;)
A note on scope
Two days, one prompt shape, two models. Short messages and short answers. Nothing here tells you anything about long contexts or tool calls, where gateways may behave very differently.
And I want to be precise about Cloudflare, because they're the interesting one. Our median lead on gpt-5.4 is 13 ms, and head to head across both models we're at 49.4% against them. That's a tie, not a win. They're ahead on Claude, we're marginally ahead on GPT, and honestly you could run this again next month and get the opposite. Our clear wins are against Kong, OpenRouter, Vercel and TrueFoundry.
Bottom line
If you run LLM traffic through a gateway, that layer isn't free, and its price varies by a factor of forty depending on which one you picked.
- +24 ms: Edgee on
gpt-5.4, the lowest overhead we measured - 66.7%: head to heads where Edgee delivered the first token first, across 11,600 comparisons
- +246 ms: OpenRouter, the most widely installed gateway
- 0 failures across 2,350 Edgee requests over the window
Same region, same second, same prompt. The gateway is the variable.
Run it yourself: gateway-bench. If our numbers are wrong, I want to know.