While adding web search to a local model’s chat panel, I wrote a very standard egress guard: resolve the target hostname, take the IP, refuse anything that is not a public address. That is the textbook shape of SSRF protection — it exists to stop the model from being talked into reaching an internal service.
Once it was in place, search started returning empty results, reliably. No error, no timeout, just nothing.
The diagnosis: the guard was rejecting every target, Wikipedia and the search API included. And its judgement was, within its own logic, entirely correct.
Addresses that were synthesised
This machine’s egress runs through a transparent proxy. To take over DNS, that class of proxy hands back a synthetic address for each domain, accepts the connection there itself, and forwards by domain name. The actual resolution looks like this:
$ python3 -c "import socket;print(socket.gethostbyname('en.wikipedia.org'))"
198.18.8.116
$ python3 -c "import socket;print(socket.gethostbyname('api.tavily.com'))"
198.18.10.9198.18.0.0/15 is reserved by RFC 2544 for network device benchmarking. Python’s ipaddress module reports is_global == False for it — which is right, that genuinely is not a public address. The IPv6 side behaves the same way, with the proxy handing out addresses from a private fdfe: range.
So every domain that routes through the proxy read as “not public” and got refused. The only hosts that passed were the ones resolved directly, returning real addresses. The reason this surfaced as a silent empty result rather than an error is that I was treating “every candidate source was unusable” as “nothing was found.”
The immediate fix is to add those two synthetic ranges to an allowlist. But once that was done, I realised the actual problem was not a missing network range.
The address you check is not the address you connect to
The entire logic of an SSRF guard rests on one premise: the IP I just resolved is the IP that will actually be connected to. Only if that holds does “check whether this IP is internal” mean anything.
A transparent proxy invalidates that premise outright. What gets resolved is a placeholder; the real connection is established elsewhere by the proxy, toward a destination this code neither knows nor controls. At that point the check is not weak — it is meaningless. There is no causal link between what it inspects and what eventually happens on the wire.
Adding the synthetic ranges to the allowlist restored the feature. It did not restore the protection. The honest statement is that in this network environment, control over egress destinations already belongs to the proxy, and the only thing this layer still enforces is a protocol and port restriction. Writing that down beats keeping code that merely looks secure.
The same gap, a second time
Having named the pattern — the thing you check is not the thing you use — I went back through the same file treating it as a search term, and found another instance.
URL parsing used the standard library’s urlsplit; the actual request went out through httpx. For internationalised domain names containing non-ASCII characters, those two use different versions of the IDNA standard: the standard library implements IDNA 2003, httpx implements IDNA 2008. The same Unicode domain can normalise to different punycode on each side.
The consequence is identical in shape to the fake-IP bug: I validated domain A and the program fetched domain B. The only difference is that one was caused by the environment and the other by an implementation gap between two libraries.
The fix here was blunt — reject non-ASCII hostnames outright. Internationalised domains are vanishingly rare in these search results, and reconciling two IDNA implementations costs far more than it returns.
A third time, left unfixed
The same gap has a third form: a window of time between the check and the connection.
I resolve the domain once to validate it, and httpx resolves again when it opens the connection. Whoever controls the authoritative DNS can make those two answers differ — a public address to pass the check, an internal one to connect to. This is classic DNS rebinding; the measured interval between the two resolutions is around 4.6 milliseconds.
I did not fix this one, and the reasoning is worth recording. A real fix means a custom httpx transport that connects to the already-validated IP, which is a substantial change. In this system, a port allowlist already narrows the reachable surface to internal ports 80 and 443, and whatever comes back enters only the local model’s context — it is never echoed to whoever initiated the request. The realistic damage is limited.
So it is filed as low risk, in writing. Known and unfixed is a different state from unknown: the first can be re-evaluated when the environment changes, the second cannot.
Other things the same review turned up
Working through the file properly surfaced several problems unrelated to that gap, some of which are likelier to bite in production.
The decompression bomb was truncated too late. What I had written was a slice of the response body:
data = r.content[:MAX_BYTES] # too lateThe problem is that by the moment r.content is accessed, httpx has already decompressed the entire body into memory; the slice happens afterwards. Measured: 0.3 MB on the wire expanded to 200 MB, driving process RSS to 747 MB. The correct approach streams, accumulating and truncating as it goes, and moves the content-type check ahead of reading the body at all:
async with client.stream("GET", url) as r:
if not r.headers.get("content-type", "").startswith("text/"):
return None
buf = bytearray()
async for chunk in r.aiter_bytes():
buf += chunk
if len(buf) >= MAX_BYTES:
breakSynchronous calls blocked the event loop. Both getaddrinfo and the text extraction are synchronous. Called directly inside a coroutine they turn “concurrent fetching” into serial fetching, and worse, they freeze every streaming conversation sharing that loop. Moving them to asyncio.to_thread fixes it — but the symptom is invisible at low concurrency.
A per-operation timeout is not a total timeout. httpx’s timeout bounds each socket operation. A peer that trickles data slowly never trips it while stretching an intended 8 seconds past 40. That needs a separate overall deadline wrapped around the whole fetch.
Parse failures must not be silent. Bing’s result titles render as <h2 class=""> rather than a bare <h2>. My first regex was wrong, so the fetch “succeeded” and parsed zero items, and what reached the model was effectively “the web has nothing on this.” The rule now is: if a page was retrieved but no items parse out of it, raise. A silent empty result is far more dangerous than an error, because downstream treats it as fact.
Guessing the encoding wrong is worse than failing to fetch. A great many Chinese-language sites declare charset only in a <meta> tag and not in the HTTP header. Forcing UTF-8 yields mojibake while the code’s fetched flag stays True — so garbage gets passed along as article text. Decoding needs to try the HTTP header, then meta, then UTF-8, then GB18030, in that order.
Three silent truncations
Pulling on the silent-failure thread further turned up three more places where data was being quietly discarded.
Deduplication merged pages that were not the same. The dedup logic stripped a URL’s query string before comparing. That is fine on most sites, but on ones that distinguish content by query parameter — a video site’s ?bvid=, a forum’s ?tid= — two entirely different pages compared equal and the second was dropped. It now strips only the fragment and keeps the query string in the comparison.
The character budget was first come, first served. Retrieved article text has a total character cap so it cannot blow out the model’s context. The original implementation filled that budget in order until it ran out. The result was that the first two or three sources consumed all of it and later sources disappeared entirely — and never appeared in the citation list, so neither the model nor the user knew they had existed. Allocating the budget evenly across sources leaves a section from each and keeps the citations complete.
Reasoning ate the token budget. Once search is enabled, page text pushes the prompt past five thousand tokens and the model’s reasoning grows accordingly. If the generation cap is still set to the few hundred that sufficed without search, reasoning consumes all of it and the final output is empty. The shape is the same as everything else here: no step reports an error, and the result is blank. There is now a warning when the budget is set too low, with 1024 as the suggested floor.
The accusations that were rejected
This review was run adversarially — assume every stage is broken, then go find the evidence. Several accusations were ultimately rejected, and those are worth recording more than the confirmed bugs, because they show where the reasoning tends to go wrong.
The first was “citation links can carry a javascript: pseudo-protocol.” It sounds reasonable and is in fact unreachable: the regex parsing search results requires an https?:// prefix, and the other code paths validate the scheme too. Attack surface has to be verified along the actual data path, not inferred from what a field could hold in principle.
The second was “the prompt contains no anti-injection language.” The error here is locating the boundary in the wrong place. What keeps prompt injection from escalating into real harm is not the wording of the prompt but the fact that this model has no tool access at all — however manipulative the page content it reads, the only thing it can produce is text. The defence is architectural, not rhetorical.
The third was “fetching has no concurrency limit.” Whoever raised it had searched only the Python sources and not the reverse proxy configuration — the rate limit lives in nginx and had been governing that endpoint all along. Looking for evidence in one file type yields a conclusion that holds within that scope and fails for the system.
What generalises
The most reusable thing here is the thread that tied three unrelated bugs together. For any guard shaped as “check first, then use,” keep one question in view: is the object you checked the same object you used?
They come apart for reasons that have nothing to do with each other — a proxy in the environment synthesising addresses, two libraries normalising the same string by different rules, a re-resolution between check and use that an attacker can steer. Three unrelated causes, one identical vulnerability shape.
The companion rule: when protection fails, the program should fail loudly. Of everything in this review, the hardest problems to find were the ones that “succeeded” with empty output — the search that silently returned nothing, the page that parsed to zero items, the body decoded into mojibake but flagged as fetched. None of them appear in an error log. They just hand wrong information to whatever comes next.