Fetching LLM-Suggested URLs Safely: an SSRF Guard in Go

More and more backends now ask a language model to find a URL and then fetch it: a search-grounded model suggests an official website, an agent follows a link it extracted from a document, a pipeline enriches records with pages the model picked. The moment your server issues an HTTP request to a URL that came out of a model, you have built a classic SSRF surface, and the usual "validate the hostname" advice is not enough. This post walks through a production Go fetcher that treats model output as untrusted input: static URL checks, dial-time IP pinning against DNS rebinding, an address blocklist that covers the ranges standard library helpers miss, per-hop redirect re-validation, and a final check that the page you fetched is actually the page you wanted.

Why model-returned URLs are untrusted input

A URL from a model is shaped by three parties you do not control. First, the model itself can hallucinate or pick a look-alike domain. Second, when the model uses a search tool, the returned links are often tracking redirectors owned by the search provider, so the URL you see is not the URL you will land on. Third, and worst, the content the model read can steer it: a page can contain text that convinces the model to emit an attacker-chosen URL. Prompt injection turns "the model suggested a link" into "an attacker suggested a link" with extra steps.

From the server's point of view this is the same threat as a user-submitted webhook URL. The attack that matters is Server-Side Request Forgery: getting your backend to issue requests to targets only it can reach. The classic prizes are cloud metadata endpoints, which hand out service account tokens, and internal services that trust any caller on the private network. If your fetcher runs on a cloud VM or a container platform, both are one HTTP request away.

So the design rule is simple to state: the fetcher must enforce, on its own, that every connection it opens goes to a public address, no matter what the URL says, no matter what DNS says the second time you ask.

Static URL checks come first, but they are not enough

Cheap checks go first because they reject most garbage before any network work:

func validateFetchURL(u *url.URL) error {
    if u.Scheme != "http" && u.Scheme != "https" {
        return fmt.Errorf("scheme not allowed: %q", u.Scheme)
    }
    if u.User != nil {
        return fmt.Errorf("userinfo in URL rejected")
    }
    switch u.Port() {
    case "", "80", "443", "8080":
    default:
        return fmt.Errorf("port not allowed: %s", u.Port())
    }
    host := u.Hostname()
    if host == "" {
        return fmt.Errorf("empty host")
    }
    if strings.EqualFold(host, "metadata.google.internal") {
        return fmt.Errorf("metadata host rejected")
    }
    return nil
}

Scheme pinning kills file, gopher, and friends. Rejecting userinfo kills the https://[email protected]/ confusion trick. A port allowlist keeps the fetcher from being used as a generic port scanner. A named metadata host is worth a special case because it is the one hostname attackers try first on GCP.

What this function deliberately does not do is resolve the hostname and check the IP. That would feel natural here, and it is exactly the mistake that opens the next hole: if you resolve during validation and then let the HTTP client resolve again during connection, the two answers do not have to match.

Resolve once, check every IP, dial what you checked

DNS rebinding is a time-of-check to time-of-use bug. An attacker-controlled nameserver answers your validation query with a harmless public address, then answers the client's connection query with the metadata address or an internal RFC 1918 address. Your check passed; your request went to the internal target anyway. Low TTLs make this reliable in practice.

The fix is structural, not another check: do the resolution and the policy decision inside the dialer, and then connect to the literal IP you just approved. Go makes this clean because http.Transport accepts a custom DialContext:

func safeDialContext(ctx context.Context, network, addr string) (net.Conn, error) {
    host, port, err := net.SplitHostPort(addr)
    if err != nil {
        return nil, err
    }
    ips, err := net.DefaultResolver.LookupIP(ctx, "ip", host)
    if err != nil || len(ips) == 0 {
        return nil, fmt.Errorf("resolve failed for %s", host)
    }
    for _, ip := range ips {
        if !publicIP(ip) {
            return nil, fmt.Errorf("non-public IP blocked: %s", ip)
        }
    }
    d := &net.Dialer{Timeout: 10 * time.Second}
    var lastErr error
    for _, ip := range ips {
        conn, dialErr := d.DialContext(ctx, network, net.JoinHostPort(ip.String(), port))
        if dialErr == nil {
            return conn, nil
        }
        lastErr = dialErr
    }
    return nil, lastErr
}

Two details carry the security weight. Every returned address must pass the policy, not just the first one, because the client may fail over to any of them. And the dial goes to ip.String(), not back to the hostname, so there is no second resolution for the attacker to poison. TLS still works: the transport performs the handshake with the original hostname for SNI and certificate verification, so pinning the TCP connection to a vetted IP costs nothing in correctness.

The addresses a private-range check misses

The obvious implementation of publicIP calls ip.IsPrivate() and ip.IsLoopback() and stops. That leaves real gaps. Here is a predicate that has survived review:

func publicIP(ip net.IP) bool {
    if ip == nil {
        return false
    }
    if ip.IsLoopback() || ip.IsPrivate() || ip.IsLinkLocalUnicast() ||
        ip.IsLinkLocalMulticast() || ip.IsMulticast() || ip.IsUnspecified() {
        return false
    }
    if v4 := ip.To4(); v4 != nil && v4[0] == 169 && v4[1] == 254 {
        return false
    }
    if v6 := ip.To16(); v6 != nil && ip.To4() == nil &&
        v6[0] == 0x00 && v6[1] == 0x64 && v6[2] == 0xff && v6[3] == 0x9b {
        return false
    }
    return true
}

The link-local check matters because 169.254.169.254, the metadata address on the major clouds, is link-local, not RFC 1918 private. IsLinkLocalUnicast covers it, and the explicit byte check is redundancy for a contract you never want to lose to a refactor.

The last block is the one most fetchers miss: the NAT64 well-known prefix, 64:ff9b::/96. On a NAT64 network, an IPv6 address in that prefix encodes an IPv4 address in its low 32 bits, and the translator will happily forward your connection to it. An attacker who cannot get a private IPv4 address past your filter can submit its NAT64-encoded IPv6 form instead. IsPrivate says that address is fine, because as an IPv6 address it is global unicast. You have to reject the prefix yourself. IPv4-mapped IPv6 addresses, the ::ffff: prefixed forms, are less of a trap in Go, since the net package normalizes them before the standard predicates run, but the NAT64 prefix gets no such help.

Redirects, proxies, and response caps

A vetted first hop proves nothing about the second. Search redirectors are the normal case for model-suggested links, so the fetcher must follow redirects, and each hop is a fresh chance to land somewhere internal. The client re-runs the same static validation on every hop and caps the chain:

client := &http.Client{
    Timeout: 20 * time.Second,
    Transport: &http.Transport{
        Proxy:       nil,
        DialContext: safeDialContext,
    },
    CheckRedirect: func(req *http.Request, via []*http.Request) error {
        if len(via) >= 5 {
            return fmt.Errorf("too many redirects")
        }
        return validateFetchURL(req.URL)
    },
}

Proxy: nil is easy to overlook. The default transport honors HTTP_PROXY environment variables, and a proxied connection dials the proxy, not the target, which means your careful dialer checks the proxy's address and nothing else. Turning the proxy off keeps the dial-time guard authoritative.

The response side gets the same distrust. Read through io.LimitReader with a hard byte cap so a hostile server cannot feed you gigabytes, require an HTTP 200, and check the Content-Type against what you intend to parse. Record the final URL after redirects, normalized, and treat that as the canonical address of what you fetched; storing the pre-redirect URL means storing the redirector.

Verify you got the page you asked for

Everything so far protects your network. It does nothing for correctness: the model can hand you a perfectly public, perfectly safe, completely wrong page. If the fetch exists to confirm that a URL is the official site of a specific entity, the fetcher should check that claim before anything downstream trusts it.

Two cheap checks catch most wrong answers. First, language: if you expect a Korean page, require at least one Hangul character in the visible text after stripping tags, comments, and script and style blocks. Second, name proximity: normalize both the expected entity name and the page text by removing whitespace and lowercasing, then require the name to appear in the text. Normalization matters because real pages space names inconsistently, and the same name with and without internal spaces should match.

func normalizeForMatch(s string) string {
    var b strings.Builder
    for _, r := range strings.ToLower(s) {
        if !unicode.IsSpace(r) {
            b.WriteRune(r)
        }
    }
    return b.String()
}

Alongside verification, keep a plain domain blocklist for hosts that can never be a correct answer in your domain, such as social profiles, map listings, and blog platforms, and check it both before the fetch and on every redirect hop, because redirectors bounce to those hosts constantly. This is a relevance filter, not a security control, but it runs in the same place and saves a fetch.

What this design does not try to solve

A few honest limits. The dialer trusts your resolver: if an attacker controls your DNS path entirely, they control more than this fetcher. IPv6 has other translation prefixes beyond the well-known one; if your network uses a custom NAT64 prefix, add it to the blocklist. The page verification is a heuristic, and a determined attacker can put the expected name on a page they control; treat it as reducing accidental wrong registrations, not as authentication. And a response cap does not help if you then hand the HTML to a parser with quadratic behavior, so keep parsing bounded too.

The pattern generalizes past LLM output. Webhook URLs, user-submitted RSS feeds, link previews, and OAuth redirect targets all want the same shape: static checks up front, policy enforced at dial time against the literal address, every hop re-validated, response size and type capped, and a domain-level sanity check for the use case. Once the guard exists as a small package with a custom dialer, using it is one Transport swap away for any client in the codebase, which is the difference between a security review finding and a default.