Learn · How-To

How to Find the Links Only Your Browser Can See

You will find plenty of confident percentages online about how much JavaScript AI crawlers execute. We are not going to quote one, because the numbers circulating are secondary citations of each other and we cannot trace them to a study we would stand behind. What you can do instead is measure your own site, which gives you an answer that is actually about you. Google renders JavaScript, though on its own schedule. Several AI crawlers do not render at all. The gap between what your browser shows and what a plain fetch returns is your exposure, and it is measurable in an hour.

1. 1

   Fetch a page the way a non-rendering crawler does

   From a terminal, request the raw HTML of an important page without executing any scripts — curl or wget is enough. Save the output to a file. This is roughly what a crawler that does not render JavaScript receives, and it is frequently not what you expect.

2. 2

   Compare it against what your browser displays

   Open the same page in a browser, then view the rendered DOM in developer tools. Put the two side by side. Anything present in the browser and absent from the raw fetch exists only after JavaScript runs, and any crawler that does not render will never see it.

3. 3

   Count the links specifically, not just the words

   Extract every href from the raw HTML and compare that list to the links visible on the rendered page. Links are the part that matters most, because a link that only exists after render is a page a non-rendering crawler cannot discover at all. Missing body copy is a content problem; missing links are a discovery problem.

4. 4

   Check your primary navigation first

   Menus built entirely in client-side frameworks are the most common and most damaging case, because every page below them inherits the problem. If your main navigation is absent from the raw HTML, fix that before anything else on this list.

5. 5

   Check pagination, filters and "load more"

   Category listings, product filters and infinite-scroll patterns very often produce URLs that exist only in response to a click. If a crawler cannot reach page two without executing JavaScript and clicking, treat everything beyond page one as undiscoverable and provide real paginated links.

6. 6

   Confirm what your own logs show

   Filter your server logs by user agent for GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and Googlebot, and look at which URLs each one actually requested. This is the ground truth: if a crawler has never requested a URL, it has not indexed it, regardless of what any tool told you. Your logs beat anyone's published percentage.

7. 7

   Fix by moving links into the served HTML

   The remedy is almost always the same: make sure the URL exists as a real anchor in the HTML the server sends. Static generation, server-side rendering or simply including real links alongside the interactive version all work. You do not need to abandon your framework; you need the link to survive without it.