If you put content behind a paywall or registration wall, you face a real tension: you need search engines and AI to know the content exists so it can rank and be cited, but you don't want them — or freeloaders — reading the whole thing for free. Show bots the full text while showing humans a paywall and you risk being flagged for cloaking. Hide the content from bots entirely and you become invisible. The resolution is a specific technique built around the isAccessibleForFree property and structured content selectors. This guide explains how to keep paywalled pages discoverable without cheating and without giving your work away.
The core problem: cloaking vs discoverability
Cloaking means showing search engines materially different content from what users see, and it is a violation that can get pages or whole sites demoted. A naive paywall implementation trips this wire: if you serve Googlebot the full article but serve visitors a "subscribe to continue" gate, you are showing two different pages. But the opposite extreme is just as bad — if bots only ever see the teaser, they have almost nothing to index, and AI assistants have nothing to cite. You need a way to let engines understand the full value of the page while being honest that humans hit a wall.
The sanctioned answer is not to hide the paywall from bots, but to declare it. You tell engines, in structured data, exactly which parts of the page are free and which are gated. Done correctly, this is explicitly allowed — it is flexible sampling, not cloaking, because you are being transparent about the arrangement rather than deceiving the crawler.
How isAccessibleForFree works
The mechanism has two parts, both expressed in JSON-LD structured data on the page. First, set isAccessibleForFree to false on the CreativeWork (your Article, NewsArticle, or similar). This declares that the content is not fully free. Second, add a hasPart entry describing the gated section: a WebPageElement with its own isAccessibleForFree set to false and a cssSelector pointing to the DOM element that contains the paid content.
That selector is the key. It tells the engine, "the element matching this selector is behind the paywall." The crawler can then read the full page — including the gated text — understand what the article is about, and index it accordingly, while knowing that human visitors will be asked to subscribe. Because you've declared the arrangement honestly, serving the full HTML to the bot is no longer cloaking. This only works if the markup accurately matches the page structure, which is why validating it, as covered in our guide to validating structured data, matters before you rely on it.
What this means for AI citation
AI assistants and answer engines want to cite authoritative content, and a lot of the best content — investigative journalism, research, expert analysis — lives behind paywalls. When you declare your paywall correctly, you give AI systems the context to understand and reference your work even if the reader is then prompted to subscribe. This is far better than being invisible: an assistant that knows your article is the definitive source on a topic can point users to it, which is a subscription opportunity rather than a lost one.
How much an assistant quotes from gated content varies by system and by the licensing arrangements behind it, and that landscape is still shifting. But the baseline is clear: a paywalled page with proper structured data is discoverable and citable, while a paywalled page that simply blocks bots is neither. Being understood is a prerequisite for being chosen — the same logic behind our guide to how ChatGPT chooses which sites to cite.
Getting the implementation right
A few rules keep a paywalled setup safe and effective:
- Never serve bots different content without declaring it. The full text is fine to show a crawler only when your structured data says the section is gated.
- Match the cssSelector to the real element. If the selector points at the wrong container, engines can't tell what's gated and may treat the page as cloaked.
- Give a meaningful free preview. A strong, self-contained opening — headline, summary, first section — gives both humans and AI enough to grasp the value, following the same self-contained approach as Article and BlogPosting schema.
- Don't block the page in robots.txt. If you disallow crawling, none of this works — the engine never reads the markup. Control access with the paywall and structured data, not by cutting off the crawler.
- Keep metadata honest. Titles and descriptions should describe the real article, not overpromise what a free visitor will see.
Paywalls and discoverability are not mutually exclusive — they only conflict when you try to trick the crawler instead of informing it. Declare your gated content with isAccessibleForFree and a precise cssSelector, offer a genuine free preview, keep the page crawlable, and you get the best of both: search engines and AI understand and surface your work, while humans still hit the subscribe button. To check whether crawlers can read your paywalled pages and whether your structured data is declaring the gate correctly, run the CheckMy.site scanner and fix what it flags.