Dynamic llms.txt in a Sitecore JSS app (Next.js)

Introduction

As AI crawlers and model providers become more prevalent in scanning the open web, many organizations are looking for a clear, machine-readable way to communicate how their content might be used, especially for training versus inference. The new llms.txt convention provides a simple root-level text file for publishing those preferences. While not a formal standard, it's a nice companion to established signals like robots.txt, meta directives, and HTTP headers

When teams are developing headless sites using Sitecore JSS and Next.js, a static llms.txt can quickly become a bottleneck. Policies and contact information may change, multi-site configurations may require different policies based on the hostname, and for the editors, they need to manage that information without any code deployment. A dynamic llms.txt resolves those pain points, called at request time, it can represent content stored and managed in Sitecore, vary by site and environment, and leverage appropriate caching and revalidation.

In this guide, we focus on a dynamic approach that:

• Lets content authors manage AI policy settings in Sitecore and see them live at llms.txt

• Supports multi-tenant host mapping, localization

• Uses Next.js (JSS) to fetch settings via Experience Edge/GraphQL, merge safe fallbacks, implement caching and return text/Plain

Implementation

1) Create an Item called llms under /sitecore/content/tenant/site/settings/ using Out of the text template from the SXA and add llms content in the text field and take a note of the Item Id

• In Sitecore, create an item named llms under /sitecore/content/tenant/site/settings/

• Use the SXA text template (or a custom template with a single text field) and add your llms.txt content in its Text field.

• Note the item Id

2) In your Front End Variable Make Sure the following environment variables are added in the .env file corresponding to your environment

• NEXT_PUBLIC_SITECORE_EDGE_URL

• NEXT_PUBLIC_SITECORE_API_KEY

• NEXT_PUBLIC_LLMS_ITEM_ID

3) Add a rewrite for llms.txt in next.config.js

 async rewrites() {
    return [
      // healthz check
      {
        source: '/healthz',
        destination: '/api/healthz',
      },
          // LLMS route
      {
        source: '/llms.txt',
        destination: '/api/llms',
      },
      // robots route
      {
        source: '/robots.txt',
        destination: '/api/robots',
      }
}

4) Create the API route under pages/api by adding the below llms.ts file

import { LLMSMiddleware } from '../../lib/llms-middleware';


/**
 * API route for serving llms.txt
 *
 * This Next.js API route generates and returns the llms.txt content dynamically
 * based on the resolved site name. It is commonly
 * used by search engine crawlers to determine crawl and indexing rules.
 */

// Wire up the LLMSMiddleware handler
const handler = new LLMSMiddleware().getHandler();

export default handler;

5) under the lib folder add the below llms-middleware.ts file

// pages/api/llms.txt.ts
import type { NextApiRequest, NextApiResponse } from 'next';
import { GraphQLRequestClient } from '@sitecore-content-sdk/nextjs/client';

const EDGE_URL = process.env.NEXT_PUBLIC_SITECORE_EDGE_URL || '';
const API_KEY = process.env.NEXT_PUBLIC_SITECORE_API_KEY || '';
const LLMS_ITEM_PATH = process.env.NEXT_PUBLIC_LLMS_ITEM_ID || '';

const GET_LLMS_SETTINGS = /* GraphQL */ `
  query GetLlmsSettings($path: String!, $language: String!) {
    item(path: $path, language: $language) {
      field(name: "Text") {
        value
      }
    }
  }
`;

type GetLlmsSettingsResponse = {
  item?: {
    field?: {
      value?: string;
    };
  };
};

function createGraphQLClient(): GraphQLRequestClient {
  if (!EDGE_URL) throw new Error('Missing SITECORE_EDGE_URL');
  if (!API_KEY) throw new Error('Missing SITECORE_API_KEY');
  return new GraphQLRequestClient(EDGE_URL, { apiKey: API_KEY });
}
export class LLMSMiddleware   {
  getHandler() {
    return this.handler.bind(this);
  }
  private async handler(
  req: NextApiRequest,
  res: NextApiResponse
): Promise<void> {
  if (req.method !== 'GET' && req.method !== 'HEAD') {
    res.setHeader('Allow', 'GET, HEAD');
    res.status(405).end('Method Not Allowed');
    return;
  }

  res.setHeader('Content-Type', 'text/plain; charset=utf-8');

  try {
    const client = createGraphQLClient();
    const variables = { path: LLMS_ITEM_PATH, language: 'en' };
    const data = await client.request<GetLlmsSettingsResponse>(
      GET_LLMS_SETTINGS,
      variables
    );

    const content = data?.item?.field?.value?.trim();
    if (!content) {
      res.status(404).send('User-agent: *\nDisallow: /');
      return;
    }

    // Cache for 5 minutes to reduce load
    res.setHeader('Cache-Control', 'public, max-age=300');
    res.status(200).send(content);
  } catch {
    // Prefer a safe fallback instead of leaking errors
    res.status(500).send('Internal Server Error');
  }
}
}

Validation

Run the Application and browse the llms.txt and validate that content is shown correctly

Temporarily clear the Text field to verify the 404 fallback

Inspect response headers to confirm Content-Type: text/plain and caching behavior

Notes

1) The location of the item and the template used can vary as per your requirement. Make sure that you update the Graphql query accordingly in llms-middleware.ts file

Conclusion

With this setup, your llms.txt stays dynamic and editorially controlled, adapts to multi-site and localization needs, and can be safely cached all without redeploying your Next.js app for every policy change.