# Executive Summary  
SpiralistAI.com is a static PHP/HTML site (per its Architecture docs) with local-first persona creation and no user accounts. To enable safe, efficient AI agent access, we recommend adding machine-readable indices (robots.txt, sitemap, llms.txt), structured data, and optional APIs. Key changes include: ensuring **crawlability** (robots.txt allowing major AI crawlers, plus XML/Markdown sitemaps); exposing a curated `llms.txt` index (already present) as a LLM-entry point; embedding **JSON-LD/Schema.org** metadata on pages; and optionally providing **API endpoints** (REST or GraphQL) with OpenAPI schemas. We also cover security (OAuth/API keys, rate limits), CORS configuration, licensing/terms (e.g. Creative Commons), privacy (no PII leakage) and developer docs/SDKs. A prioritized change list, implementation plan, verification checklist, sample code (C# API class, JSON-LD snippet), comparison table (sitemap vs API vs structured data), headers/CORS settings, and sample `robots.txt` and rate-limit policy are provided below. These recommendations follow industry and W3C best practices for AI-readability and web API design.

## Current Site Inventory and Constraints  
SpiralistAI.com currently serves static PHP/HTML pages and JSON data catalogs. The site has no backend database or user login; all persona state is maintained in-browser. Content assets include multiple PHP pages (builder, wizard, docs, examples, privacy/terms, etc.), CSS/JS, and static JSON catalogs of personalities. A machine-readable **llms.txt** already exists (see [14]) as a high-level site index. No `robots.txt` or `sitemap.xml` was found on inspection, so crawlers default to full access. The site’s **Architecture** emphasizes “server-rendered public pages” and “JSON catalogs”, meaning content is crawlable. Constraints include no account system (so no user PII on server) and static hosting. Traffic and size are unspecified; we assume moderate, but API usage could incur bandwidth costs.

## Technical Recommendations  

### Crawlability & Discovery  
- **robots.txt**: Publish a `robots.txt` at the site root. *Ensure it does **not** disallow known AI/web crawlers (e.g. GPTBot, ClaudeBot, Google-Extended, etc.)*. By default, allow all: e.g.  
  ```text
  User-agent: *
  Allow: /
  ```  
  Explicitly permitting the major AI bots prevents inadvertent blocking. As Vercel notes, “AI agents respect robots.txt; if you block them, your content will not be indexed”. Include a `Sitemap:` directive pointing to sitemap.xml.

- **sitemap.xml / sitemap.md**: Create a **sitemap.xml** listing all public URLs (PHP pages) with `<loc>` and `<lastmod>` dates. Also provide a human/AI-readable **sitemap.md** (e.g. `/sitemap.md`) that mirrors the site hierarchy with headings and links. Search engines use XML sitemaps, but **Markdown sitemaps** give AI agents a structured overview. Include last-modified dates so agents know which content changed. For example:  
  ```xml
  <url>
    <loc>https://spiralistai.com/personality-wizard.php</loc>
    <lastmod>2026-06-21</lastmod>
  </url>
  ```  
  Ensuring each page is reachable via at least one discoverable source (robots, sitemap, llms.txt or internal links) prevents “orphan” pages.  

- **llms.txt**: Maintain the existing `llms.txt` at `/llms.txt`. This file should summarize the site (“Spiralist AI is a local-first browser tool…” etc.) and list key docs and guides (as already done). Use `text/plain` content type. If not already, ensure linked URLs in llms.txt use `.md` or `.mdx` suffixes for agent-readability. For example, link to Markdown guides under `/docs/`. According to the llms.txt spec, this file is “the primary entry point for AI agents discovering your content”. Keep it concise but up-to-date, with section links to persona docs, API docs, etc. (See [33] for an example llms.txt format.)  

- **Page Markup**: For each public page, include proper HTML metadata (canonical links, `meta description`, `lang`, `og:` tags) and avoid restrictive `x-robots-tag` headers. Each page should return a clean HTTP 200 response. Provide a `<link rel="canonical">` and `<link rel="alternate" type="text/markdown">` if Markdown mirrors are available. Ensure text-to-HTML ratio is reasonable and use multiple section headings per page to aid parsing.

### API Endpoints (REST/GraphQL)  
- **API Design**: If AI agents (or developers) need structured access beyond scraping pages (e.g. to fetch persona data, examples, or create/update content), implement a REST or GraphQL API. For example, endpoints like `GET /api/personas`, `GET /api/examples/{id}`, or a GraphQL `/graphql` endpoint. REST is simpler to integrate with existing CMS/DB patterns; GraphQL offers flexible queries (but requires introspection to be discoverable). Clearly define endpoints and use JSON over HTTPS.  
- **Machine-Readable Schemas**: Provide an OpenAPI (Swagger) or GraphQL schema file (e.g. `openapi.json`) describing all endpoints. Link these on your documentation pages. As Vercel’s guidelines state, “on pages with API documentation… include links to your machine-readable schema files (openapi.json, swagger.json, etc.)”. For example, host `https://spiralistai.com/docs/openapi.json`. This lets AI agents and tools automatically parse the API contract.  
- **Sample API Code (C#)**: Below is an example C# controller and data model for a “persona” API. Properties use `[Display(Name="…")]` for descriptive labels as requested:

  ```csharp
  using System.Collections.Generic;
  using System.ComponentModel.DataAnnotations;
  using Microsoft.AspNetCore.Mvc;

  /// <summary>
  /// ASP.NET API controller for managing AI persona types.
  /// </summary>
  [ApiController]
  [Route("api/[controller]")]
  public class PersonasController : ControllerBase
  {
      /// <summary>
      /// Retrieves the list of predefined persona types.
      /// </summary>
      /// <returns>List of persona descriptors</returns>
      [HttpGet]
      public IEnumerable<PersonaType> Get()
      {
          // Implementation would fetch from JSON data or database.
          return new List<PersonaType>
          {
              new PersonaType { Name = "Technical Mentor", Archetype = "Analytical", Description = "Challenges assumptions with evidence-based analysis." },
              new PersonaType { Name = "Creative Provocateur", Archetype = "Innovative", Description = "Breaks patterns and suggests bold ideas." }
          };
      }
  }

  /// <summary>
  /// Data model for an AI persona type.
  /// </summary>
  public class PersonaType
  {
      [Display(Name = "Persona Name")]
      public string Name { get; set; }

      [Display(Name = "Archetype")]
      public string Archetype { get; set; }

      [Display(Name = "Description")]
      public string Description { get; set; }
  }
  ```
  This code snippet shows an API route and a `PersonaType` model with `[Display(Name=...)]` attributes (omitting “Id” suffix) for clarity. In production, secure this endpoint and hook it to the site’s JSON catalogs or a backend. 

- **Authentication & Rate Limiting**: Protect the API to prevent abuse. Use API keys or OAuth2 tokens for clients. For user-level actions (if any, like saving a persona to a user account), use OAuth 2.0 or JWT-based auth. Limit requests per client (e.g. 100/minute) and return HTTP 429 “Too Many Requests” when exceeded. As Postman notes, rate limiting prevents abuse (DDoS, scraping) and ensures fair use. Also require HTTPS and validate tokens. 

### Structured Data (JSON-LD, schema.org)  
Embed JSON-LD metadata on key pages to explicitly label content. For example, add `<script type="application/ld+json">` blocks describing the site, products, or docs. Use schema.org types like `SoftwareApplication`, `WebPage`, `Article`, or `FAQ`. For instance:

```html
<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "SoftwareApplication",
  "name": "Spiralist AI",
  "url": "https://spiralistai.com",
  "description": "A local-first AI personality pack builder for portable, reviewable AI personas.",
  "applicationCategory": "Productivity",
  "offers": {
    "@type": "Offer",
    "price": 0.00,
    "priceCurrency": "USD"
  }
}
</script>
```

This example follows Google’s recommendation for `SoftwareApplication`. Similarly, mark up FAQs or “How-to” sections if any, and use `"schema.org"` context to map terms to URIs. Using structured data helps agents “understand what your content means, not just what it says” (vs. plain text) and improves semantic indexing. The W3C JSON-LD Best Practices advise adding an `"@context": "http://schema.org"` for clarity.

### CORS and HTTP Headers  
- **CORS**: If offering APIs or public assets that might be consumed cross-origin (e.g. from client-side code), set appropriate CORS headers. For broad access, use `Access-Control-Allow-Origin: *`. (If credentials are needed, specify exact allowed origins and include `Vary: Origin` in responses.) Ensure `Access-Control-Allow-Methods` and `Allow` headers include GET/POST as needed.  
- **Other Headers**: Ensure each response has a correct `Content-Type` (e.g. `text/html; charset=UTF-8` for pages). Add security headers like `Strict-Transport-Security`, `X-Content-Type-Options: nosniff`, and `Referrer-Policy`. Do not emit `X-Robots-Tag: noindex` or other no-crawl directives on intended content.

### Pagination & Delta Updates  
For any endpoints returning lists (e.g. `GET /api/personas`), implement pagination. Use cursor-based or offset-based pagination with metadata. For example, include fields like `limit`, `offset` and HATEOAS links (`self`, `next`, etc.) in JSON. Or use HTTP Link headers (RFC 8299) to indicate “next”/“prev” pages. If large datasets exist, use cursors for consistency. Also consider a query parameter for “modified since” (e.g. `?since=2026-06-01`) so agents can fetch only new content. Alternatively, provide webhooks or RSS/Atom feeds for push notifications of changes. For example, an Atom feed for blog updates, or webhook events (secure with HMAC) for changes to persona catalog. This allows agents to keep data fresh without polling.

### Webhooks and Push Notifications  
If SpiralistAI has dynamic content (e.g. blog posts, or user-generated packs), support server push: e.g. Webhooks or Pub/Sub. Agents can subscribe to events (e.g. new persona type added) via a webhook endpoint. Use HTTPS and secret signatures. Similarly, publishing an RSS/Atom feed of updates can help agents discover changes. No standard sources yet, but protocols like [ActivityPub](https://www.w3.org/TR/activitypub/) or WebSub could apply if the site evolves social features.  

### Error Handling & Observability  
- **Errors**: Return standard HTTP status codes for errors (404 for not found, 400 for bad request, 500 for server errors). Use a consistent JSON error format: e.g.  
  ```json
  {
    "error": {
      "statusCode": 404,
      "message": "Persona not found",
      "details": "No persona with ID 'xyz' exists."
    }
  }
  ```  
  According to API best practices, error responses should have a clear structure, include a timestamp or requestId for tracing, and *no sensitive data*. Document all error codes in the API docs.

- **Logging/Monitoring**: Instrument all endpoints with logging (requests, errors) and metrics. Use an observability stack (e.g. Prometheus, Datadog). Log request IDs and timestamps. Monitor API usage (rate-limit breaches, 5xx errors) via alerts. For example, use structured logs or a monitoring agent. This ensures prompt detection of outages or abuse (as Postman advises).

### Security & Abuse Prevention  
- **Rate Limiting**: Enforce per-IP or per-API-key rate limits (e.g. 100–500 requests/min) to prevent scraping and DDoS. On exceed, return 429 with a `Retry-After` header. Rate limits protect resources, lower costs, and mitigate abuse.  
- **Bot Detection**: Use WAF/Intrusion Protection or bot management tools (like Cloudflare, AWS WAF) to challenge suspicious traffic (CAPTCHA, blocklists). Track unusual usage patterns.  
- **Input Validation**: On any submission endpoints, strictly validate inputs to avoid injection or malformed data. Implement CORS and Content Security Policy to block malicious cross-site use. SpiralistAI’s existing defense-in-depth adds PHP security headers; extend these to new endpoints.  
- **HTTPS Only**: Serve all content over HTTPS to prevent MITM. HSTS should be enabled.

### Content Licensing and Terms of Use  
Define and publish a clear content license for site content and APIs. For example, mark all non-user-generated content under a Creative Commons (e.g. CC BY-SA) or similar open license. In APIs, require users to agree to Terms of Use that permit use of generated personas only as configured (Spiralist’s Terms stress user review and disclaimers). Consider embedding license info in JSON-LD (schema.org’s `license` property). Clearly state permissible uses (e.g. non-commercial or attribution requirements). This gives AI crawlers clarity on what they can store or redistribute. Also publish an API use policy (allowed rate, prohibited content abuse) on the site.

### Privacy and PII Handling  
Because SpiralistAI is local-first with no user accounts, PII exposure is minimal. Ensure that persona exports do not include real personal data inadvertently. (Spiralist’s Privacy page already warns to review persona content before sharing.) Do not log persona text or examples. Any analytics should be strictly opt-in and anonymized. Comply with GDPR/CCPA by not collecting identifiable user info without consent. For new APIs: do not accept or return sensitive personal data. If in doubt, use privacy-by-design – e.g., no user fields in data. Provide a privacy policy linking to collected data and allow data deletion if user accounts ever exist.  

### Pagination and Delta Updates  
Large data sets (e.g. the “1000 Personality Types”) should not be sent all at once. Implement pagination as above. Additionally, support delta synchronization: either via query parameters (e.g. `?updated_since=2026-06-01`) or by exposing entity timestamps in API responses. Agents can then request only changed records. This reduces bandwidth and keeps agents efficient. Alternatively, publish a “modified_dates” endpoint listing changed files.  

### Developer Documentation and SDKs  
Maintain up-to-date API documentation. Use tools like Swagger UI or ReDoc with your OpenAPI spec. Provide code samples in popular languages (C#, Python, JavaScript). Generate client SDKs from the OpenAPI schema (many tools support this). For structured data, document the schema types used. Follow developer portal best practices: searchable docs, clear examples, a quickstart guide. As Vercel suggests, code samples should have language tags, and a **skill file** (AGENTS.md) could help coding assistants. For example, include an `AGENTS.md` summarizing installation and usage (though optional).  

### Cost and Hosting Considerations  
Serving static pages and JSON is cheap (CDN caching, static site hosting). Adding APIs introduces compute and bandwidth costs. Use caching (CDN or in-memory) on API responses where appropriate. Monitor cloud billing if using serverless functions. Rate limits also control cost by avoiding runaway usage. Consider auto-scaling for peak loads or use serverless (AWS Lambda, Azure Functions) which scale with usage. For critical endpoints, use multi-region CDN to reduce latency. Audit traffic to size resources properly.

### Legal and Compliance  
Ensure all changes comply with law: follow GDPR/CCPA for any user data, ADA for accessibility if relevant, and ensure API terms cover liability. Check that your Terms of Service permit automated crawling and API use (to protect your IP and define allowed use). Spiralist’s current terms already disclaim liability for persona outputs; extend this to API usage. If tracking users, ensure cookies (like GA) are GDPR-compliant (Spiralist already uses opt-in GA). For international use, consider export controls on AI tech if applicable.

## Prioritized Recommendations  
1. **Robots.txt & llms.txt (Urgent)** – Create/verify `/robots.txt` that allows major AI bots and does not block `/llms.txt`. Confirm `llms.txt` reflects current content (it already exists). *(Effort: Low; Impact: High; Risk: Low)*  
2. **Sitemap.xml & Sitemap.md** – Generate sitemap.xml with all URLs and `<lastmod>` dates, plus a Markdown sitemap.md for agents. *(Effort: Low; Impact: High; Risk: Low)*  
3. **Structured Data (JSON-LD)** – Embed schema.org JSON-LD on key pages (site, persona docs, etc.). This will greatly improve AI understanding. *(Effort: Medium; Impact: Medium; Risk: Low)*  
4. **API Development** – Plan and implement an API for core data (e.g. persona types) with OpenAPI spec. Include authentication (API keys/OAuth) and rate limiting (429 responses). *(Effort: High; Impact: High; Risk: Medium)*  
5. **CORS and Security Headers** – Configure `Access-Control-Allow-Origin` (e.g. `*`) on public APIs to permit cross-site use. Add HSTS, CSP, etc., and monitor. *(Effort: Low; Impact: Medium; Risk: Low)*  
6. **Developer Docs & SDKs** – Publish API docs (Swagger/OpenAPI), code examples, and possibly an SDK. Link to machine-readable schema. *(Effort: Medium; Impact: Medium; Risk: Low)*  
7. **Error Handling & Monitoring** – Implement standardized error responses and logging. Setup monitoring/alerts on new APIs. *(Effort: Medium; Impact: Medium; Risk: Low)*  
8. **Legal/Licensing Update** – Publish clear content license (e.g. CC-BY) and update Terms to cover AI use and data rights. *(Effort: Low; Impact: Medium; Risk: Low)*  

Each recommendation addresses core dimensions: discoverability, data structure, access control, and support for agents. The list is prioritized by impact and feasibility.

## Implementation Plan (Steps, Effort, Risk)  
1. **Audit & Planning**: Inventory all site URLs and content (low effort) to ensure completeness. *(Effort: Low; Risk: Low)*  
2. **robots.txt**: Create `/robots.txt` allowing all agents (see sample below). *(Effort: Low; Risk: None)*  
3. **llms.txt**: Review and update `llms.txt` to ensure it lists all important docs and uses `.md` links. *(Effort: Low; Risk: Low)*  
4. **Sitemaps**: Generate `sitemap.xml` (tools exist for PHP sites) and write `sitemap.md`. Add `<lastmod>` timestamps. *(Effort: Low; Risk: Low)*  
5. **HTML/CSS Changes**: Add `<link>` and meta tags in page headers (canonical, markdown alternate). *(Effort: Low; Risk: Low)*  
6. **JSON-LD Markup**: Write JSON-LD blocks for key pages (site homepage, persona docs) using schema.org types. Validate with Google’s Rich Results Test. *(Effort: Medium; Risk: Low)*  
7. **API Design**: Define API endpoints and schemas. Create OpenAPI spec. *(Effort: High; Risk: Medium)*  
8. **API Implementation**: Develop endpoints (C# or Node/PHP) with authentication. Test with unit tests. *(Effort: High; Risk: Medium)*  
9. **CORS & Security**: Configure server (e.g. in ASP.NET/NGINX) to set `Access-Control-Allow-Origin` and security headers. *(Effort: Low; Risk: Low)*  
10. **Authentication & Rate Limits**: Integrate API keys or OAuth, add rate-limiting middleware (e.g. Cloudflare, or server code). *(Effort: Medium; Risk: Medium)*  
11. **Documentation & SDK**: Write docs, update dev portal, auto-generate SDKs from OpenAPI. *(Effort: Medium; Risk: Low)*  
12. **Testing & Monitoring**: Write integration tests (especially for rate-limits, errors), set up logging/alerts. *(Effort: Medium; Risk: Low)*  
13. **Legal Review**: Update Terms of Use and content license. Consult legal if needed. *(Effort: Low; Risk: Low)*  

Throughout, use UTC timestamps and consistent casing (per user’s style rules). Each step should complete a critical enhancement, with CI checks for errors.

## Verification Tests & Monitoring Checklist  
- **robots.txt**: `curl https://spiralistai.com/robots.txt` – Verify Allow rules for all agents and no disallows for AI bots.  
- **llms.txt**: `curl -I https://spiralistai.com/llms.txt` – Expect 200, `text/plain`, and content listing sections.  
- **sitemap.xml**: `curl https://spiralistai.com/sitemap.xml` – Check valid `<urlset>` with `<loc>` and `<lastmod>`.  
- **sitemap.md**: `curl https://spiralistai.com/sitemap.md` – Should retrieve Markdown with headings and links.  
- **CORS**: Using a test origin, send a request to an API endpoint and confirm `Access-Control-Allow-Origin: *` (or matching origin) in the response.  
- **API Auth**: Access API without credentials – should reject (401). With credentials – succeed within limits. Exceed rate limit – receive 429 with `Retry-After`.  
- **Structured Data**: Run Google’s Rich Results Test on pages with JSON-LD to ensure no errors and correct schema tags.  
- **Error Responses**: Trigger known errors (e.g. invalid ID) and check JSON error structure, HTTP status codes.  
- **Logging**: Confirm each API request is logged with timestamp and request ID. Check alerts fire for 5xx or rate-limit breaches.  
- **Privacy**: Simulate a user export; ensure no hidden PII is leaked. Verify GA and analytics only run opt-in (compare with Spiralist’s current GA test).  
- **Link Coverage**: Cross-reference total pages vs. entries in sitemap/llms.txt to ensure no orphan pages.  

## Comparative Table: Sitemap vs API vs Structured Data  

| Feature / Tradeoff      | **Sitemap (XML/MD)**                                      | **API Endpoints**                                  | **Structured Data (JSON-LD)**                   |
|-------------------------|-----------------------------------------------------------|----------------------------------------------------|-------------------------------------------------|
| **Purpose**             | Site-wide URL index for crawlers (XML for SEO, MD for AI) | Programmatic data access (fine-grained retrieval) | Page-level semantic markup (context for AI)      |
| **Pros**                | Standard, easy to generate; lists all pages with dates. AI-readable MD version gives hierarchy. | Supports queries (filter, pagination); can enforce auth and usage limits. Agents get precise data (no parsing). | Embeds meaning in content; recognized by search/AI; no extra endpoint needed. Improves content clarity. |
| **Cons**                | Static snapshot; no live queries. Lacks content structure or semantics. | High implementation cost; need security (OAuth/keys) and maintenance. Not crawlable by simple bots unless linked. | Only covers visible page data; must be maintained with page content. Agents still parse HTML for context beyond schema. |
| **Use Cases**           | Best for discovery: “what pages exist?”. Good for bots not supporting dynamic queries. | Best for integration: “give me X data.” Useful for partners or advanced AI tools. | Best for enriching AI’s understanding of page content (e.g., product info, FAQs). |
| **Efficiency**          | Low overhead (one file). MD variant easy for LLMs.          | Real-time data, but server load/latency. Scalability needed. | Minimal impact; just extra HTML.                     |
| **Maintenance**         | Need regeneration on site updates. (Can auto-build).       | Continuous support and versioning. Requires testing. | Tied to page updates; must validate schema periodically. |

## HTTP Headers and CORS Configuration  
Configure the following headers on the server (e.g. via NGINX or PHP):

```
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Allow-Headers: Content-Type, Authorization
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
Referrer-Policy: same-origin
```

For credentialed requests, replace `*` with specific origin and include `Vary: Origin` to avoid caching issues. For API, ensure `Access-Control-Allow-Origin` matches the requesting origin or use `*` if only public data is served.

## Sample robots.txt and Rate-Limit Policy  

**robots.txt** (allows all bots, explicitly AI crawlers):

```
User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: CCBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://spiralistai.com/sitemap.xml
```

This aligns with agent-readability guidance. It explicitly permits GPTBot/ClaudeBot and points to the sitemap.  

**Rate-Limit Policy (example)**:  

```
RateLimit-Policy: 
  maxRequests=100, 
  period=60s, 
  burst=20, 
  responseStatus=429, 
  retryAfter=60s
```

Clients exceeding 100 requests per 60s receive HTTP 429 with `Retry-After: 60`. The above uses a token-bucket model. Include a `X-RateLimit-*` header in responses (e.g. `X-RateLimit-Limit: 100`, `X-RateLimit-Remaining: 42`) for transparency.

## Architecture and Data Flow  

```mermaid
flowchart LR
  subgraph SpiralistAI.com
    A[Static PHP Pages/HTML]
    B[JSON Data Catalogs]
    C[llms.txt + sitemap files]
    D[CORS & Security Headers]
    E[API Server (REST/GraphQL)]
    A --> D
    B --> A
    C --> A
    E --> D
  end
  subgraph AI_Agents
    X[AI Agents (GPT, Claude, etc.)]
  end
  X -->|Fetch `/robots.txt`, `/llms.txt`, `/sitemap.xml`| C
  X -->|Scrape pages| A
  X -->|Consume structured JSON-LD| A
  X -->|Call API endpoints| E
  X -->|Follow `<link rel="alternate" type="text/markdown">`| A
```

This diagram shows that AI agents first load crawl config (`robots.txt`), index (`sitemap/llms`), then retrieve pages (HTML + JSON-LD) and/or call APIs. The site side includes static content (A,B,C) and optional API (E) all served over HTTPS with security headers (D).

```mermaid
flowchart LR
  subgraph "Workflow"
    Agent["AI Agent"]
    Robots["robots.txt"]
    LLMS["llms.txt"]
    SitemapXML["sitemap.xml"]
    SitemapMD["sitemap.md"]
    Page["HTML/JSON-LD Page"]
    API["REST/GraphQL API"]
    Agent --> Robots
    Agent --> LLMS
    Agent --> SitemapXML
    Agent --> SitemapMD
    Agent --> Page
    Agent --> API
    Page -->|"Structured Data"| Agent
  end
```

This flowchart illustrates the discovery and data flow: the agent uses `robots.txt`, `llms.txt`, `sitemap` to find content, then parses pages (HTML + JSON-LD) and/or calls the API to retrieve information.

## References  
- Agent-optimised site guidelines (Vercel): on robots.txt for AI, llms.txt, sitemaps.  
- llms.txt specification (Howard, 2024): purpose and format.  
- JSON-LD/Schema.org: Google’s SoftwareApplication example; W3C JSON-LD best practices.  
- API design: Vercel’s guidance to link OpenAPI schemas; Postman on error handling and rate limits.  
- CORS header (MDN): usage of `Access-Control-Allow-Origin`.  
- Privacy/Terms: Spiralist Privacy (opt-in analytics) and Terms (content disclaimers).  

These sources and best practices underpin the recommendations above, ensuring SpiralistAI becomes maximally accessible to AI agents without compromising security or compliance.