SEO & AI Engine Optimization Framework · May 2026

Agentic AI Search: optimization for autonomous agent-driven search and task execution

A comprehensive installation and audit reference for agentic AI search — the new layer where AI systems don't just answer queries with cited sources, but autonomously browse the web, execute…

Multi-Step AI Agents, Autonomous Browsing, Tool-Using AI, Site Compatibility for Agent Workflows, and the Emerging Layer Above Traditional Search

A comprehensive installation and audit reference for agentic AI search — the new layer where AI systems don't just answer queries with cited sources, but autonomously browse the web, execute multi-step research workflows, fill out forms, complete transactions, and act on behalf of users. This is a 2026 frontier requiring different optimization than traditional search or even AI engine citation.

Cross-stack implementation note: the code samples in this framework are written in plain HTML for clarity. For React, Vue, Svelte, Next.js, Nuxt, SvelteKit, Astro, Hugo, 11ty, Remix, WordPress, Shopify, and Webflow equivalents of every pattern below, see framework-cross-stack-implementation.md. For pure client-rendered SPAs (no SSR/SSG) see framework-react.md. For Tailwind-specific concerns (purge, dynamic classes, dark-mode CLS, focus accessibility) see framework-tailwind.md.

1. Document Purpose

The shift from "AI assistants that cite sources" to "AI agents that take actions" is profound. Where ChatGPT in 2023 read static training data, where Perplexity in 2024 retrieved current sources, the 2026 layer of AI search is different again: agents like Claude with computer use, OpenAI's Operator, Google's Project Astra-derived systems, and emerging open agents browse the live web, execute multi-step research, evaluate options across sites, and complete tasks on behalf of users.

For SEO, this creates new optimization dimensions:

This framework specifies optimization for the agentic AI layer. It complements rather than replaces:

The agentic layer adds requirements specific to autonomous AI workflows.

1.1 The Agent Landscape (May 2026)

current_agent_landscape:
  
  consumer_agents:
    - Claude with computer use (Anthropic)
    - OpenAI Operator
    - Google's Astra-derived consumer agents
    - Perplexity Pages and assistants
  
  developer_platforms:
    - Anthropic Computer Use API
    - OpenAI Assistants API with browsing
    - LangChain-powered custom agents
    - LlamaIndex agentic frameworks
  
  enterprise_agents:
    - Internal agents at major companies
    - Industry-specific agents (legal research, medical research, etc.)
  
  emerging_patterns:
    - Browser-native agents (extensions, sidepanels)
    - Workflow automation tools (Zapier-like with AI)
    - Multi-agent systems coordinating tasks

The pace of change is rapid. Specific tool names will date quickly. The patterns this framework captures will remain.

1.2 Required Tools

2. Site Architecture for Agents

2.1 What Agents Need

Agents share many needs with traditional crawlers but differ in important ways:

agent_requirements:
  
  shared_with_crawlers:
    - Clean HTML structure
    - Server-rendered content (or fast JS rendering)
    - Schema markup for semantic understanding
    - Sitemap for discovery
    - HTTPS for trust
  
  unique_to_agents:
    - Action elements identifiable by purpose, not just appearance
    - Form fields with proper labels and ARIA
    - Multi-step workflows possible to complete
    - Error states recoverable
    - Confirmation states clear
    - State preservable across navigation
    - Anti-bot protection that distinguishes agents from malicious bots

2.2 Action Affordances

Where traditional SEO focused on content discovery, agentic optimization adds focus on actions agents can take.

Common agent actions on websites:

For each, the question: can an agent do this reliably?

Test methodology: Use Claude computer use or similar to attempt the workflow on your site. Observe:

2.3 Form Optimization for Agents

Forms are common agent interaction points. Optimization:

<!-- Agent-friendly form -->
<form>
  <label for="name">Full Name</label>
  <input 
    type="text" 
    id="name" 
    name="name" 
    required
    aria-required="true"
    autocomplete="name"
    placeholder="Enter your full name">
  
  <label for="email">Email Address</label>
  <input 
    type="email" 
    id="email" 
    name="email" 
    required
    aria-required="true"
    autocomplete="email"
    placeholder="you@example.com">
  
  <label for="subject">Subject</label>
  <select id="subject" name="subject" required>
    <option value="">Choose a subject</option>
    <option value="quote">Request a quote</option>
    <option value="support">Technical support</option>
    <option value="general">General inquiry</option>
  </select>
  
  <label for="message">Message</label>
  <textarea 
    id="message" 
    name="message" 
    required
    minlength="10"
    aria-describedby="message-help"></textarea>
  <span id="message-help">Please describe your inquiry (minimum 10 characters)</span>
  
  <button type="submit">Send Message</button>
</form>

Agent-friendly form patterns:

2.4 Multi-Step Workflows

For workflows requiring multiple steps (e.g., booking, checkout):

multi_step_workflow_optimization:
  
  state_preservation:
    - Use URL parameters or session state agents can preserve
    - Provide clear back navigation
    - Allow editing previous steps
  
  progress_indication:
    - Clear "Step X of Y" indicators
    - Progress visualization
    - Estimated time remaining
  
  error_recovery:
    - Specific error messages identifying the problem
    - Path back to fix
    - Don't lose user input on errors
  
  confirmation:
    - Clear confirmation page after completion
    - Reference number or confirmation token
    - Email confirmation as backup
    - Schema markup confirming the action

3. Bot Identification & Access

3.1 The Agent Identification Problem

In 2026, distinguishing between:

...is increasingly important and increasingly difficult.

3.2 Robots.txt for Agents

Building on framework-aicitations.md Section 4, the 2026 robots.txt landscape:

# Search crawlers — allow
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: DuckDuckBot
Allow: /

# AI training/research bots — decide based on policy
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: CCBot
Allow: /

User-agent: Applebot-Extended
Allow: /

# Agent-specific identification (emerging in 2026)
User-agent: Claude-User                  # Claude acting on behalf of a user
Allow: /

User-agent: ChatGPT-User                 # ChatGPT acting on behalf of a user
Allow: /

User-agent: PerplexityBot-User           # Perplexity acting on user behalf
Allow: /

# Malicious or unwanted scrapers
User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: DotBot
Disallow: /

# Catch-all
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

The "Claude-User" / "ChatGPT-User" distinction is emerging — bots acting on behalf of specific users (rather than indexing for training) often want to be allowed even when training bots are blocked. Decisions are policy-specific.

3.3 IP-Based Identification

Some agent platforms publish IP ranges. Allowing those IP ranges while blocking other scrapers gives finer control than user-agent string alone.

3.4 Cloudflare and Similar Platforms

Modern WAF platforms have agent-specific rules:

cloudflare_agent_handling:
  
  ai_audit_tab:
    description: "Cloudflare's AI Audit shows traffic from major AI platforms"
    actions: "Block, challenge, or allow per platform"
  
  super_bot_fight_mode:
    description: "Distinguishes verified bots from malicious"
    settings:
      - allow_verified_bots: true
      - challenge_likely_bots: true
      - block_known_bad_bots: true
  
  custom_rules:
    - Whitelist specific user agents
    - Block specific user agents
    - Rate limit per user agent
    - Challenge based on behavior

3.5 Verification Mechanisms

For agents claiming to act on user behalf, emerging verification:

These standards are still emerging in 2026.

4. Content Patterns for Agent Consumption

4.1 What Agents Extract Differently

Agents process content differently than traditional crawlers:

For an agent helping a user "find the best web developer in Cassville, MO":

  1. Discover candidates
  2. Evaluate each (services, pricing, reviews, credentials)
  3. Compare options
  4. Present recommendation or take next action

The agent needs to extract specific data points from each candidate site to compare.

4.2 Structured Data Becomes Critical

For agent comparison workflows, schema is essential:

<!-- Service business with comprehensive schema for agent comparison -->
<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "ProfessionalService",
  "@id": "https://thatdeveloperguy.com/#organization",
  "name": "ThatDeveloperGuy",
  "description": "...",
  "founder": {"@id": "https://thatdeveloperguy.com/about/joseph-anady/#person"},
  "foundingDate": "2020",
  "serviceArea": [
    {"@type": "Country", "name": "United States"}
  ],
  "knowsAbout": [
    "Web Development",
    "Search Engine Optimization",
    "AI Search Optimization",
    "WordPress Development",
    "Custom Web Hosting"
  ],
  "hasCredential": [
    {
      "@type": "EducationalOccupationalCredential",
      "name": "Service-Disabled Veteran-Owned Small Business",
      "credentialCategory": "certification",
      "recognizedBy": {
        "@type": "GovernmentOrganization",
        "name": "U.S. Department of Veterans Affairs"
      }
    }
  ],
  "offers": [
    {
      "@type": "Offer",
      "name": "Custom Website Development",
      "description": "Full-service custom website builds",
      "priceSpecification": {
        "@type": "PriceSpecification",
        "minPrice": 597,
        "maxPrice": 2997,
        "priceCurrency": "USD"
      }
    },
    {
      "@type": "Offer",
      "name": "Engine Optimization",
      "description": "14-tier optimization across 112 individual optimizations",
      "priceSpecification": {
        "@type": "PriceSpecification",
        "minPrice": 1497,
        "maxPrice": 11997,
        "priceCurrency": "USD"
      }
    }
  ],
  "aggregateRating": {
    "@type": "AggregateRating",
    "ratingValue": "5.0",
    "reviewCount": "47"
  }
}
</script>

This level of structured data lets an agent compare ThatDeveloperGuy to alternatives systematically.

4.3 FAQ Patterns for Agent Q&A

When agents are answering user questions, FAQ pages with Question/Answer schema are particularly valuable:

<script type="application/ld+json">
{
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "What does ThatDeveloperGuy charge for a basic website?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Basic website builds start at $597 for foundation packages with about a 4-page scope."
      }
    },
    {
      "@type": "Question",
      "name": "Are you SDVOSB certified?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Yes, ThatDeveloperGuy is a Service-Disabled Veteran-Owned Small Business eligible for federal contracting set-aside programs."
      }
    },
    {
      "@type": "Question",
      "name": "What service area do you cover?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "We serve clients across the United States remotely from our Cassville, Missouri office."
      }
    }
  ]
}
</script>

4.4 Pricing Transparency

Agents trying to compare options need pricing visibility:

5. API Endpoints for Agents

For high-value workflows, providing API endpoints alongside human-readable pages:

5.1 Why APIs Help

api_advantages_for_agents:
  
  reliability:
    - Structured response formats
    - Consistent across requests
    - Less prone to layout changes breaking
  
  efficiency:
    - Less data transferred
    - Faster processing
    - Lower agent compute cost
  
  trust:
    - APIs are explicitly opt-in for programmatic access
    - Authentication mechanisms available
    - Rate limiting controllable

5.2 Common API Patterns

Service catalog API:

GET /api/services
{
  "services": [
    {
      "id": "web-dev",
      "name": "Custom Website Development",
      "description": "...",
      "starting_price": {"amount": 597, "currency": "USD"},
      "details_url": "https://thatdeveloperguy.com/services/web-development/"
    }
  ]
}

Quote request API:

POST /api/quote-request
{
  "name": "...",
  "email": "...",
  "service_interest": "web-dev",
  "project_description": "...",
  "budget_range": "..."
}

Availability API:

GET /api/availability
{
  "next_available_consultation": "2026-05-08T14:00:00-05:00",
  "booking_url": "https://thatdeveloperguy.com/book/"
}

5.3 API Documentation

If providing APIs, document them:

Make them discoverable from main site (link from footer, mention in robots.txt comments).

6. The llms.txt File for Agents

Building on framework-aicitations.md Section 5, llms.txt helps agents understand site structure:

# {{Business Name}}

> One-paragraph description of what this site is and offers.

## Services and Pricing
- [Web Development](https://thatdeveloperguy.com/services/web-development/): Starting at $597
- [Engine Optimization](https://thatdeveloperguy.com/services/seo/): Starting at $1,497
- [Hosting](https://thatdeveloperguy.com/services/hosting/): $25/month
- [Computer Repair](https://thatcomputerdude.com/): $75-300 typical

## About
- [Company Overview](https://thatdeveloperguy.com/about/): Founder bio, credentials, business overview
- [SDVOSB Certification](https://thatdeveloperguy.com/sdvosb/): Veteran-owned small business details

## Contact
- Phone: 505-512-3662
- Email: joseph.w.anady@icloud.com
- [Booking](https://thatdeveloperguy.com/book/): Schedule consultation

## Frameworks and Resources
- [14-Tier Framework](https://thatstupidcomputer.com/14-tier-framework/): Comprehensive optimization framework
- [Blog](https://thatdeveloperguy.com/blog/): Educational articles

## Optional API
- [Service Catalog API](https://thatdeveloperguy.com/api/services): Programmatic service listings

This file provides agents with structured navigation help.

7. Trust Signals for Agents

Agents acting on behalf of users need confidence in source quality. Trust signals:

7.1 Credentials Visibly Marked

trust_signal_visibility:
  
  business_credentials:
    - SDVOSB certification badge with link to verification
    - BBB rating with badge linking to BBB profile
    - Industry certifications with verification links
    - Awards with source attribution
  
  individual_credentials:
    - Education with school names
    - Professional certifications
    - Years of experience
    - Notable clients (with permission)
  
  schema_implementation:
    - Person schema with hasCredential
    - Organization schema with award
    - LocalBusiness with appropriate certifications

7.2 Transparency Signals

transparency_for_agents:
  
  pricing_visibility:
    - Clear pricing where possible
    - Service descriptions complete
    - Terms of service accessible
  
  process_visibility:
    - How you work explained
    - Typical timelines stated
    - What's included clarified
  
  contact_accessibility:
    - Multiple contact methods
    - Response time commitments
    - Service hours
  
  legitimacy_signals:
    - Real address (or service area for SAB)
    - Real phone number
    - Real team members with bios
    - Real portfolio/case studies

7.3 Anti-Patterns Agents Detect

Modern agents are increasingly sophisticated at detecting:

These hurt agent confidence in citing or recommending the source.

8. Measurement & Analytics

8.1 Tracking Agent Traffic

Distinguishing agent traffic from human traffic:

agent_traffic_identification:
  
  user_agent_strings:
    - "Claude-User/x.x"
    - "ChatGPT-User/x.x"
    - "PerplexityBot-User/x.x"
    - Custom agent identifiers
  
  ip_ranges:
    - Major platform IP ranges (publicly documented)
    - Reverse DNS lookups
  
  behavioral_signals:
    - Speed of interaction
    - Pattern of pages visited
    - Form submission patterns
    - Lack of typical human dwell time

8.2 GA4 Custom Dimensions

Set up custom dimensions for agent identification:

// In GA4 / GTM
const isAgent = detectAgentTraffic(navigator.userAgent);
gtag('config', 'G-XXXXX', {
  user_properties: {
    traffic_type: isAgent ? 'agent' : 'human'
  }
});

This lets you analyze:

8.3 Server Logs

Server logs catch agent activity GA4 can't:

# Identify agent traffic from logs
grep -E "(Claude|ChatGPT|Perplexity|Anthropic|GPT)" /var/log/nginx/access.log | wc -l

# Top pages accessed by agents
grep -E "(Claude|ChatGPT|Perplexity)" /var/log/nginx/access.log | \
  awk '{print $7}' | sort | uniq -c | sort -rn | head -20

9. Audit Mode

# Criterion Pass/Fail
AG1 Site renders content server-side or via fast SSR
AG2 All forms have proper labels and ARIA
AG3 Multi-step workflows preserve state
AG4 Robots.txt explicitly addresses agent user-agents
AG5 Schema markup comprehensive across content types
AG6 Pricing visibility on relevant pages
AG7 FAQ schema for common questions
AG8 llms.txt file present
AG9 API endpoints provided for high-value workflows (optional)
AG10 Trust signals visible and structurable
AG11 Credentials in Person/Organization schema
AG12 Site tested with actual agent (Claude computer use, etc.)
AG13 Anti-bot configured to allow legitimate agents
AG14 Agent traffic monitoring in analytics
AG15 Form submissions completable by agents end-to-end
AG16 Confirmation states clear after agent actions

Score: 16. World-class agent optimization: 14+/16.

10. Common Mistakes

  1. Heavy JavaScript without SSR — agents struggle with client-rendered content
  2. Forms without labels — agents can't identify fields reliably
  3. Aggressive anti-bot — blocks legitimate agents along with malicious ones
  4. Pricing hidden — agents can't compare without pricing visibility
  5. No schema for service offerings — agents can't extract structured info
  6. No FAQ markup — common questions not extractable
  7. No llms.txt — missed agent navigation help
  8. Confirmation pages with no clear success state — agents can't verify success
  9. Multi-step workflows that lose state — agent can't recover from errors
  10. Ignoring agent traffic in analytics — missing emerging traffic source

11. The Frontier (Things Likely to Change)

This is genuinely a 2026 frontier. Things likely to evolve:

Monitor for emerging standards. Update implementation as patterns stabilize.

End of Framework Document

Document version: 1.0 Last updated: 2026-04-29

The agentic AI layer is genuinely emerging. Optimization patterns will mature over coming years. The principles in this framework — clean architecture, accessible interactions, trust signals, structured data — are durable. Specific tactics will evolve.

Companion documents:

Want this framework implemented on your site?

ThatDevPro ships these frameworks as productized services. SDVOSB-certified veteran owned. Cassville, Missouri.

See Engine Optimization service ›