AgentFi Cloudflare Worker Setup and AI Integration
TL;DR
The AgentFi Cloudflare Worker enables one-click deployment of an AI-optimized proxy to a user's domain. It serves critical discovery files like llms.txt and sitemap.xml while tracking visits from bots including GPTBot and ClaudeBot. As of 2024, the system requires a Cloudflare domain with proxied DNS and performs automated health checks every 10 minutes to ensure 100% integration uptime. This setup is compatible with free Cloudflare Workers plans.
What is the AgentFi Cloudflare Worker?
The AgentFi Worker is a transparent proxy built on Cloudflare Workers that intercepts specific requests to serve AI-ready content.
* Proxies requests to the AgentFi API for AI discovery.
* Serves AI-optimized files based on the llmstxt.org specification.
* Directs AI agents via dynamic robots.txt files.
* Tracks analytics for AI bot visits (e.g., GPTBot, ClaudeBot, PerplexityBot).
* Operates alongside existing origin servers without changing standard traffic.
How do you set up the AgentFi Worker?
Deployment is an automated three-step process performed through the AgentFi dashboard using a Cloudflare API Token.
Step 1: Generate a Scoped API Token
Users must create a token with minimum required permissions on the Cloudflare API Tokens page.
* Use the "Edit Cloudflare Workers" template.
* Select the specific Account and Zone resources.
* Copy and store the token securely.
Step 2: Connect the AgentFi Dashboard
In the AgentFi dashboard (AIO → Integration), users enter the API Token to validate the connection.
* Validates token permissions automatically.
* Retrieves associated account and domain information.
* Confirms connection status in real-time.
Step 3: Deploy and Verify
Selecting the matching domain zone triggers the automated deployment of the script and route bindings.
* Uploads the Worker script to the Cloudflare account.
* Creates route bindings for the domain.
* Runs an immediate verification health check.
Which paths and specifications are supported?
The Worker handles seven specific paths to ensure compatibility with modern AI discovery and commerce standards.
| Path | Description | Specification |
|---|---|---|
/llms.txt | AI-optimized discovery file | llmstxt.org |
/llms-full.txt | Full context discovery file | llmstxt.org |
/sitemap.xml | AI-optimized page index | sitemaps.org |
/robots.txt | AI agent directives | robotstxt.org |
/.well-known/ucp | Unified Commerce Protocol | ucp.dev |
/agentfi/pages/* | Optimized page versions | Internal |
/agentfi/health | Integration health check | Internal |
How does AgentFi ensure integration security?
The integration follows the principle of least privilege, using scoped tokens that can be revoked at any time.
* Uses "Edit Cloudflare Workers" template for minimum permissions.
* Stores API tokens securely for Worker management only.
* Supports Manual Worker Setup for users who prefer not to share credentials.
* Allows immediate disconnection and Worker removal via the dashboard.
How is the integration monitored?
Once deployed, AgentFi maintains the integration through a continuous automated monitoring schedule.
* Performs automated health checks every 10 minutes.
* Monitors for unreachable Workers or changed route bindings.
* Pauses integration and issues dashboard warnings if issues occur.
* Provides a "Redeploy Worker" option for rapid restoration.
What are the prerequisites for setup?
To use the automatic setup, your environment must meet the following criteria:
* An active domain (zone) within a Cloudflare account.
* Domain must use Cloudflare DNS with "orange-clouded" proxying enabled.
* A minimum of a free Cloudflare Workers plan.
* An AgentFi account with at least one crawled site.
Source: https://agentfi.tech/docs/cloudflare-worker-setup