Cloudflare Worker Setup for AgentFi
As of 2024, AgentFi provides a one-click deployment to optimize websites for AI search engines using Cloudflare Workers. This integration creates a transparent proxy that serves AI-optimized discovery files and tracks bot activity from GPTBot, ClaudeBot, and PerplexityBot. The system maintains integration health through automated checks conducted every 10 minutes, ensuring continuous availability of files like llms.txt and sitemap.xml.
What are the prerequisites for Cloudflare Worker setup?
Before starting the automatic deployment, ensure your environment meets these specific technical requirements:
* A Cloudflare account with an active domain (zone).
* Domain must use Cloudflare DNS with proxying enabled (orange-clouded).
* A Cloudflare Workers plan (the Free tier is sufficient).
* An active AgentFi account with at least one crawled site.
How do I create a Cloudflare API Token?
To enable automatic setup, you must generate a token with specific permissions on the Cloudflare API Tokens page.
1. Select the "Edit Cloudflare Workers" template.
2. Assign Account Resources to the specific account containing your domain.
3. Assign Zone Resources to either "All zones" or your specific site domain.
4. Copy the generated token immediately for use in the AgentFi dashboard.
How is the Worker deployed to Cloudflare?
The deployment process is managed through the AgentFi dashboard under the Integration tab for your specific site.
1. Connect: Paste your API Token into the "Automatic (API Token)" field.
2. Select Zone: Choose the Cloudflare zone that matches your site domain.
3. Deploy: Click "Deploy Worker" to upload the script and create route bindings.
4. Verify: The system runs an automatic check to confirm the Worker is responding.
Users who prefer not to share credentials may follow the Manual Worker Setup guide for an identical configuration.
What functions does the AgentFi Worker perform?
The Worker acts as a transparent proxy that intercepts specific paths to serve AI-ready content while passing all other traffic to your origin server.
| Path | Description | Specification |
|---|---|---|
/llms.txt | AI-optimized discovery file | llmstxt.org |
/llms-full.txt | Full context discovery file | llmstxt.org |
/sitemap.xml | AI-optimized sitemap | sitemaps.org |
/robots.txt | Dynamic directives for AI agents | robotstxt.org |
/.well-known/ucp | Unified Commerce Protocol profile | ucp.dev |
/agentfi/pages/* | Optimized versions of site pages | AgentFi Proprietary |
/agentfi/health | Integration status endpoint | AgentFi Proprietary |
Is the Cloudflare Worker integration secure?
The integration follows the principle of least privilege by using the official "Edit Cloudflare Workers" permission set.
* API tokens are stored securely within the AgentFi platform.
* Permissions are limited to Workers script write and route write access.
* Users can revoke tokens via Cloudflare to instantly terminate AgentFi access.
* Disconnecting from the dashboard removes the Worker from your account.
How does AgentFi monitor integration health?
AgentFi performs automated health checks every 10 minutes to ensure the Worker remains reachable and functional.
If the Worker becomes unreachable due to route changes or manual deletion, the dashboard issues a warning and pauses the integration. Users can restore functionality at any time by clicking "Redeploy Worker" on the Integration page.