Details
Overview
SEO Technical: robots.txt Skill is an open-source Agent Skill from the kostja94/marketing-skills collection (MIT). It teaches AI coding agents focused marketing / SEO workflows so they can produce production-ready guidance instead of generic advice.
Guides configuration and auditing of robots.txt for search engine and AI crawler control.
Identify: 1. Site URL: Base domain (e.g., https://example.com) 2. Indexing scope: Full site, partial, or specific paths to exclude 3. AI crawler strategy: Allow search/indexing vs. block training data crawlers
Official skill id: robots-txt. Source path: skills/seo/technical/robots.
Key Features
- R: obots.txt: Configure Disallow/Allow, Sitemap, Clean-param; audit for accidental blocks
- C: rawler access: Path-level crawl control; AI crawler allow/block strategy
- D: ifferentiation: robots.txt = crawl control (who accesses what paths); noindex = index control (what gets indexed). See indexing for page-level exclusions.
- C: urrent state (if auditing)
- R: ecommended robots.txt (full file)
- C: ompliance checklist
How to Use
- Use a Skill-compatible agent such as Claude Code, Cursor, or Codex.
- Install from the terminal (Node.js required):
npx skills add https://github.com/kostja94/marketing-skills --skill robots
Or paste the install prompt from this tool page into your agent and ask it to install for you. 3. Restart / reload the agent so the new Skill is loaded. 4. Invoke the Skill with its slash commands or by asking the agent to run the workflow described in Overview / Key Features. 5. Review outputs before publishing — treat model-generated copy, images, and data as drafts.
Pricing
| Plan | Billing | Price | Notes |
|---|---|---|---|
| Open source | — | Free | MIT license; install via npx skills add |
| Full pack | — | Free | Optional: install the entire marketing-skills repo |
No paid tiers. Check the GitHub repository for license and updates.
Best For
Marketers, indie hackers, ecommerce / SaaS founders, and SEO operators who want agents to execute a specific marketing task with structured best practices.
Trigger context from the skill author: When the user wants to configure, audit, or optimize robots.txt. Also use when the user mentions "robots.txt," "crawler rules," "block crawlers," "AI crawlers," "GPTBot," "allow/disallow," "disallow path," "crawl directives," "user-agent," "block Googlebot," "fix robots.txt," "robots.txt blocking," or "search engine…
FAQ
How do I install this skill?
npx skills add https://github.com/kostja94/marketing-skills --skill robots-txt
Or copy the skill folder into .cursor/skills/, .agents/skills/, or .claude/skills/.
Does this skill run code on my machine?
No. Skills in this collection are pure Markdown instructions for agents — no executable payloads.
Can I install only this skill instead of the full pack?
Yes. Prefer --skill robots-txt for a single task. Install 2–3 related skills per job rather than the entire library.
Where should I add project context?
Copy templates/project-context.md from the repo into .cursor/ or .claude/ and fill in product, audience, and brand details so outputs stay tailored.