BuildQuill
Lessons 4 / 6Technical SEO Basics8 min
Lesson 4 of 68 minTechnical SEO Basics

Robots.txt and Sitemaps

Robots.txt gives crawler instructions. Sitemaps help crawlers discover important URLs.

Core concepts

  • Robots.txt can block crawling.
  • Sitemaps list URLs.
  • Blocking crawling is not the same as noindex.
  • Keep sitemap URLs canonical.

Example

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Practice

Draft a robots.txt file that allows public pages and points to your sitemap.

Robots.txt GeneratorSitemap XML Generator

Checklist

  • Do not block important pages.
  • Include sitemap URL.
  • Use canonical URLs.
  • Test before publishing.

Quiz

What is the main job of a sitemap?

Playground

SERP preview

https://buildquill.com/tools/json-formatter

Free JSON Formatter and Validator

Format, validate, and debug JSON in your browser with a fast private tool from BuildQuill.