Guide

1. Add the gem. It needs Ruby 3.3 and Rails 8.0, or newer; its only runtime dependency is Active Support.

   # Gemfile
   gem "maptain"

2. Name the host of the site:

   # config/initializers/maptain.rb
   Maptain.configure do |config|
     config.host = "https://www.example.com"
     config.locales = %i[en fr de]   # default: I18n.available_locales
     config.x_default = :en          # the language of hreflang="x-default"; none when left out
   end

3. Write one class per kind of page in app/sitemaps, as in Sources below:

   # app/sitemaps/product_sitemap.rb
   class ProductSitemap < Maptain::Source
     scope { Product.published }
     url { |product, locale| product_url(product, locale:) }
   end

4. Look at what it would write, without writing it. It fails on a URL outside the site, a URL listed twice, a date in the future, and a few other mistakes:

   bin/rails maptain:check

5. Build, then tell crawlers where the index is, in public/robots.txt:

   bin/rails maptain:build

   Sitemap: https://www.example.com/sitemap.xml

6. Run it on a schedule, every hour or every night. With Solid Queue:

   # config/recurring.yml
   production:
     sitemap:
       command: "Maptain.build"
       schedule: every hour

That is all most sites need. The sections below go further: records in some languages only, dates from several columns, files on S3, IndexNow.

A source is a class in app/sitemaps that inherits from Maptain::Source, one per kind of page. Its blocks run in an object that has your route helpers, already set to config.host, or in the url block to the host of the language being written. That block also runs with its language as I18n.locale, so a slug read from a translated attribute comes out in the language of its page; a language that is not in I18n.available_locales runs in the app's.

# app/sitemaps/product_sitemap.rb
class ProductSitemap < Maptain::Source
  # The records: an Active Record relation. Add the includes your other blocks need.
  scope { Product.published.includes(:photos) }

  # The URL of a record in a language. A path gets the host of its language; nil leaves that language out.
  url { |product, locale| product_url(product, locale:) }

  # Image URLs, for Google Images.
  images { |product| product.photos.map(&:url) }
end

The model needs an integer primary key: files are cut by ranges of ids.

Pages that are not records

The scope can be any list. Such a source is one file, rendered at each build and published when it differs from the last one.

class PageSitemap < Maptain::Source
  scope { %i[root about contact] }
  url { |page, locale| public_send(:"#{page}_url", locale:) }
end

Every record gets one URL per language of config.locales. When records are not all translated, locales names the languages each one exists in:

class ProductSitemap < Maptain::Source
  scope { Product.published }
  url { |product, locale| product_url(product, locale:) }
  locales { |product| product.translated_locales }
end

Each <url> then lists every language of its page, itself included, as hreflang links. A page in one language gets no link. A page that does not exist in the x_default language gets no x-default link. A Rails locale such as pt_BR is written hreflang="pt-BR", and two languages that share a URL list it once.

A URL your block writes by hand is percent-encoded: spaces, accents and control characters cannot break a file.

One host per language

When some languages are served from a host of their own, name those hosts:

config.host = "https://www.example.com"
config.host_by_locale = {fr: "https://fr.example.com", de: "https://de.example.com"}

The other languages stay on config.host, and so does the index. The robots.txt of every host names that one index: a sitemap may list the pages of a host only when that host's robots.txt points to it. Each key is one of config.locales: a build refuses any other.

lastmod is read from updated_at. Name another column when the content has its own:

lastmod :content_updated_at

A column that moves when the content does not, such as an updated_at touched by a view counter, makes every build rewrite files for nothing and tells search engines a date that is not true. Give such a model a column for content changes.

When the date depends on more than one column

lastmod also takes a block. The build then needs to know what to watch in the database, as an SQL expression:

class ProductSitemap < Maptain::Source
  scope { Product.published }
  url { |product, locale| product_url(product, locale:) }
  lastmod { |product| [product.updated_at, product.reviewed_at].compact.max }
  fingerprint "GREATEST(products.updated_at, COALESCE(products.reviewed_at, products.updated_at))"
end

A source named ProductSitemap writes sitemap-product-00000.xml.gz, sitemap-product-00001.xml.gz and so on, one per range of ids. slug "products" changes the middle part, config.filename the first.

The size of a range comes from the number of languages, so that a file stays under the 50,000 URLs and 50 MiB the protocol allows: 50,000 ids with one language, 554 with 24. shard_size 1_000 sets it yourself. A range that still outgrows a file continues in sitemap-product-00000-2.xml.gz.

sitemap.xml is the index. It lists every file with the latest lastmod it holds, so a crawler fetches again only the files that changed.

bin/rails maptain:build     # rewrite the files whose records changed
bin/rails maptain:rebuild   # rewrite everything

Maptain.build does the same from Ruby and returns what it did: full, written, kept, deleted, url_count, submitted and warnings. Run it on a schedule, every night or every hour. With Solid Queue:

# config/recurring.yml
production:
  sitemap:
    command: "Maptain.build"
    schedule: every hour

How a build decides

For each source, one query groups the records by range of ids and returns the count, the sum of the ids and the latest change of each. A range whose answer is the one the last build stored keeps its file. The others are rendered again, and a file that comes out the same is not uploaded (a full build uploads every file). Nothing hooks into your models, so update_all, imports and deletions are seen too.

In the index, a file that changed takes the latest lastmod it holds, or the time of the build when a record left it, so that crawlers read it again.

Some changes leave those numbers alone: update_columns, a change in another table that the url, locales or images blocks read, a new url block, a file someone removed from the storage. Name the other table in fingerprint, or touch the record from it. For the rest, a full build comes once a week by itself:

config.rebuild_every = 1.day   # default: a week; nil leaves it to maptain:rebuild

After deploying a change to a url block, run maptain:rebuild rather than wait.

A build notes the files it is about to add in sitemap-manifest.json, its memory of what each file held, then writes the sitemap files, then the index, then removes the files no range fills any more, then writes the manifest again. A build that fails midway is repaired by the next one. Changing a host, the languages, the size of a range or the version of Maptain rewrites everything it moves once.

Files go to public/ by default.

Through Active Storage

An app that already uses Active Storage can send the sitemaps through one of its services, with the credentials it already has. Give them a service of their own in config/storage.yml, in the bucket you want:

sitemaps:
  service: S3
  bucket: example-sitemaps
  region: eu-west-3
  public: true
  access_key_id: <%= Rails.application.credentials.dig(:aws, :access_key_id) %>
  secret_access_key: <%= Rails.application.credentials.dig(:aws, :secret_access_key) %>

Maptain.configure do |config|
  config.storage = Maptain::Storage::ActiveStorage.new(:sitemaps, prefix: "sitemaps")
  config.sitemaps_url = "https://cdn.example.com/sitemaps"   # where those files are served from
end

No blob is created: the files keep their names as keys, and nothing is written to the database. Any service works that serves a key at a stable URL: S3, Google Cloud Storage, Azure, a mirror of them. The Disk service does not, it scatters keys in subfolders: keep the default for files on disk.

Sitemaps served from another host than the pages they list are accepted when the site's robots.txt names the index there.

Your own

A storage is any object with read(name), write(name, path) and delete(name).

IndexNow tells Bing, Yandex, Seznam and Naver that a URL changed. Google does not take part. Choose a key of 8 to 128 characters among a-z, A-Z, 0-9 and -:

config.index_now_key = "c1f0e2b4a6d84f3e9b7a5c3d1e0f2a4b"

Each build then publishes the key as <key>.txt at the root of the site, and sends, from the files it rewrote, the URLs whose lastmod is later than the previous build, and those that appeared or went away whatever their lastmod. The first build sends nothing, nor does a new configuration beyond the pages whose lastmod moved, and only URLs on the hosts of the site are sent.

When the sitemaps are served from elsewhere (config.sitemaps_url), the key cannot follow them: IndexNow reads it on the site. Put <key>.txt in public/ and set config.index_now_key_location to its path, "/<key>.txt"; a build warns until you do.

With one host per language, every host must serve the key: IndexNow reads it on the host of the URLs it is given.

A refusal or an endpoint that cannot be reached is a warning, never a failed build: the URLs it did not take are kept in the manifest and sent again with the next build. Once the endpoint cannot be reached, a build asks it nothing more.

bin/rails maptain:check

It renders every sitemap without publishing or submitting anything, and fails on:

• a URL outside the hosts of the site, listed twice in a source, or longer than 2,048 characters, and a page whose languages all share one URL;
• an hreflang that is not a language code (en, pt-BR, zh-Hant, es-419);
• a lastmod in the future, or one that is not a W3C date, which is then left out;
• an index with more than the 50,000 files it may hold.

When many URLs of a source are outside the hosts of the site, listed twice, too long or dated in the future, one line names the first and counts the others. A build reports the same in warnings, for the files it rewrites.

Maptain never requests a page of the site. Whether a URL answers 200 or carries a noindex is what Search Console reports, from the visits crawlers make anyway.

build.maptain wraps a build, with full and result in its payload. shard.maptain wraps each range it renders, with source and shard.

ActiveSupport::Notifications.subscribe("build.maptain") do |event|
  built = event.payload[:result]
  Rails.logger.info "Sitemap: #{built.written.size} files written in #{event.duration.round} ms"
end

sitemap_generator	Maptain
config/sitemap.rb, one add per URL	One Maptain::Source per kind of page, in app/sitemaps
alternates: written by hand for each URL	hreflang links written from config.locales, or a source's locales
default_host	config.host, and config.host_by_locale for one host per language
rake sitemap:refresh	bin/rails maptain:build, which rewrites only what changed
lastmod:	updated_at by default, or lastmod :column
priority:, changefreq:	Left out: Google ignores them
Pinging search engines	IndexNow, with config.index_now_key
sitemaps_host, adapters	config.sitemaps_url, and any Active Storage service

sitemap_generator names its index sitemap.xml.gz, Maptain sitemap.xml: change the Sitemap: line of robots.txt, and delete the old files once the first build has written the new ones.