make robots.txt restrictive by default - #3905

https://chat.openai.com/share/aa80093e-3f71-43af-9f48-87d7c622311d

I wonder what the solution should be for the Cloud version. Is omitting robots.txt OK or should we COPY different files when building the container image?

Cloud (to keep the previous behavior)

# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-agent: *
# Disallow: /

Self-hosted

User-agent: *
Disallow: /

The current robots.txt isn't event used in prod because plausible/website serves its own one which the reverse proxy prefers: https://plausible.io/robots.txt

It should be fine to omit. But we should add a comment/doc line for some clarity. We never want CE to be indexed but in Cloud we do want pages like public dashboards and live demo to be indexed. This is controlled via X-Robots-Tag header rather than robots.txt file.

Then instead of omitting, would it be OK to simply use

User-agent: *
Disallow: /

for both? With a comment, of course.

Like this c71c114

I prefer the original solution of omitting it. It's quite likely we will move off the Jekyll-based static site in the future. And in that case, if we forget about robots.txt, we might end up in a place where we disallow all indexing for plausible.io which would be bad.

I know omitting adds more code, but it feels safer to me.

What about copying different robots.txt into the container image based on some ARG? So that EE would get

# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-agent: *
# Disallow: /

and CE would get

User-agent: *
Disallow: /

Or having it configured with app env and served by a custom plug.

# in endpoint.ex
plug :serve_robots_txt

on_full_build do
# ...
else
  def robots_txt do
    # or app env lookup
    """
    User-agent: *
    Disallow: /
    """
  end
end

def serve_robots_txt(conn, _opts) do
  case conn.path_info do
    ["robotx.txt"] -> conn |> send_resp(200, robots_txt()) |> halt()
    _other -> conn
  end
end

I think omitting, the way you've done, is good enough for now.

Ok! I've added a note about the Cloud's robots.txt in 11409f3