From 50a76f56ba87c778be08ff23f6bd72b0677e8069 Mon Sep 17 00:00:00 2001
From: Craig Oates <craig@craigoates.net>
Date: Sat, 22 Oct 2022 01:42:51 +0100
Subject: [PATCH] add robot.txt file to the /static directory.

I'm sure I will need to update this at some point. I didn't spend too much time
on it, just jotted things down as a I came across routes in web.lisp. The
Sitemap URL will need to change, the closer the site goes into production and an
domain name has been bought/finalised.
---
 static/robot.txt | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
 create mode 100644 static/robot.txt

diff --git a/static/robot.txt b/static/robot.txt
new file mode 100644
index 0000000..e4ea963
--- /dev/null
+++ b/static/robot.txt
@@ -0,0 +1,28 @@
+# Nicola Ellis and Ritherdon Archive
+User-agent: *
+Sitemap: https://www.nicolaellisandritherdonarchive.com
+Disallow: /setup
+Disallow: /run-setup
+Disallow: /sign-up
+Disallow: /login
+Disallow: /logout
+Disallow: /dashboard
+Disallow: /site-settings
+Disallow: /site-settings/
+Disallow: /users
+Disallow: /user/
+Disallow: /create/
+Disallow: /user/
+Disallow: /edit/
+Disallow: /rename/
+Disallow: /page/
+Disallow: /storage/manage
+Disallow: /storage/rename/
+Disallow: /storage/delete/
+Disallow: /create/
+Disallow: /archive/
+Disallow: /search
+Allow: /archive
+Allow: /view/archive/
+Allow: /pages
+Allow: /view/page/