From: Zachary Vance <za3k@za3k.com>
Date: Thu, 23 Apr 2020 09:24:09 +0000 (-0700)
Subject: Simplify robots.txt
X-Git-Url: https://git.za3k.com/?a=commitdiff_plain;h=4851ef772cc077c852468c8e8285a981f21ced6a;p=za3k.git

Simplify robots.txt
---

diff --git a/robots.txt b/robots.txt
index 62a5e12..06e4ae9 100644
--- a/robots.txt
+++ b/robots.txt
@@ -1,5 +1,6 @@
+# These bots are too stupid and too frequent. They download very large data files.
 User-agent: AhrefsBot
-Disallow: /
-
 User-agent: dotbot
 Disallow: /
+
+# If you are a human, feel free to ignore this robots.txt file.