larder
/
larder.web
1
0
Fork 0
Code Issues Releases Wiki Activity
Browse Source

update robots.txt file.

unstable
Craig Oates 9 months ago
parent
f386dd88bd
commit
b2a2513845
1 changed files with 0 additions and 18 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Split View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 18
      static/robots.txt

18
static/robots.txt
Unescape View File

@ -30,21 +30,3 @@ Disallow: /
User-agent: SemrushBot-CT
Disallow: /
User-agent: SplitSignalBot
Disallow: /
User-agent: SemrushBot-COUB
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-agent: Sogou inst spider
Disallow: /

Write Preview
Loading…
Cancel
Save