Browse Source

Added Some constraints to robots file

Yoginth 7 years ago
parent
commit
7f6dbf3926
1 changed files with 6 additions and 1 deletions
  1. 6 1
      public/robots.txt

+ 6 - 1
public/robots.txt

@@ -8,14 +8,19 @@ Disallow: /admin
 Disallow: /*/*/forks
 Disallow: /*/*/forks
 Disallow: /*/*/stars
 Disallow: /*/*/stars
 Disallow: /*/*/pulls
 Disallow: /*/*/pulls
-Disallow: /*/*/archive/
+Disallow: /*/*/archive
+Disallow: /*/*/settings
 Disallow: /*/*/raw/*
 Disallow: /*/*/raw/*
+Disallow: /*/*/wiki/_new
+Disallow: /*/*/wiki/*/_edit
 Disallow: /*/*/commits
 Disallow: /*/*/commits
 Disallow: /*/followers
 Disallow: /*/followers
 Disallow: /*/following
 Disallow: /*/following
 Disallow: /*/*/src/*
 Disallow: /*/*/src/*
+Disallow: /*/*/_edit
 Disallow: /*/*/issues/new
 Disallow: /*/*/issues/new
 Disallow: /*/*/issues?
 Disallow: /*/*/issues?
+Disallow: /*/*/compare
 Disallow: /*?
 Disallow: /*?
 
 
 Sitemap: https://gitote.in/sitemap.xml
 Sitemap: https://gitote.in/sitemap.xml