Domovská stránka Prehľadávať Pomoc
Prihlásiť sa
yo
/
gitote
zrkadlo https://gitlab.com/gitote/gitote
Pridať medzi pozorované 1
Hviezda 1
Fork 0
Súbory Issues 0 Wiki
Strom: 3db5a7027b
Branche Tagy
gitote
/
public
/
robots.txt

robots.txt 1.6 KB
História Raw

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394 
  1. # If you would like to crawl Gitote contact us at support@gitote.in.
    2. 

  2. 

  3. User-agent: *
    4. 

  4. Allow: /humans.txt
    5. 

  5. Disallow: /
    6. 

  6. 

  7. User-agent: Googlebot
    8. 

  8. Allow: /
    9. 

  9. Disallow: /admin/*
    10. 

  10. Disallow: /*/*/forks
    11. 

  11. Disallow: /*/*/stars
    12. 

  12. Disallow: /*/*/archive/
    13. 

  13. Disallow: /*/*/raw/*
    14. 

  14. 

  15. User-agent: duckduckbot
    16. 

  16. Allow: /
    17. 

  17. Disallow: /admin/*
    18. 

  18. Disallow: /*/*/forks
    19. 

  19. Disallow: /*/*/stars
    20. 

  20. Disallow: /*/*/archive/
    21. 

  21. Disallow: /*/*/raw/*
    22. 

  22. 

  23. User-agent: HTTrack
    24. 

  24. Allow: /
    25. 

  25. Disallow: /admin/*
    26. 

  26. Disallow: /*/*/forks
    27. 

  27. Disallow: /*/*/stars
    28. 

  28. Disallow: /*/*/archive/
    29. 

  29. Disallow: /*/*/raw/*
    30. 

  30. 

  31. User-agent: ia_archiver
    32. 

  32. Allow: /
    33. 

  33. Disallow: /admin/*
    34. 

  34. Disallow: /*/*/forks
    35. 

  35. Disallow: /*/*/stars
    36. 

  36. Disallow: /*/*/archive/
    37. 

  37. Disallow: /*/*/raw/*
    38. 

  38. 

  39. User-agent: msnbot
    40. 

  40. Allow: /
    41. 

  41. Disallow: /admin/*
    42. 

  42. Disallow: /*/*/forks
    43. 

  43. Disallow: /*/*/stars
    44. 

  44. Disallow: /*/*/archive/
    45. 

  45. Disallow: /*/*/raw/*
    46. 

  46. 

  47. User-agent: Bingbot
    48. 

  48. Allow: /
    49. 

  49. Disallow: /admin/*
    50. 

  50. Disallow: /*/*/forks
    51. 

  51. Disallow: /*/*/stars
    52. 

  52. Disallow: /*/*/archive/
    53. 

  53. Disallow: /*/*/raw/*
    54. 

  54. 

  55. User-agent: naverbot
    56. 

  56. Allow: /
    57. 

  57. Disallow: /admin/*
    58. 

  58. Disallow: /*/*/forks
    59. 

  59. Disallow: /*/*/stars
    60. 

  60. Disallow: /*/*/archive/
    61. 

  61. Disallow: /*/*/raw/*
    62. 

  62. 

  63. User-agent: Slurp
    64. 

  64. Allow: /
    65. 

  65. Disallow: /admin/*
    66. 

  66. Disallow: /*/*/forks
    67. 

  67. Disallow: /*/*/stars
    68. 

  68. Disallow: /*/*/archive/
    69. 

  69. Disallow: /*/*/raw/*
    70. 

  70. 

  71. User-agent: Swiftbot
    72. 

  72. Allow: /
    73. 

  73. Disallow: /admin/*
    74. 

  74. Disallow: /*/*/forks
    75. 

  75. Disallow: /*/*/stars
    76. 

  76. Disallow: /*/*/archive/
    77. 

  77. Disallow: /*/*/raw/*
    78. 

  78. 

  79. User-agent: Twitterbot
    80. 

  80. Allow: /
    81. 

  81. Disallow: /admin/*
    82. 

  82. Disallow: /*/*/forks
    83. 

  83. Disallow: /*/*/stars
    84. 

  84. Disallow: /*/*/archive/
    85. 

  85. Disallow: /*/*/raw/*
    86. 

  86. 

  87. User-agent: Yandex
    88. 

  88. Allow: /
    89. 

  89. Disallow: /admin/*
    90. 

  90. Disallow: /*/*/forks
    91. 

  91. Disallow: /*/*/stars
    92. 

  92. Disallow: /*/*/archive/
    93. 

  93. Disallow: /*/*/raw/*
    94. 

  94.