Announcement

  •  » Requests
  •  » add robots.txt to restrict admin pages

#1 2017-09-26 02:14:08

ceinmart
Member
2017-08-17
20

add robots.txt to restrict admin pages

Hi,

As far I know , all search spiders will scan all pages if doesn't have a robots.txt disallowing certain pages.
I just add my site at google/webmasters and notice the piwigo doesn't have a robots.txt .

If I'm not misunderstand, is very important have one to avoid spiders on pages like /admin.php?

Offline

 

#2 2017-09-26 11:32:49

eliz82
Member
Romania
2016-04-27
281

Re: add robots.txt to restrict admin pages

ceinmart wrote:

Hi,
As far I know , all search spiders will scan all pages if doesn't have a robots.txt disallowing certain pages.
I just add my site at google/webmasters and notice the piwigo doesn't have a robots.txt
If I'm not misunderstand, is very important have one to avoid spiders on pages like /admin.php?

That is something that you would easily do it yourself.

Bad crawlers do not care about what you write in the robots.txt , even more if you write where not to search you give them directions where to search. They can mask behind a good crawler user-agent, so the only way to block them is to watch their activities and block them by IP. Also good crawlers like google or bing do not take some rules like crawl delay into consideration.

If you have some sensitive directories of files better use server rules like htaccess to block them (like directory indexing disabled, or enabled some files access based on ip), or put index.php in every directory (piwigo already has this).

Last edited by eliz82 (2017-09-26 11:35:34)

Offline

 
  •  » Requests
  •  » add robots.txt to restrict admin pages

Board footer

Powered by FluxBB

github twitter newsletter Donate Piwigo.org © 2002-2024 · Contact