Announcement

#1 2014-06-12 07:37:41

matthys
Member
2014-04-02
188

Advise how to populate robots.txt

Hello everyone,

Not sure where to put this request, so hope this is OK.

I was wondering if there is any advise on how to populate the robots.txt?
There are some items on the forum about it, but mostly about preventing spidering.

My question (I think) is simple, what folders should be disallowed?

I found this at http://piwigo.org/robots.txt, would this be a good example?

User-agent: *
Crawl-delay: 10
# Directories
Disallow: /includes/
Disallow: /misc/
Disallow: /modules/
Disallow: /profiles/
Disallow: /scripts/
Disallow: /themes/
# Files
Disallow: /CHANGELOG.txt
Disallow: /cron.php
Disallow: /INSTALL.mysql.txt
Disallow: /INSTALL.pgsql.txt
Disallow: /install.php
Disallow: /INSTALL.txt
Disallow: /LICENSE.txt
Disallow: /MAINTAINERS.txt
Disallow: /update.php
Disallow: /UPGRADE.txt
Disallow: /xmlrpc.php
# Paths (clean URLs)
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /logout/
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/login/
# Paths (no clean URLs)
Disallow: /?q=admin/
Disallow: /?q=comment/reply/
Disallow: /?q=logout/
Disallow: /?q=node/add/
Disallow: /?q=search/
Disallow: /?q=user/password/
Disallow: /?q=user/register/
Disallow: /?q=user/login/

PS .. I also use these options, so my url's are a bit different;
$conf['question_mark_in_urls'] = false;
$conf['php_extension_in_urls'] = false;
$conf['category_url_style'] = 'id-name';
$conf['picture_url_style'] = 'id-file';


Thanks,
Matthijs

Piwigo version: 2.6.3
PHP: 5.5.9-1ubuntu4 (Show info) [2014-06-12 07:35:14]
MySQL: 5.5.37-0ubuntu0.14.04.1-log [2014-06-12 07:35:14]

Offline

 

#2 2014-06-12 08:07:31

flop25
Piwigo Team
2006-07-06
7037

Re: Advise how to populate robots.txt

Hello
if you read carefully you will understand that's for Piwigo.org not for Piwigo
Then I would suggest to disallow all paths others than upload and galleries


To get a better help : Politeness like Hello-A link-Your past actions precisely described
Check my extensions : more than 30 available
who I am and what I do : http://fr.gravatar.com/flop25
My gallery : an illustration of how to integrate Piwigo in your website

Offline

 

#3 2014-06-12 08:41:29

matthys
Member
2014-04-02
188

Re: Advise how to populate robots.txt

Thanks .. I had to start somewhere ...

What about the files? You will need index.php... but NOT install.php, password.php profile.php etc?

Offline

 

Board footer

Powered by FluxBB

github twitter newsletter Donate Piwigo.org © 2002-2024 · Contact