Hello everyone,
Not sure where to put this request, so hope this is OK.
I was wondering if there is any advise on how to populate the robots.txt?
There are some items on the forum about it, but mostly about preventing spidering.
My question (I think) is simple, what folders should be disallowed?
I found this at http://piwigo.org/robots.txt, would this be a good example?
User-agent: *
Crawl-delay: 10
# Directories
Disallow: /includes/
Disallow: /misc/
Disallow: /modules/
Disallow: /profiles/
Disallow: /scripts/
Disallow: /themes/
# Files
Disallow: /CHANGELOG.txt
Disallow: /cron.php
Disallow: /INSTALL.mysql.txt
Disallow: /INSTALL.pgsql.txt
Disallow: /install.php
Disallow: /INSTALL.txt
Disallow: /LICENSE.txt
Disallow: /MAINTAINERS.txt
Disallow: /update.php
Disallow: /UPGRADE.txt
Disallow: /xmlrpc.php
# Paths (clean URLs)
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /logout/
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/login/
# Paths (no clean URLs)
Disallow: /?q=admin/
Disallow: /?q=comment/reply/
Disallow: /?q=logout/
Disallow: /?q=node/add/
Disallow: /?q=search/
Disallow: /?q=user/password/
Disallow: /?q=user/register/
Disallow: /?q=user/login/
PS .. I also use these options, so my url's are a bit different;
$conf['question_mark_in_urls'] = false;
$conf['php_extension_in_urls'] = false;
$conf['category_url_style'] = 'id-name';
$conf['picture_url_style'] = 'id-file';
Thanks,
Matthijs
Piwigo version: 2.6.3
PHP: 5.5.9-1ubuntu4 (Show info) [2014-06-12 07:35:14]
MySQL: 5.5.37-0ubuntu0.14.04.1-log [2014-06-12 07:35:14]
Offline
Hello
if you read carefully you will understand that's for Piwigo.org not for Piwigo
Then I would suggest to disallow all paths others than upload and galleries
Offline
Thanks .. I had to start somewhere ...
What about the files? You will need index.php... but NOT install.php, password.php profile.php etc?
Offline