OHappyDay wrote:

Look at the associated IP address. If it is 54.36.148.121 then it belongs to the ahrefs.com bot (it is a marketing company).

Hello OHappyDay.
I have the following information about it:

The AhrefsBot visits your website regularly and downloads many (sometimes all) pages, images, and resources. In essence, it performs the following tasks:

What exactly does AhrefsBot do on your website?
It crawls your page content
Reads HTML content, titles, meta data, headings, text content, and internal links.
Goal: Structural analysis of your website for SEO purposes.

It scans your backlinks
Identifies which external websites are linking to your pages.
This information is used in Ahrefs tools like the Backlink Checker.

It analyzes keywords and rankings
Attempts to determine how relevant your content is to specific search terms.
These insights help Ahrefs users understand your search engine rankings.

It tracks internal linking
Detects how your pages are linked internally.
Helps evaluate the architecture of your website.

It downloads resources
CSS, JavaScript files, images, or fonts – depending on the page.
Goal: A full understanding of the page (e.g., for load time analysis).

What does this mean for you?
Advantages:
- If you use Ahrefs, you'll get valuable data about your SEO performance.

Your site will be analyzed more frequently, making its structure and visibility more transparent for you and your SEO team.

Disadvantages:
- Server load: Heavy crawling can slow down your site.

- Data exposure: Your backlinks, content, and structure become publicly accessible through Ahrefs – including to competitors.

- Bandwidth usage: Especially on shared hosting, AhrefsBot may consume unnecessary resources.

- Privacy and rights: Even content not meant for public indexing (e.g., forgotten “noindex” tags) may be scanned unless properly blocked.

Olaf

You sure ?  compa is not a user agent, it's the beginning of compatible and the rest of the line cannot been displayed.

I block these ones :

SemrushBot
AhrefsBot
MJ12bot
PetalBot
DotBot
Yandex
BaiDuSpider
Mail.ru
BLEXBot
SEOkicks-Robot
amazonaws.com

I am not sure how to enter amazonaws.com and Mail.ru.

Maybe I'll add the generic crawler and spider

I have to report a problem with bot protection:

When I try to connect/login to my website through the piwigo app on my Android mobile device I get blocked.
I have no idea how to resolve this. Deactivating the plugin is not an option now that it works so good.

In the log I see this:
Missing referrer and cookies UA: Dart/3.4 (dart:io)

Ideas?

Klaus

Last edited by OHappyDay (2025-06-08 21:39:42)

I am aware of that possibility but such IP addresses will change over time and they also depend on which network I am connected with (home wireless, other wireless connections when not at home etc.).

OHappyDay wrote:

I found a solution: I added the identification for the app ("Dart") to the file whitelisted_bots.txt. Now the app can connect to the website.

Exactly, you can then add Dart and Piwigo to the whitelist later!

Olaf