ProjectSegfault/go-away
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

readme: note existence of snippets

Browse Source
This commit is contained in:
WeebDataHoarder
2025-04-23 17:41:45 +02:00
parent d56d621f7a
commit 612362dbe5
1 changed files with 8 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
README.md
View File

@@ -235,6 +235,14 @@ Important notes:
* Add or modify rules to target specific pages on your site as desired.
* By default Googlebot / Bingbot / DuckDuckBot / Kagibot / Qwantbot / Yandexbot are allowed by useragent and network ranges.
### Snippets
You can define snippets to be included. YAML anchors/aliases are supported.
See [examples/snippets/](examples/snippets/) for some defaults including indexer bots, challenges and other general matches.
## Why do this?
In the past few years this small git instance has been hit by waves and waves of scraping.
This was usually fought back by random useragent blocks for bots that did not follow [robots.txt](/robots.txt), until the past half year, where low-effort mass scraping was used more prominently.