AINewsBot
The automated reader of Future Financial Lab, a digital publication on real estate and investment in the UAE. The name AINewsBot is kept from the publication's working title; the sources' permissions and robots.txt rules already refer to it.
What it does
- Reads the public news and press pages, feeds and sitemaps of sources that have given us permission, or that publish under an open licence.
- Uses facts, figures and short quotations in articles that always name the source and link to the original page.
- Keeps a copy of the page text only for a limited time, to check each statement against the original.
How it behaves
- It identifies itself as
AINewsBotand links to this page. - It obeys
robots.txt, uses only HTTPS, and never visits pages that need a login. - It reads a source at most once an hour by default, at most 15 new pages per visit, with at least one second between page requests.
- It sends conditional requests (ETag and Last-Modified), so unchanged pages are not downloaded again.
- It stops when asked to, and when a site answers with errors.
Contact
To ask us to slow down, stop, correct or remove something, or to offer a feed or an API: editor@fflab.news
По-русски
AINewsBot — автоматический робот издания Future Financial Lab (имя осталось от рабочего названия, на него уже ссылаются разрешения источников). Он читает только открытые страницы новостей, ленты и карты сайта источников, давших разрешение или публикующих данные по открытой лицензии, соблюдает robots.txt, ходит только по HTTPS, не заходит на страницы с входом, обращается к источнику не чаще раза в час и сразу останавливается по вашей просьбе.