Radar reads the home page on every briefing run, and pricing and careers pages at most weekly. If robots.txt blocks Radar or all readers from the whole site, it fetches nothing. Page-level rules are not read yet.
What it does
For each account with a web address, Radar looks at the home page and, when linked, a pricing page and a careers page. It reads at most three pages per company. The home page is read on every briefing run. The pricing and careers pages are read at most once a week each.
Before that it reads the site's robots.txt. If the file tells Radar's reader, or every reader, to stay out of the whole site, Radar fetches nothing from that site.
Why
A company's website is theirs. If the owner has asked automated readers to stay away, a competitor tool shouldn't ignore it.
What it won't do
- It doesn't read a site whose robots.txt blocks the whole site for it.
- It doesn't read more than three pages per company.
- It doesn't yet honour rules that block single pages such as a careers path.
How to use it
Open the Radar guide in your dashboard, or start a trial to try it.