Page URL: https://ahrefs.com/robot/site-audit
Analyze websites, track social media, build backlinks – Ahrefs has you covered. Try our marketing and SEO tools Site Explorer and Content Explorer today!
Page Headers
| 0 | HTTP/1.1 200 OK |
| Server | nginx |
| Date | Sun, 01 Nov 2020 08:49:10 GMT |
| Content-Type | text/html; charset=UTF-8 |
| Connection | close |
| Vary | Array |
| Cache-Control | private, must-revalidate |
| pragma | no-cache |
| expires | -1 |
| Set-Cookie | XSRF-TOKEN=aNgFZ3tfLBR1oAozE4Mvhj0PddzxzeEsZ7DVncAD; expires=Sun, 01-Nov-2020 12:49:10 GMT; Max-Age=14400; path=/; domain=.ahrefs.com; secure |
| Strict-Transport-Security | max-age=63072000 |
| X-Frame-Options | SAMEORIGIN |
| Content-Security-Policy | frame-ancestors ‘self’ |
| X-Request-ID | 4c9febdb4296f607f3ebaab14a2663fd |
Keyword Frequency
| ahrefssiteaudit | 18 |
| your | 17 |
| website | 16 |
| robots | 12 |
| txt | 12 |
| free | 11 |
| try | 10 |
| site | 10 |
| seo | 10 |
| ahrefs | 10 |
Keyword Cloud
Ahrefs SEO Tools Backlink Checker Report script Introducing Webmaster Free audits backlink and keyword data Learn more Site Explorer Keywords Audit Rank Tracker Content Pricing Help Blog Academy Our Deutsch English Espa ol Fran ais Italiano Nederlands Polski Portugu s Svenska T rk e Sign in Start a -day trial for AhrefsBot AhrefsSiteAudit bot Version Bot Type Good Identifies itself has an official moniker Category Obeys Robots txt Yes by default website owners can request to disobey robots on their sites Crawl Delay crawl delay User-Agent String Mozilla compatible http ahrefs com robot Reverse DNS suffix IP address range Check the status of your our crawler What is Web Crawler that powers Ahrefs’ tool This used users analyze websites search issues with technical on-page any unless disallowed To prevent excessive load server crawling limited per seconds Website at higher speeds allow ignore do they need verify ownership doing If you noticed log this means someone had performed analysis using apos crawls following its internal links from one URL another It also checks resources such as CSS files JS images will not add numbers Google Analytics traffic Does it respect file By Only verified site How control follows disallow directives So if some reason want put two lines into User-agent br Disallow You or part Read about www robotstxt org check prior new Please note contains errors won t be able recognize commands continue way did before think somehow misbehaving have questions please don hesitate contact support team Main tools Extra Domain Comparison Batch Analysis Link Intersect Gap Email Alerts Toolbar WordPress Plugin Word Count Traffic Features Try free Broken Authority Keyword Generator YouTube Tool Amazon nbsp Bing SERP Difficulty About Team Jobs blog Tech Robot API Contact us Pte Ltd H Raffles Quay – Hong Leong Building Singapore Terms amp conditions Privacy policy days dollars Full access gives full all features chosen plan Lite then mo Essential small businesses hobby projects keywords tracked pages month crawled Standard Perfect professionals in-house marketers I accept Conditions Continue upgrade downgrade cancel subscription anytime No contracts no hidden charges