linkedin
  • Become a Seller

Semrush Questions & Answers

View Product | Have a Question?

Showing 1- 1 of 1 questions

img
Editor's ChoiceMost Popular
Q:

Why am I getting page is not accessible error note for some of my pages in SEMRUSH?

Farman Hussain . Sep 19, 2023 A:

If you encounter the 'page is not accessible' error on SEMrush, it could mean that the On-Page SEO Checker crawler is either blocked or unable to crawl your page. Here are some possible reasons and solutions you can try:

  1. Check your robots.txt file: Make sure that SEMrush's user agents are allowed to crawl your website's pages. If they are not blocked in robots.txt, you will need to allow the following IP addresses and User-agent with your hosting provider and any plugins/services you may manage your site with (such as Cloudflare or ModSecurity):
    • IP addresses: 85.208.98.53, 85.208.98.0/24
    • User-agent: SemrushBot-SI
    • Port 80: HTTP, Port 443: HTTPS
    • Site Audit bot IP address: 85.208.98.128/25 (a subnet used by Site Audit only)
    • User-agent name: SiteAuditBot
  2. Check your crawl-delay settings: If you receive the error message 'SEMRushBot-Desktop couldn't crawl the page because it was blocked by robots.txt,' it means that your crawl-delay settings do not meet On Page SEO Checker's requirements. On Page SEO Checker crawlers only accept a crawl-delay of 1 second. Anything more than that would cause the crawler to ignore the page. To fix this issue, change the crawl-delay in your robots.txt to 1 second.

If these solutions do not work, you may want to check for other common crawlability problems such as nofollow links, redirect loops, bad site structure, or slow site speed. The SEMrush Site Audit tool can help you identify these issues.

Chandan Singh . Sep 19, 2023

How was your experience with this product?

  • img
  • img
  • img
  • img
  • img
img

150+ experts available

Get Free Advice to any Individual or Business Related Queries.

Get Expert Advice

Got any questions?

Ask Question from Real Users or Software Experts

Ask Question

Still got Questions on your mind?

Get answered by real users or software experts

Disclaimer

Techjockey’s software industry experts offer advice for educational and informational purposes only. A category or product query or issue posted, created, or compiled by Techjockey is not meant to replace your independent judgment.

Software icon representing 20,000+ Software Listed 20,000+ Software Listed

Price tag icon for best price guarantee Best Price Guaranteed

Expert consultation icon Free Expert Consultation

Happy customer icon representing 2 million+ customers 2M+ Happy Customers