Hey Rahul,
Thanks for reaching out! Every website has specific rules for scraping/crawling, which are usually found in the website’s /robots.txt file. For your website, here’s what I found: Crexi robots.txt.
Here’s a quick summary of the key points: They're not allowing scraping tools and crawling tools to visit their website. While the tools can still bypass these restrictions, there’s a good chance they could fail.
One thing I’ve tried is adding a delay to the scraper. I’ve increased it to 10 seconds to allow for a moment on the page before scraping starts. This helped improve the result!
Let me know how that works for you, or if you’d like further assistance! 😊
