Moving to a CDN? Google says don't lock out Googlebot
Google explains why crawling spikes after a CDN switch and how bot-blocking rules can quietly push a small business site out of Search.
Plenty of small business websites now sit behind a content delivery network. Sometimes the owner turned one on to speed up the site, sometimes a web host or a site builder did it for them, and sometimes it came bundled with a security plan after a scare. Google's Search Central team has published a plain-language post about what happens to crawling during that switch, and the most useful part for local businesses is not about speed at all. It is about a security setting that can make a site disappear from Google without anyone noticing for weeks.
Here is what Google said, and what a dentist, roofer, or law firm should check before and after a CDN change.
Why crawling jumps right after the switch
A CDN keeps copies of your pages on servers spread around the world, so visitors get them from somewhere nearby instead of from your main server, which Google calls the origin. That is the whole appeal: faster pages and less load on the machine that actually runs your site.
Google's post explains a side effect that surprises people. Its crawlers watch how quickly a server answers. When responses get slower, they back off. When responses become fast and steady, Google treats that as a sign the site can handle more, and the crawl rate can climb if there is demand for it. Pointing your domain at a CDN makes the site look much faster overnight, so Google may start asking for more pages.
The catch is that a new CDN starts empty. Google calls this a cold cache. The first time any page is requested at a CDN location, the CDN has no copy yet, so it fetches the page from your origin. Visitors mostly hit the same popular pages, which get cached quickly. Crawlers do the opposite and work through every URL they know, including old service pages, blog archives, and filtered listings. Until the cache fills, those requests still land on your origin. Google notes that crawlers do not ramp up instantly and that many CDNs smooth out traffic, but the origin still does the heavy lifting at first.
For most local business sites with a few dozen or a few hundred pages, this burst is small. Google's own crawl budget guide says it is aimed mainly at sites with tens of thousands of pages or more. The bigger risk for a small site is the next part.
The real danger: blanket bot blocking
When owners see a spike of automated traffic in their logs, Google says it is tempting to switch on a CDN or firewall setting that challenges every bot. Google's warning is blunt. Its crawlers cannot pass a CAPTCHA. If a rule treats all automated visitors the same, Googlebot hits the challenge page instead of your content. In Google's words, from its side the site then "appears deliberately restricted or broken," and if the block stays in place Google's crawlers back off and pages, shopping listings, and images can start dropping out of Search.
For a plumber or an HVAC company, that means service pages and location pages fading from results during the exact months you need calls. And because the site still loads normally for you and your customers, nobody on the team sees a problem.
What Google recommends
Google's post lists three practical steps:
- Let verified bots through. If you need bot protection, use the setting most CDNs and firewalls offer that allows known, verified bots, and target specific unwanted bots instead of blocking all of them. Google publishes its crawler IP ranges and explains how to confirm a visitor really is Googlebot on its page about verifying Google's crawlers.
- Warm the cache first. Before sending all traffic through the CDN, request your most important pages through it at a steady pace so copies are ready. Google suggests your highest-traffic pages, upcoming promotions or seasonal pages, and the URLs crawled most often.
- Use a temporary brake if needed. If crawling truly overloads your server while the cache warms up, Google says you can briefly return a 429 or 503 status code. Its guide to reducing the crawl rate treats this as an emergency measure, and the CDN post says to keep it to a short window of a few hours or one to two days at most. Leave it longer and you risk pages being dropped.
What this means for a local business
This is guidance from us, based on Google's post. If you are moving to a CDN, or a vendor is moving you, take these steps:
- Ask who changed what. Many owners do not know their site sits behind a CDN or that an aggressive bot-fighting setting is on. Ask your web person or host which security rules are active, and whether verified bots such as Googlebot are allowed.
- Test from the outside. After any switch, check that your pages return a normal 200 status and not a challenge page. Our free HTTP header checker shows the status code and headers a page returns, and the spider simulator shows roughly what a crawler sees.
- Do not block crawlers in robots.txt by accident. If you want to keep certain bots out, name them specifically. Our robots.txt generator helps you write targeted rules instead of a blanket ban.
- Watch Search Console for two weeks. In the weeks after a switch, look for new crawl errors or pages dropping out of the index. A sudden fall in indexed pages right after a security change is a strong hint that something is blocking Google.
- Mind the timing. Google's November core update is still settling, as we covered in our story on reading traffic during the update. Avoid stacking a CDN migration on top of other big site changes, so if traffic moves you can tell which change caused it.
Faster pages are good for visitors and for conversions, and a CDN is a reasonable investment for a growing business. The lesson from Google's post is simply that speed and security settings need a quick check before you call the job done. A firewall that keeps out spam bots but also keeps out Google can cost more leads than any slow page ever did.
If you run a practice like a dental office or a trade business like roofing, your service and location pages are your storefront in Search. Make sure the front door is open to the one bot that sends you customers.