Google crawls most sites with a smartphone user agent because it indexes the mobile version of pages. If your mobile page lacks content or links the desktop page has, Google may never see them.
Need help with SEO?
Comppound runs SEO and AI search for SaaS companies. We plan what to publish, produce it with AI, and every page passes automated checks, and a person reviews samples and anything high-risk.
- Published to your site and tracked in Search Console
- One person answers for your account
- Plans from $400 a month, terms from 3 months
What this means
Keep content, structured data, metadata and internal links equal across mobile and desktop. Serve the same important resources to the smartphone crawler.
Verifying visits
The user-agent string can be spoofed. Verify Googlebot with a reverse DNS lookup or Google's published IP ranges before trusting log entries.
Common mistakes
Hiding content on mobile
Anything missing from the mobile page may be missing from the index.
Frequently Asked Questions
Yes, though most crawling uses the smartphone agent. Check Google's crawler documentation for current details.
Keep reading
- Googlebot: Googlebot is the web crawler Google uses to fetch pages for Search. It discovers URLs, downloads them and passes them on to be rendered and indexed.
- Mobile-first indexing: Mobile-first indexing means Google crawls and indexes the mobile version of a site's pages. Content missing on mobile may not be indexed at all.
- User agent: A user agent is the string a browser or crawler sends to identify itself. In SEO it is how robots.txt rules and server logs tell bots apart.
- Log File Analysis: Log file analysis means reading server access logs to see exactly which URLs search engine and AI crawlers requested, when, and what they got back.
- Algorithmic Penalty: An algorithmic penalty is a ranking loss caused by Google's automated systems, with no notice in Search Console and no reconsideration request.
- Baidu: Baidu is the leading search engine in mainland China. Ranking there requires Simplified Chinese content and usually local hosting and licensing.
- BERT: BERT is a language model Google applied to Search in 2019 to understand how word order and small words change the meaning of a query.
- Dwell Time: Dwell time is how long a searcher stays on a page after clicking a result before returning to the results. Pogo-sticking is bouncing straight back and clicking another result.
