A technical SEO audit is the fastest way to find the invisible problems that keep great content from ranking. Before you touch keywords or backlinks, you need to confirm Google can crawl, render, and understand your pages. This guide walks through a practical, prioritized checklist covering crawlability, Core Web Vitals, structured data, and mobile parity. Here is where most sites quietly lose rankings.
Start Your Crawlability Audit With robots.txt and Indexing Rules
Crawlability is the foundation of everything else, so start here. The single most dangerous file on your site is robots.txt, and the most common mistake is using it to do a job it was never designed for. Google’s robots.txt guide is blunt about this: a robots.txt file tells crawlers which URLs they can access, but it is not a mechanism for keeping a page out of Google. A page disallowed in robots.txt can still be indexed if other sites link to it, appearing in results without a description.
This creates a trap that ensnares even mature teams. If you want a page out of the index, you need noindex, and for the rule to work, the page must not be blocked by robots.txt so the crawler can actually see it. Blocking and noindexing the same URL at once is self-defeating, because Google never reaches the tag. Audit for these conflicts first.
Work through these crawlability checks:
- Confirm the right host. List every crawlable host, then fetch robots.txt on each one. Teams frequently edit the wrong file.
- Strip unsupported directives. Googlebot ignores crawl-delay entirely, and crawl speed is managed through Search Console settings instead.
- Never block rendering resources. Blocking CSS, JavaScript, or key images stops Google from understanding the page.
- Check Search Console reports. Use both the Page Indexing report and the Crawl Stats report, since each contains different information about your URLs.
For bigger properties, crawl budget and URL sprawl become the main concern. If you manage thousands of templated pages, our guide to enterprise technical SEO goes deeper on taming faceted navigation and parameter noise.
Diagnose Core Web Vitals and Page Experience Signals
Once Google can reach your pages, the next question is whether the experience is fast and stable. Core Web Vitals are three field metrics measured from real Chrome users, not lab tests. A page passes only when all three meet the good threshold at the 75th percentile of real visits, measured separately for mobile and desktop. That percentile detail matters: at least three out of four visits must hit the target.
The current thresholds, documented by Google, are worth committing to memory:
- Largest Contentful Paint (LCP): loading speed. Good is 2.5 seconds or less, and poor is more than 4 seconds.
- Interaction to Next Paint (INP): responsiveness. Good is 200 milliseconds or less, and poor is more than 500 milliseconds. In March 2024, Google replaced FID with INP as the official responsiveness metric.
- Cumulative Layout Shift (CLS): visual stability. Good is 0.1 or less, and poor is more than 0.25.
Passing is harder than most teams assume. According to the 2025 Web Almanac from HTTP Archive, only 48 percent of mobile sites and 56 percent of desktop sites achieve a good score across all three metrics. Diagnosis is about matching symptoms to causes. Slow LCP usually traces back to server response time, unoptimized images, or render-blocking CSS and JavaScript. Server response time sets a hard floor under LCP, because nothing can render until the browser receives the HTML document. Poor INP almost always points to heavy JavaScript tying up the main thread, while CLS comes from images without dimensions, late-loading fonts, and content injected above existing elements.
Use the Core Web Vitals report in Search Console for a site-wide view from field data, then drill into individual URLs with PageSpeed Insights to see lab diagnostics. Fix at the template level so one change improves thousands of pages at once.
Validate Structured Data and Schema Markup for Rich Results
Structured data helps Google understand your pages and makes them eligible for enhanced appearances. It is important to frame expectations correctly. A structured data manual action removes a page’s eligibility for rich results, but it does not affect how the page ranks in web search. Schema is an eligibility and understanding signal, not a direct ranking lever.
During your audit, check these fundamentals:
- Use the recommended format. Google supports JSON-LD, Microdata, and RDFa, with JSON-LD recommended.
- Include required properties. Items missing required properties are not eligible for rich results, and more complete recommended properties improve result quality.
- Match markup to visible content. Do not mark up content that is not visible to readers of the page.
- Keep image URLs crawlable. Images referenced in schema must be accessible, not blocked by robots.txt or hidden behind authentication.
Validate before you deploy. The Rich Results Test identifies errors that prevent eligibility and warnings that may reduce quality, and shows which rich result types the page qualifies for. One important update: FAQ rich results stopped appearing in Google Search on 7 May 2026, and Google removed its FAQ structured data documentation on 15 June 2026. The JSON-LD pattern itself did not break, but the visible SERP enhancement is gone, so stop treating FAQPage markup as a rich-result play. Because AI-driven search increasingly leans on clean entity signals, pairing schema with a strong entity-first approach is a smarter use of the same effort.
Confirm Mobile-First Indexing and Content Parity
Google indexes the mobile version of your site, which makes mobile content parity non-negotiable. Google’s documentation lists equivalent primary content as the first requirement, and only the content shown on the mobile site is used for indexing. If you trimmed copy, links, or structured data from mobile for a cleaner look, you likely trimmed it out of the index too.
The most frequent parity failures involve how content loads. Text in accordions, tabs, or “show more” areas is fully counted as long as it is present in the rendered DOM, even when visually hidden. The problem is interaction-gated content. If text is only loaded on click and is not in the DOM beforehand, Googlebot does not see it, because the bot does not click. The same applies to menus and primary copy injected by JavaScript after the initial response.
Run this parity checklist:
- Same content, titles, and metadata across mobile and desktop.
- Same structured data shipped on both versions.
- Primary content in the initial HTML, not revealed only after a tap, swipe, or type.
- No stray noindex on the mobile template.
- Rendering resources unblocked, so Googlebot sees the page as users do.
Verify with the URL Inspection tool. It shows the rendered HTML and a screenshot of how Googlebot sees the page, which is how you confirm lazy-loaded content actually appears in the rendered source. Compare it against your desktop view and the original source. What matters is the rendered DOM, not the raw markup.
Audit JavaScript Rendering and Crawl Efficiency
Modern frameworks add a rendering step between crawling and indexing, and that step is where many sites lose content. Googlebot renders pages with a headless Chrome browser and executes JavaScript, so client-side rendered content is generally indexed, though with a potential crawl delay while Google processes the script. Delay is the risk: the longer Google waits to see your main content, the slower and less reliably it enters the index.
Mobile makes this worse. A JavaScript bundle that takes 150 milliseconds to execute on a high-end laptop can take around 600 milliseconds on the mid-tier Android device Google uses as a PageSpeed benchmark. The fix is architectural. Server-side rendering or static generation remains the gold standard for indexability, because the full HTML is available in the initial response without JavaScript execution.
Protect crawl efficiency alongside rendering:
- Keep redirect chains short. Long redirect chains have a negative effect on crawling.
- Watch serving capacity. Googlebot scales back crawling when it detects the server is struggling to respond.
- Avoid infinite spaces. Calendars, internal search results, and uncontrolled parameters drain budget that deeper pages need.
- Keep your sitemap clean. It should contain canonical URLs, and every URL in it should be crawlable.
Technical fixes compound with content quality. If you want the strategic layer that sits on top of a clean crawl, our overview of the ongoing value of SEO and our broader SEO checklist connect the dots between technical health and visibility.
Build Your Recurring Technical SEO Audit Workflow
A one-time audit fixes today’s problems. A recurring workflow stops tomorrow’s from shipping. The reason is practical: robots.txt is not an instant switch. Google notices robots.txt changes during automatic crawling and updates the cached version roughly every 24 hours, with a faster refresh available through the Search Console robots.txt report. Mistakes take time to surface and time to undo, so prevention beats cleanup.
Structure your recurring audit in priority order. First, confirm access: robots.txt, indexing rules, and rendering resources. Second, confirm understanding: mobile parity, JavaScript rendering, and structured data validity. Third, confirm experience: Core Web Vitals at the template level. Re-run after every significant release, and schedule a full pass quarterly. Demonstrating expertise, experience, authority, and trust starts with a site Google can actually read, which is exactly what EEAT best practices reward. If internal bandwidth is tight, our guide to choosing a growth partner can help you scope the work.
Technical SEO is not glamorous, but it decides whether your best content ever gets a fair shot. Audit crawlability first so Google can reach your pages, fix Core Web Vitals so the experience holds up, validate structured data for eligibility, and enforce mobile parity so nothing gets lost in rendering. Make it recurring. The takeaway is simple: a site Google can crawl, render, and understand is a site that can rank.
Frequently Asked Questions About Technical SEO Audits
How often should I run a technical SEO audit? Run a focused audit after every significant site release and a full audit quarterly. Because Google refreshes cached robots.txt roughly every 24 hours, errors can take time to appear and resolve, so catching them early matters.
Does robots.txt remove a page from Google? No. A disallowed URL can still be indexed if linked from other sites, appearing without a description. To reliably remove a page, allow crawling and apply noindex, or use password protection.
What counts as a good Core Web Vitals score? Good means LCP under 2.5 seconds, INP under 200 milliseconds, and CLS under 0.1, each at the 75th percentile of real visits. A page must pass all three to pass overall.
Will structured data improve my rankings directly? Not directly. Even a manual action for bad schema only removes rich-result eligibility and does not affect web-search ranking. Schema earns eligibility and improves understanding, not position.
Are FAQ rich results still worth marking up? Not for SERP enhancements. Google stopped showing FAQ rich results on 7 May 2026 and removed the documentation on 15 June 2026. The markup causes no harm, but it no longer produces a visible feature.
Why is my JavaScript content not getting indexed? It likely loads only after user interaction or heavy script execution. Googlebot does not click, so content loaded only on click and absent from the DOM is never seen. Move critical content into the initial server-rendered HTML.
Related Articles
How Digital PR & SEO Work Together
Entity-First SEO, Knowledge Graph, and AI Search Guide
