What's the difference between robots.txt and noindex, and which should I use?

These directives operate at fundamentally different layers of Google's crawling and indexing pipeline, and confusing them causes real problems. robots.txt is a crawling directive: it instructs Googlebot not to fetch a URL. noindex is an indexing directive embedded in the page's HTML (or HTTP header) that tells Googlebot not to include the page in its search index. The critical misunderstanding: a page blocked by robots.txt can still be indexed if Google discovers it from external backlinks. Google knows the page exists (because links point to it), but can't read the noindex tag (because it's blocked from fetching the page), so it may index a blank entry for that URL. The practical rule: use robots.txt to prevent crawling of pages you want to keep private but don't care about indexation status (staging environments, internal tools). Use noindex for pages you want Googlebot to fetch but not index (thank-you pages, paginated archive pages beyond page 2, thin content you're keeping for UX reasons). Never use robots.txt as a substitute for noindex on pages you actively want removed from search results.

How do I check my real Core Web Vitals scores, not the PageSpeed lab data?

PageSpeed Insights lab scores are diagnostic tools. They tell you what your page performance would be under controlled conditions. They are not what Google uses for ranking. The ranking-relevant scores come from the Chrome User Experience Report (CrUX), real performance data collected from actual Chrome users visiting your site. There are three ways to access your real CrUX data. First, Google Search Console: the Core Web Vitals report shows field data segmented by device type (mobile/desktop) and status (good/needs improvement/poor). This is the most actionable view for SEO purposes. Second, PageSpeed Insights API: when run with a real URL (not a lab crawl), PSI shows both lab and field data side by side, look for the 'Field Data' section at the top, not 'Lab Data.' Third, Ranking Lens monitoring: continuous automated tracking of your CrUX scores across your full URL inventory, not just spot-checks. The common mistake is optimizing based on lab scores while your field scores (what Google actually measures) remain poor. This happens especially with JavaScript-heavy sites where real user performance differs significantly from controlled lab conditions.

What's the fastest way to fix CLS (Cumulative Layout Shift)?

CLS above 0.1 is almost always caused by one of four specific patterns, and diagnosing which pattern you have determines the fix. Pattern 1: Images without explicit width and height attributes. The browser can't reserve space before the image loads, so content shifts down when it arrives. Fix: always include width and height on img tags, or use CSS aspect-ratio. Pattern 2: Late-loading web fonts causing text reflow. Text renders in a fallback font, then shifts when the custom font loads. Fix: use font-display: optional (prevents layout shift entirely by not swapping if the font loads after the render deadline) or font-display: swap with matching fallback font metrics. Pattern 3: Dynamically injected content, ad slots, cookie banners, newsletter signup bars that load after initial render. Fix: pre-allocate the space with min-height on the container before the content loads. Pattern 4: Animations that affect layout properties (height, width, top, left). Fix: animate transform and opacity instead, these are GPU-composited and don't trigger layout recalculation. Use Chrome DevTools' Performance panel → Layout Shifts to identify exactly which elements are shifting and when.

Does page speed actually affect Google rankings, or is that overblown?

The honest answer based on our analysis across several hundred sites: Core Web Vitals are a significant ranking factor at the extremes and a marginal tiebreaker in the middle. Specifically: a page with CLS above 0.25 will be demoted enough to lose meaningful position in competitive SERPs, this is well-documented and consistent. A page with LCP above 4 seconds faces similar treatment. But the difference between a site with LCP of 1.2 seconds versus 2.1 seconds? In our testing, negligible direct ranking impact if content quality and authority are equal. Where speed matters more than most SEOs admit is in the indirect signals: faster pages generate lower bounce rates, higher engagement, and more return visits, all of which feed into Google's user satisfaction signals. A page where users wait 3 seconds for content and immediately bounce is performing poorly on every signal Google can measure. Fix performance because it improves the user experience that generates ranking signals, not because a 0.3-second LCP improvement will directly move you from position 4 to position 1.

How should I handle JavaScript-rendered content for SEO?

Googlebot can render JavaScript, but with a delay, inconsistently, and with significant resource constraints compared to a standard browser. The practical reality: content that requires JavaScript execution to appear in the DOM is typically indexed days to weeks after initial discovery, not immediately. Content critical for ranking (your main article body, headers, internal links) should be present in the initial HTML response, not dependent on client-side JavaScript. There are three viable approaches. Server-Side Rendering (SSR): the server executes JavaScript and returns complete HTML. Best for highly dynamic content that needs fresh data. Static Site Generation (SSG): pages pre-built to complete HTML at build time. Best for content that doesn't change per-user. Hybrid with selective hydration: static HTML shell with JavaScript progressively enhancing interactive elements. Best for complex apps. The test: fetch your URL in Google Search Console's URL Inspection tool → View Crawled Page. Compare the rendered content to what a user sees. If they diverge, you have a rendering problem. Next.js (which this blog uses) handles this correctly by default with App Router and RSC, HTML is complete before hydration.

What structured data should every blog post have in 2025?

For a blog article optimized for both Google search features and AI citation, three schema types are non-negotiable. First, Article schema (or BlogPosting, which is a subtype): include headline, datePublished, dateModified, author (as Person with name and URL), publisher (as Organization with logo), and the article's description. The author Person entity is particularly important for E-E-A-T, Google uses this to build a knowledge graph entity for the author. Second, FAQPage schema: if your article contains a FAQ section (which it should for GEO purposes), wrap each Q&A pair in FAQPage schema. This unlocks FAQ rich results in Google, which can more than double your SERP real estate on featured queries. Third, BreadcrumbList: provides Google with explicit navigation context and enables breadcrumb rich results. Optional but valuable: HowTo schema for step-by-step instructional content. Validate everything with Google's Rich Results Test before deploying. Common mistake: implementing FAQPage schema on pages where the questions and answers aren't actually visible in the page content, Google will reject these.

What's the most commonly overlooked technical SEO issue in 2025?

Based on auditing dozens of sites, the single most overlooked issue is orphaned pages, pages with no internal links pointing to them. These pages receive zero PageRank from your site's internal link distribution, regardless of how strong your domain is externally. They are effectively invisible to Google's crawlers unless discovered through an XML sitemap or external backlinks. The insidious part: orphaned pages tend to be your weakest content (that's often why they weren't linked to), so they're also dragging down your site's overall content quality signal under the Helpful Content system. A secondary commonly missed issue is redirect chain accumulation on sites that have gone through multiple redesigns or CMS migrations. A chain of A→B→C is so common that most site owners don't realize they have it. Each hop adds latency (which affects LCP for pages accessed via those redirects) and dilutes PageRank. Use Screaming Frog or Sitebulb to map your redirect graph and flatten everything to single-hop 301s. Third: hreflang implementation errors on multilingual sites, specifically, pages that reference hreflang URLs that don't themselves contain the reciprocal hreflang tag. Google ignores unconfirmed hreflang signals.

Technical SEO Checklist 2025: 47 Checks That Actually Matter

Why This Checklist Is Different

Most technical SEO checklists are recycled from 2020. They list "have an XML sitemap" alongside "pass Core Web Vitals" as if these are equivalent concerns. They don't tell you which issues create hard ceilings on rankings and which are nice-to-haves.

This checklist prioritizes by impact. We've divided 47 checks across seven domains and marked each with its ranking impact level, so you know where to focus limited time and budget.

A few items that were important in 2020 but are now table stakes aren't here. If your site is on HTTPS and has a sitemap, congratulations, that's baseline. This list focuses on where sites actually fail in 2025.

Key Takeaway: Technical SEO issues don't suppress rankings gradually, they create hard ceilings. A site with JavaScript rendering problems will cap out regardless of content quality and backlinks. Fix the structural issues before optimizing anything else.

Domain 1: Crawlability & Indexation

These are foundational. Crawlability problems prevent Google from seeing your content at all. Indexation problems mean your content exists but can't rank.

robots.txt

Check	What to Verify	Impact
robots.txt exists and is valid	Fetch yourdomain.com/robots.txt, confirm it returns 200, not 404	Medium
No critical pages disallowed	Verify your main content URLs are not blocked; run Screaming Frog with "Check robots.txt"	High
User-agent specificity	If blocking AI crawlers (GPTBot, ClaudeBot), confirm you're using named user-agents, not wildcard blocks that affect Googlebot	Medium

robots.txt accessible and syntactically valid
No money pages, category pages, or article URLs blocked
AI crawler policy intentionally set (allow or selectively disallow)

XML Sitemap

Sitemap submitted in Google Search Console → Sitemaps
Sitemap includes only indexable URLs (no noindex pages, no 301 redirects, no 404s)
Sitemap <lastmod> dates are accurate (not all set to today's date, which is a common generator error)
For large sites: sitemap index file with multiple sitemaps by content type

Common mistake: Sitemaps that include URLs returning 301 redirects. Google reads this as a signal that your sitemap is inaccurate, which reduces how often it's fetched.

Canonical Tags

Every page has a self-referencing canonical tag
Paginated pages (page 2, page 3) have canonical pointing to themselves, not to page 1
Canonical in HTTP header matches canonical in HTML (if both are present)
No canonical loops (page A canonicals to page B which canonicals back to A)

Redirect Architecture

No redirect chains longer than one hop (A→B→C is a chain; flatten to A→C)
All old internal links updated to point directly to final destination URLs
404 pages return actual 404 status code (test with curl -I url)

Key Takeaway: Run a redirect crawl with Screaming Frog or Sitebulb quarterly. Sites that have gone through even one CMS migration or redesign almost always have chain accumulation they don't know about.

JavaScript Rendering

This is where most modern sites fail silently.

Main content body is present in raw HTML (not injected by JavaScript)
Internal navigation links are present in initial HTML response
Test: GSC URL Inspection → View Crawled Page; compare rendered to raw HTML
No critical content hidden behind JavaScript-triggered modals or tabs

How to test in 30 seconds: curl -s yourdomain.com/your-article | grep "first paragraph text". If the grep returns nothing, Googlebot sees the same blank page.

Domain 2: Core Web Vitals

Core Web Vitals are Google's performance-to-ranking bridge. The thresholds below are hard limits from Google's documentation.

Metric	Pass	Needs Improvement	Fail	Ranking Impact
LCP (Largest Contentful Paint)	< 2.5s	2.5–4.0s	> 4.0s	High at failure threshold
INP (Interaction to Next Paint)	< 200ms	200–500ms	> 500ms	Significant for interactive sites
CLS (Cumulative Layout Shift)	< 0.1	0.1–0.25	> 0.25	High, directly suppresses rankings

Critical note: INP replaced FID (First Input Delay) as a Core Web Vital in March 2024. If your technical SEO documentation still references FID, it's outdated. INP measures the full interaction lifecycle, input delay, processing time, and rendering time, for all interactions, not just the first. It is significantly harder to pass on JavaScript-heavy applications.

LCP Optimization Checklist

Identify your LCP element for each page type using DevTools → Performance → LCP
LCP element (usually hero image or H1) is in the initial HTML, not lazy-loaded
Hero images use <link rel="preload" as="image"> in the <head>
Images served in WebP or AVIF format (30–50% smaller than JPEG at equivalent quality)
CDN with edge caching active, TTFB (Time to First Byte) under 600ms
No render-blocking <script> tags in <head> without defer or async

INP Optimization Checklist

No JavaScript tasks longer than 50ms blocking the main thread
Event handlers debounced or throttled appropriately
Third-party scripts (analytics, chat widgets) loaded asynchronously
React/Next.js: heavy computations moved to Web Workers or server components

CLS Optimization Checklist

All images have explicit width and height attributes
Font loading uses font-display: swap or font-display: optional
Ad slots and dynamic content containers have reserved height (CSS min-height)
No elements using layout-affecting CSS animations (animate transform not height)

Key Takeaway: Use Google Search Console's Core Web Vitals report for ranking-relevant data, not PageSpeed Insights lab scores. Lab scores diagnose problems. Field scores (CrUX) determine ranking impact.

Domain 3: Mobile-First Technical Requirements

Google uses mobile-first indexing for all sites. The mobile version of your page is the version that ranks.

Viewport meta tag present and correct: <meta name="viewport" content="width=device-width, initial-scale=1">
No horizontal scrolling at any viewport width (test at 320px, 375px, 414px)
Touch targets at least 44×44px (Google's minimum; Apple's HIG recommends 44pt)
Text readable without zooming at 16px base font size
No intrusive interstitials (pop-ups that cover main content on mobile), Google can penalize these
Test with Google's Mobile-Friendly Test (Search Console) and real device testing

Common Mobile Issues That Are Routinely Missed

Fixed-width elements: A container with width: 800px in a mobile viewport causes horizontal scroll. Audit with DevTools device emulation and look for horizontal overflow.

Font size below 12px: Google flags text smaller than 12px as unreadable on mobile. Check your CSS for font sizes below this threshold, especially in captions, footnotes, and disclaimers.

Interstitials timing: A pop-up that appears after 30 seconds on page doesn't trigger Google's penalty, but one that appears before the user can access content does. Time your overlays carefully.

Domain 4: Structured Data

Structured data serves two purposes in 2025: unlocking Google rich results features (which improve CTR) and signaling content type to AI retrieval systems (which affects GEO citation rates).

Required for Every Article

Article schema with: headline, datePublished, dateModified, author (Person), publisher (Organization with logo), description
Author Person entity: include name, url (link to author page), and ideally sameAs (links to social profiles, LinkedIn) for knowledge graph building
BreadcrumbList schema matching visible breadcrumb navigation

Site-Level Schema

Organization schema on homepage with logo, url, sameAs (social profiles), contactPoint
WebSite schema with potentialAction SearchAction for sitelinks search box (if applicable)

Validation

All schema validated with Google's Rich Results Test
No schema on pages where the content type doesn't match (FAQPage on a page with no FAQ)
Schema is in <script type="application/ld+json"> format, not RDFa or Microdata (JSON-LD is Google's preferred format)

Key Takeaway: FAQPage schema is the single highest-ROI structured data implementation for blog content. It can expand your SERP listing to 3–5x the normal height for featured queries, dramatically increasing click-through rate.

Domain 5: HTTPS & Security

These are table stakes in 2025, but the nuances matter.

Valid SSL certificate with at least 90 days until expiration (set up renewal alerts)
HTTP URLs redirect to HTTPS with 301 (not 302)
www and non-www redirect consistently (pick one and 301 the other)
HSTS header present: Strict-Transport-Security: max-age=31536000; includeSubDomains
No mixed content warnings (HTTP resources loaded on HTTPS pages)
Security headers present: X-Content-Type-Options: nosniff, X-Frame-Options: SAMEORIGIN

Mixed content check: Browser DevTools → Console. Any "Mixed Content" warnings need to be fixed. Common culprits: hardcoded HTTP URLs in images or scripts, third-party embeds using HTTP endpoints.

Domain 6: Performance & Asset Optimization

Performance beyond Core Web Vitals, these don't have direct ranking thresholds but affect the user experience signals that indirectly drive rankings.

Images

All images served in WebP or AVIF format (convert with Squoosh, ImageMagick, or at the CDN level)
loading="lazy" on all images below the fold
loading="eager" (or no loading attribute) on hero/LCP images, lazy loading the LCP element is one of the most common self-inflicted LCP problems
Responsive images with srcset for different viewport densities
No images larger than 200KB after compression (use 100KB as a target for most use cases)

Server & Delivery

Brotli compression enabled (20–30% better compression ratio than Gzip; check with curl -H "Accept-Encoding: br" -I url)
Browser caching headers set (Cache-Control: max-age=31536000, immutable for hashed assets)
CDN with edge PoPs covering your primary traffic geography
TTFB under 600ms from your primary geography (check with GTmetrix from relevant server location)

Fonts

Web fonts loaded with font-display: swap or font-display: optional
Font preconnect: <link rel="preconnect" href="https://fonts.googleapis.com">
Not more than 2–3 font families with 2–3 weights each (each additional weight is an additional HTTP request)
Consider self-hosting fonts to eliminate third-party DNS lookup latency

Key Takeaway: loading="lazy" on your hero image is one of the most common LCP problems we see in audits. It looks right in code review and completely breaks real-world performance. Always set loading="eager" or omit the attribute on above-the-fold images.

Domain 7: Internal Linking Architecture

Internal linking is PageRank distribution. Your internal link structure determines which pages receive the most PageRank from your domain's authority, and therefore which pages have the best chance of ranking.

Issue	Detection Method	Impact
Orphaned pages	Screaming Frog crawl → filter to pages with 0 inlinks	High, zero internal PageRank
Pages buried > 3 clicks from homepage	Screaming Frog crawl depth report	Medium, reduced crawl priority
Generic anchor text ("click here", "read more")	Screaming Frog → Anchor text report	Medium, missed keyword signals
Broken internal links	Screaming Frog → Response codes → 404	High, crawl budget waste

Internal Linking Checklist

No orphaned pages (every published URL has at least one internal link pointing to it)
No content buried more than 3 clicks from the homepage
All internal anchor text is descriptive (describes the target page's content)
No broken internal links (any returning 404 or 5xx)
Related articles linked within the body content, not just in sidebars
Hub pages (category/topic overview pages) link to all relevant sub-articles

Building an Effective Internal Linking System

The principle: internal links should reflect the semantic structure of your content, not just your navigation menu. If you have 10 articles about Core Web Vitals, they should all link to each other with descriptive anchor text, not just exist in a category page.

The practical implementation:

For each new article published, identify 3–5 existing articles it's relevant to and add contextual links in both directions
Quarterly: run a crawl and check the inlink count for each URL. Any page with under 3 internal links needs attention
Maintain a topic cluster map: which articles belong to which semantic cluster, and which is the hub that should receive the most internal link equity

The Priority Stack: Where to Start

Not all 47 checks carry equal weight. If you're starting a technical audit with limited time, work through these in order:

JavaScript rendering, if your content isn't in initial HTML, nothing else matters until this is fixed
Core Web Vitals field data, check GSC, identify your worst-performing pages, start with CLS (fastest to fix)
Orphan page audit, high impact, often overlooked, quick to fix
Redirect chain audit, run Screaming Frog, flatten all chains
Structured data, Article + FAQPage schema on all content pages
Image optimization, WebP conversion + lazy loading audit (catch the LCP lazy-load bug)
robots.txt + canonical review, confirm nothing important is accidentally blocked or miscanonicalized

The full 47-point audit above covers the complete picture. But these seven, executed well, address roughly 80% of the technical SEO impact available on a typical content site.

Key Takeaway: A technical SEO audit is not a one-time exercise. The sites that maintain technical excellence run automated checks continuously, not manual crawls once a year. Ranking Lens's technical audit score tracks your 47-point checklist status automatically so you catch regressions before they damage rankings.