Advanced SEO Analysis: Logs, APIs and Crawl Data

Updated 2026-03-27 · By SPUNK13 · Back to spunk.bet

Most SEO reporting describes what happened. The analysis below tells you why, because it works from data the search engine produced rather than from a third-party estimate. All of it uses sources you already own.

Read Your Server Logs

Log files are the only record of what a crawler actually requested. Pull a month of access logs, filter to verified crawler traffic, and you can answer questions no rank tracker can:

# which URLs Googlebot hits most
grep -i googlebot access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -40

# status code distribution for crawler traffic
grep -i googlebot access.log | awk '{print $9}' | sort | uniq -c

Verify the bot before trusting the user agent: run a reverse DNS lookup on the IP and confirm it resolves into googlebot.com or google.com, then forward-resolve that hostname back to the same IP. Roughly speaking, a large share of traffic claiming to be Googlebot is not.

What you are looking for: crawler budget spent on faceted URLs, paginated archives, or parameterised duplicates; a rising share of 404s and 301 chains; and important pages that have not been requested in weeks. The last one is usually an internal linking problem, not a content problem.

Use the Search Console API, Not the UI

The interface caps you at 1,000 rows and hides the long tail where most of the opportunity is. The Search Analytics API returns up to 25,000 rows per request and lets you pull query and page dimensions together, which is what makes the following analyses possible:

• Striking distance. Queries at average position 8–20 with meaningful impressions. These respond to on-page work faster than anything else.
• High impressions, low CTR. Compare each page's CTR to the median CTR at its position. A page well below the curve usually has a title or snippet problem, not a ranking problem.
• Keyword cannibalisation. Group by query and count distinct URLs ranking for it. Two of your own pages trading places for one query means you should consolidate.

Diff Your Crawl Against Your Index

Crawl the site (Screaming Frog, Sitebulb or a scripted crawler) and compare three sets: URLs in your sitemap, URLs discoverable by crawling internal links, and URLs Search Console reports as indexed. The gaps are diagnostic. In the sitemap but not linked internally means orphaned. Linked but not indexed means quality, duplication or crawl-budget. Indexed but not in the sitemap means you are shipping pages you did not intend to.

Measure INP, Not the Retired Metrics

Interaction to Next Paint replaced First Input Delay as a Core Web Vital in March 2024. The threshold is 200 ms at the 75th percentile of real users. Lab tools cannot measure it properly because it needs actual interactions — use the Chrome UX Report data in Search Console or the web-vitals JavaScript library reporting into your own analytics. The usual culprit is a long task on the main thread during hydration; break work up with scheduler.yield() or defer non-critical hydration.

Internal Links Are a Graph, So Analyse Them Like One

Export your crawl's link edges and compute internal PageRank, or simply count unique internal inlinks per URL. Pages with commercial intent and two inlinks are being starved. The cheapest ranking improvement available to most sites is adding contextual links from established pages to the ones that need authority — and unlike external link building, you control it entirely.

Watch the Things That Silently Break

Set up monitoring for four failure modes rather than checking manually: a noindex tag appearing on a template, robots.txt changing, canonical tags pointing somewhere unexpected, and structured data failing validation after a deploy. Each of these has taken sites out of the index within days, and each is trivially detectable by a scheduled script that fetches a sample of URLs and asserts on the response.