Data Plumbing #2: Turn Google Autocomplete suggestions into keyword clusters

Hi, this is Data Plumbing Weekly: one practical data/automation problem per week, with code you can run.

This week's problem

Google Autocomplete is a cheap source of long-tail keyword ideas: type a seed, read what Google suggests, then repeat with "seed a", "seed b", question words and so on. The list grows fast, and most of it says the same thing in slightly different words: "apify actors" and "apify actor id", "extract emails from website" and "extract email from website extension", the seed itself coming back as the first suggestion.

Two more catches before you build on it. First, Google doesn't document a public API for search Autocomplete suggestions. The endpoint people and tools call (suggestqueries.google.com) is unofficial and undocumented, so it can change or be limited without notice. Second, the suggestions are not a popularity ranking. Google's own Trends FAQ says: "Autocomplete is subject to Google's removal policies as well as algorithmic filtering designed to try to catch policy-violating predictions and not show them. Because of this, Autocomplete should not be taken as always reflecting the most popular search terms related to a topic." (Google Trends Help, FAQ). Autocomplete also gives you no search volume.

So treat the list as raw ideas, not demand data, and shrink it into a few groups you can actually review.

The fix (step by step)

1. Cache the raw responses once. Save {seed: [suggestions]} as JSON with the date. Work on the file, not the live endpoint, so every rerun gives the same numbers and you don't send the same requests again.
2. Normalize: lowercase, trim, collapse repeated spaces, and fold simple plurals (countries → country, keywords → keyword), leaving words ending in -ss/-us/-is alone.
3. Dedupe across seeds, keeping the first raw form you saw so you can still show the original text.
4. Cluster by a shared head term. Drop stopwords (how, to, from, is…), take adjacent word pairs, and give each query the pair that the most queries share. If none of its pairs is shared, fall back to the first content word.
5. Count what collapsed (raw → unique → clusters) and review the clusters, not the raw list.

The code

Standard library only, no network calls. The input is a real fixture: one request per seed to Google's public Autocomplete endpoint, captured on 2026-10-17 (20 English seeds, up to 10 suggestions each).

# issue02.py - Data Plumbing Weekly #2. Python 3 standard library only.
# Reads a cached fixture of real Google Autocomplete responses; no network calls.
import json
import re
from collections import Counter, defaultdict
from pathlib import Path

FIXTURE = Path(__file__).resolve().parents[3] / "autocomplete-check-20261017.json"

STOPWORDS = {"a", "an", "the", "to", "for", "from", "of", "on", "in", "into", "and",
             "or", "vs", "with", "how", "what", "is", "can", "i", "do", "you", "does",
             "are", "it", "any"}


def singular(word):
    """Very simple plural folding: countries -> country, keywords -> keyword.
    Leaves -ss/-us/-is words (class, status, analysis) and short words alone."""
    if len(word) > 4 and word.endswith("ies"):
        return word[:-3] + "y"
    if len(word) > 3 and word.endswith("s") and not word.endswith(("ss", "us", "is")):
        return word[:-1]
    return word


def normalize(text):
    """Lowercase, trim, collapse whitespace, fold simple plurals."""
    words = re.sub(r"\s+", " ", text.strip().lower()).split(" ")
    return " ".join(singular(w) for w in words if w)


def dedupe(fixture):
    """{seed: [suggestion, ...]} -> {normalized: first raw form seen}, plus stats."""
    kept, stats = {}, Counter()
    for seed, suggestions in fixture.items():
        for raw in suggestions:
            stats["raw"] += 1
            key = normalize(raw)
            if key != raw:
                stats["changed_by_normalize"] += 1
            if key == normalize(seed):
                stats["echo_of_seed"] += 1  # Google often returns the seed itself
            if key in kept:
                stats["duplicates_removed"] += 1
                stats.setdefault("dup_examples", []).append((raw, kept[key]))
                continue
            kept[key] = raw
    return kept, stats


def bigrams(query):
    """Adjacent content-word pairs, stopwords removed."""
    words = [w for w in query.split(" ") if w not in STOPWORDS]
    if len(words) == 1:
        return words  # one-word query: the word is its own head term
    return [f"{a} {b}" for a, b in zip(words, words[1:])]


def cluster(queries):
    """Head term = the query's bigram that appears in the most queries
    (ties: the earlier bigram). If no bigram is shared by 2+ queries,
    fall back to the first content word. Same head term -> same cluster."""
    df = Counter(bg for q in queries for bg in set(bigrams(q)))
    clusters = defaultdict(list)
    for q in queries:
        grams = bigrams(q)
        head = max(grams, key=lambda bg: (df[bg], -grams.index(bg)))
        if df[head] < 2:
            head = grams[0].split(" ")[0]
        clusters[head].append(q)
    return dict(sorted(clusters.items(), key=lambda kv: (-len(kv[1]), kv[0])))


if __name__ == "__main__":
    fixture = json.loads(FIXTURE.read_text(encoding="utf-8"))
    kept, stats = dedupe(fixture)
    clusters = cluster(list(kept))
    empty = [s for s, v in fixture.items() if not v]
    print(f"fixture: {FIXTURE.name}, {len(fixture)} seeds, "
          f"{len(empty)} with no suggestions: {empty}")
    print(f"raw suggestions:          {stats['raw']}")
    print(f"  echo of the seed:       {stats['echo_of_seed']}")
    print(f"  changed by normalize:   {stats['changed_by_normalize']}")
    print(f"  duplicates removed:     {stats['duplicates_removed']}")
    for raw, first in stats.get("dup_examples", []):
        print(f"    {raw!r} == {first!r}")
    print(f"unique after normalizing: {len(kept)}")
    print(f"clusters (head terms):    {len(clusters)}")
    singles = sum(1 for v in clusters.values() if len(v) == 1)
    print(f"  of which 1-query:       {singles}")
    print()
    print("top 10 clusters:")
    for head, qs in list(clusters.items())[:10]:
        print(f"  {len(qs):>3}  {head:<22} e.g. {kept[qs[-1]]!r}")
    print()
    print("1-query clusters:", sorted(kept[qs[0]] for qs in clusters.values() if len(qs) == 1))

Output (Python 3.13.5, run before publishing):

fixture: autocomplete-check-20261017.json, 20 seeds, 1 with no suggestions: ['long tail keywords from autocomplete']
raw suggestions:          166
  echo of the seed:       17
  changed by normalize:   53
  duplicates removed:     2
    'google trends compare more than 5' == 'google trends compare more than 5'
    'pdf to markdown python' == 'pdf to markdown python'
unique after normalizing: 164
clusters (head terms):    24
  of which 1-query:       6

top 10 clusters:
   34  google trend           e.g. 'google trends java vs python'
   21  pdf markdown           e.g. 'pdf to markdown python langchain'
   20  audio text             e.g. 'elevenlabs audio to text api'
   10  apify actor            e.g. 'apify actors marketplace'
   10  extract email          e.g. 'extract emails from url'
   10  google autocomplete    e.g. 'google autocomplete api test'
    8  llms.txt               e.g. 'llms.txt checker'
    8  website contact        e.g. 'allwish company website contact number free'
    7  website screenshot     e.g. 'website screenshot history'
    6  transcribe video       e.g. 'can i transcribe a video'

1-query clusters: ['companies online contact', 'extract contact details from website', 'keyword extension tool', 'page screenshot api', 'pdf to text for rag', 'sound to text api']

How to read it:

• 166 raw suggestions from 20 seeds (one seed, "long tail keywords from autocomplete", returned nothing). This is a small, one-request-per-seed sample, not an a–z expansion; a full expansion sends dozens of requests per seed and the same steps apply.
• Dedupe alone removed only 2, both exact repeats across seeds. In this sample, normalization changed 53 strings but produced no extra exact matches.
• Clustering does the real work: 164 unique queries became 24 groups, 6 of them single queries. The biggest group, "google trend" (34 queries), pulls in four seeds about Google Trends.
• 17 of the 166 were just the seed coming back. Drop them before you count "new ideas".

3 quick tips

• Protect brand names from plural folding. Our simple rule turned "elevenlabs" into "elevenlab" and "google trends" into "google trend". That's fine for grouping, but display the raw text (the code keeps it) and keep a short list of words not to fold.
• Validate before you write. Autocomplete tells you a phrase exists, not how many people search it, and Google says it's filtered (FAQ). Check the shortlist against data with real counts (see Tool of the week).
• Record the date and settings with every capture. Google says predictions relate to your location and previous searches (FAQ), and they change over time, so a cache without a date can't be compared with next month's.

What we tested (and what didn't work)

• Tested: the code above on the box, on the cached 2026-10-17 fixture, plus six plain-assert tests (normalization of case, spaces and plurals; non-plurals such as "analysis" and "llms.txt" left alone; cross-seed dedupe counts; stopwords removed from head terms; nothing lost in clustering; one-word fallback). All passed.
• Variant: the same pipeline with plural folding switched off gave 166 → 164 → 26 clusters instead of 24. Folding didn't remove extra duplicates here, but it did join "extract emails…" with "extract email…" and "apify actors…" with "apify actor…".
• What didn't work: our first clustering version used only word pairs. The "llms.txt" queries share no pair ("llms.txt generator", "llms.txt checker"…), so it left 14 single-query clusters (31 clusters in total). The first-word fallback fixed most of them; two "llms.txt file…" queries still form their own group.
• Not tested here: a full a–z or question expansion, other languages, and any live requests. Twenty seeds is a small sample; treat the 166 → 24 ratio as an illustration, not a benchmark.

Tool of the week (not ours)

Google Search Console Performance report, Google's own report for your site in Search results. Its Queries tab shows which search queries your site appeared for, with clicks, impressions, CTR and average position (default view: the past three months). Look up your clusters' head terms there: if a cluster already gets impressions, it is a real search, not just an Autocomplete string.

From our workshop (disclosure: we built this and sell it)

We built Google Autocomplete Keyword Expander, an Apify Actor that runs the expansion for you: seed plus a–z, 0–9, question words, prepositions or your own modifiers, with language (hl) and country (gl) settings and optional recursion (max depth 2). It saves one row per keyword plus a flat CSV. Price: USD 0.002 per keyword that returns suggestions (USD 2.00 per 1,000), no start fee; keywords with no suggestions, or where every query failed, are free. Our largest test was 50 seeds in one run (17,467 suggestions, no throttling); runs larger than 50 seeds are not tested. It removes exact repeats but does not fold plurals, cluster, or give search volume, and it uses the same unofficial endpoint, so Google can change or limit it. You don't need it: the code above works on any cached list.

---

How this was made: drafted with AI assistance and checked by our team before publishing. The code was run before publishing, and the output above is copied from that run. All numbers come from the 2026-10-17 fixture unless a source is linked.