This week's problem

You have 50 keywords and you want one chart: which ones are big, which are small, and how they move over a year. Google Trends won't give you that in one go. The Explore page compares up to 8 terms at a time (Google Trends Help; Google's own Trends API page says "the Trends UI only lets you make comparisons between eight terms"). The classic Explore page allowed 5.

So you split the list into 7 requests and paste the results together. The numbers look fine. They are not. Google scales every request on its own 0–100 range (FAQ): the biggest point in that request becomes 100, and everything is rounded to whole numbers. A "40" from request A and a "40" from request B are on different scales.

This issue measures how wrong that gets with 50 keywords, and which fix holds up.

If you want the basic anchor method with a short walkthrough first, we covered it in Compare a long keyword list in Google Trends on one scale. This issue goes further: the 8-term limit, a 50-keyword simulation, and which anchor choice loses keywords.

The fix (step by step)

1. Put one shared term, the anchor, in every request. Its real interest is the same in every request, so any change in its numbers comes from the scaling. That gives you a conversion factor per request: factor = sum(anchor in reference request) / sum(anchor in this request).
2. Fill each request up to the limit: anchor + 7 keywords = 8 terms, so 50 keywords take 7 requests. Keep geo, time range and category identical.
3. Multiply each request by its factor, then rescale so the overall peak is 100.
4. Pick the anchor from the middle of your list, not the top. If you don't know popularity yet, do two passes: pass 1 with any anchor, then sort keywords by the pass-1 estimate, pick the middle one as anchor, and fetch again. Sorting puts the giants in one request and the tiny keywords in another, so a tiny keyword never shares a request with a giant that rounds it down to 0.
5. Flag, don't guess. Warn when the anchor's mean in a request is below 5 (rounding error grows), and skip any request where the anchor is all zeros or the dates don't line up.

The code

No network calls: the bottom half generates synthetic data (not real Google Trends values) where we know the truth, then imitates a Trends request (own peak = 100, rounded to integers). That lets us measure each strategy's error exactly. rescale_groups is the part you'd reuse on real data: one DataFrame per request, indexed by date.

# issue01.py - Data Plumbing Weekly #1. Needs: pip install pandas numpy
import numpy as np
import pandas as pd

LOW_ANCHOR_MEAN = 5.0  # below this, integer rounding error gets amplified


def make_groups(keywords, anchor, terms_per_request=8):
    """Anchor + up to (terms_per_request - 1) other keywords per request."""
    others = [k for k in keywords if k != anchor]
    size = terms_per_request - 1
    return [[anchor] + others[i:i + size] for i in range(0, len(others), size)]


def rescale_groups(groups, anchor):
    """One DataFrame per request (index=dates, columns=terms, 0-100 ints).
    Returns (combined on one 0-100 scale, factors, warnings)."""
    warnings, factors, scaled = [], {}, []
    ref_idx = next((i for i, g in enumerate(groups) if g[anchor].sum() > 0), None)
    if ref_idx is None:
        raise ValueError("anchor is zero in every group; cannot rescale")
    ref = groups[ref_idx]
    for i, g in enumerate(groups):
        if not g.index.equals(ref.index):
            warnings.append(f"group {i}: date axis differs; skipped")
            continue
        s = float(g[anchor].sum())
        if s == 0:
            warnings.append(f"group {i}: anchor all zeros; skipped")
            continue
        if s / len(g) < LOW_ANCHOR_MEAN:
            warnings.append(f"group {i}: anchor mean {s / len(g):.2f} < 5")
        factors[i] = float(ref[anchor].sum()) / s
        cols = g.columns if i == ref_idx else g.columns.drop(anchor)
        scaled.append(g[cols] * factors[i])
    combined = pd.concat(scaled, axis=1)
    peak = float(combined.to_numpy().max())
    return (combined * 100 / peak).round(2), factors, warnings


# ---------- simulation: synthetic data, NOT real Google Trends values ----------
def make_truth(seed, n_kw=50, n_weeks=52):
    rng = np.random.default_rng(seed)
    t = np.arange(n_weeks)
    level = np.exp(rng.normal(np.log(200), 1.3, n_kw))  # ~3 orders of magnitude
    data = {
        f"kw{k:02d}": level[k]
        * (1 + 0.25 * np.sin(2 * np.pi * (t / 52 + rng.uniform())))
        * (1 + rng.uniform(-0.3, 0.3) * t / n_weeks)
        * rng.lognormal(0, 0.08, n_weeks)
        for k in range(n_kw)
    }
    return pd.DataFrame(data, index=pd.date_range("2025-10-19", periods=n_weeks, freq="W"))


def fake_trends(truth, terms):
    """Model of one request: its own peak = 100, integers."""
    block = truth[terms]
    return (block / block.to_numpy().max() * 100).round().astype(int)


def anchored(truth, anchor, keywords=None):
    groups = make_groups(keywords or list(truth.columns), anchor)
    combined, _, warnings = rescale_groups([fake_trends(truth, g) for g in groups], anchor)
    return combined, len(groups)


def naive(truth, per_request=8):
    kws = list(truth.columns)
    chunks = [kws[i:i + per_request] for i in range(0, len(kws), per_request)]
    return pd.concat([fake_trends(truth, c) for c in chunks], axis=1), len(chunks)


def score(est, truth):
    true100 = truth / truth.to_numpy().max() * 100
    rel = (est.mean() - true100.mean()).abs() / true100.mean()
    return rel, int((est.sum() == 0).sum())


rows = []
for seed in range(1, 21):
    truth = make_truth(seed)
    by_size = truth.mean().sort_values()
    first, median, biggest = truth.columns[0], by_size.index[25], by_size.index[-1]
    pass1, n1 = anchored(truth, first)
    order = pass1.mean().sort_values(ascending=False).index.tolist()
    runs = {
        "naive concat": naive(truth),
        "anchor = biggest": anchored(truth, biggest),
        "anchor = first (arbitrary)": (pass1, n1),
        "anchor = median": anchored(truth, median),
        "two-pass sorted": (anchored(truth, order[25], order)[0], 2 * n1),
    }
    for name, (est, n) in runs.items():
        rel, lost = score(est, truth)
        rows.append({"strategy": name, "requests": n, "median_err_%": rel.median() * 100,
                     "worst_err_%": rel.max() * 100, "kw_err_>10%": (rel > 0.10).sum(),
                     "kw_lost": lost})

summary = pd.DataFrame(rows).groupby("strategy", sort=False).agg(
    requests=("requests", "first"),
    median_err=("median_err_%", "median"),
    worst_err=("worst_err_%", "max"),
    kw_over_10pct=("kw_err_>10%", "mean"),
    kw_lost=("kw_lost", "mean"),
)
print("20 synthetic lists x 50 keywords x 52 weeks, 8 terms per request")
print(summary.round(2).to_string())

Output (Python 3.13, pandas 3.0.6, run before publishing):

20 synthetic lists x 50 keywords x 52 weeks, 8 terms per request
                            requests  median_err  worst_err  kw_over_10pct  kw_lost
strategy                                                                           
naive concat                       7      178.74   12137.39          42.85     0.95
anchor = biggest                   7        0.64     100.00           6.70     2.05
anchor = first (arbitrary)         7        0.88     100.00           8.70     0.90
anchor = median                    7        0.56     100.00           4.25     0.80
two-pass sorted                   14        0.56      43.37           2.25     0.00

How to read it: errors are on each keyword's average level, in percent. kw_over_10pct and kw_lost (keywords that came out as all zeros) are averages per list of 50.

• Naive concatenation is useless for comparison: about 43 of 50 keywords end up more than 10% off, typically by about 180%.
• Any anchor fixes the typical keyword (median error under 1%). The differences are in the tail: small keywords that share a request with a big one get rounded to 0 and are lost (worst error 100%).
• Using your biggest keyword as the anchor is the worst anchored option (most keywords lost, about 2 per list). It's also the most tempting choice.
• Two-pass sorted lost no keywords in any of the 20 lists and had the fewest keywords off by more than 10%, at the cost of twice the requests (14 instead of 7).

3 quick tips

• Drop the partial last point before comparing. In the data the Trends site returns, the newest period is often flagged isPartial (incomplete), and its value can change on a later fetch.
• Fetch all requests close together in time. Trends is computed from a sample, and Google adds privacy noise that is most noticeable on low-interest queries (FAQ). Two fetches of the same term at different times can differ.
• 0 can mean "too small", not "no searches." Google says low-volume searches appear as 0 (FAQ). Don't build "keyword X has zero demand" conclusions on it.

What we tested (and what didn't work)

• Tested: the code above, on the box, before publishing; plus five plain-assert tests (rescaling recovers the true scale exactly when there is no rounding; groups never exceed the term limit; zero-anchor and mismatched-date groups are skipped, not guessed; the global peak is 100). All passed.
• Not tested here: real Google Trends data. The simulation models only two documented behaviours (per-request 0–100 scaling and integer rounding). It leaves out sampling noise, privacy noise and low-volume filtering, so real-world errors will be larger than these numbers. Treat the table as "which strategy breaks first", not as an accuracy promise.
• What didn't work: an anchor from the bottom of the list (we tried the 6th smallest of 50 in an earlier run of the same simulation). Its integer values are so small that the rounding error goes into every factor: in that run, 43–49 of 50 keywords were more than 10% off.

Tool of the week (not ours)

The official Google Trends API (alpha). Google says it provides "consistently scaled data" and "makes it easier to compare dozens of terms", which removes the need for anchors. When we read the page on 2026-10-17, access was through an alpha-tester application only (announcement, July 2025). If you get in, use it instead of stitching.

From our workshop (disclosure: we built this and sell it)

We built Google Trends Bulk Compare, an Apify Actor that runs the single-anchor method for you: it splits your list into an anchor plus up to 4 keywords per request, rescales with sum-based factors, sets the global peak to 100, and saves the factors and low-anchor warnings with each run. It does not do the two-pass sorting above; choose a mid-popularity anchor yourself. Price: USD 0.003 per keyword that returns data; keywords that fail are free. We have tested it with up to 30 keywords per run, not 50, so treat a 50-keyword run as untested. You don't need it: the code above works with any fetcher.

How this was made: drafted with AI assistance and checked by our team before publishing. The code was run before publishing, and the output above is copied from that run. All numbers come from synthetic data unless a source is linked.

This post explains why, and walks through the anchor-term rescaling method that fixes it, with a small pandas implementation you can run offline.

Why separate requests aren't comparable

Google Trends doesn't return search volume. It returns a relative index: within a single request, the highest point across all terms in that request becomes 100, and everything else is scaled against it and rounded to an integer.

So "60" in request A and "60" in request B are on different scales. If request A contains a giant term, everything else in A gets squashed toward zero. If request B contains only small terms, one of them gets stretched to 100. Concatenating the two is like merging two charts that use different y-axes without telling anyone.

The fix: a shared anchor term

The trick is to put the same term (the anchor) in every request. Because the anchor's real interest is the same no matter which request it's in, any difference in its numbers between requests comes from the different scaling. That gives you a conversion factor.

The procedure:

1. Pick an anchor term.

2. Split your other keywords into groups small enough that each group plus the anchor fits in one request (groups of 4 plus the anchor, 5 terms per request, work under both the old and the new limit).

3. Fetch each group with identical geo, timeframe, and category.

4. Pick a reference group (e.g. the first one where the anchor isn't all zeros).

5. For each group g, compute a factor:

   factor_g = sum(anchor in reference group) / sum(anchor in group g)
   

6. Multiply every series in group g by factor_g. Report the anchor once, from the reference group.

7. Renormalize globally: divide everything by the overall max and multiply by 100, so the biggest point across all keywords is 100 again.

Why sums instead of per-point ratios?

You could compute a ratio for each date and average them, but each data point is an integer from 0 to 100. A point where the anchor reads 2 could really be anywhere from 1.5 to 2.5. That's ±25% on one point. Summing the whole series first and dividing once spreads that rounding error across the full timeline. Points where the anchor reads 0 would also break a per-point ratio (division by zero), but they're harmless in a sum.

A runnable implementation in pandas

The code below has no network calls. The DataFrames are made-up example data so you can see the math work. In real use you'd fill them with whatever your fetcher returns (one DataFrame per request, indexed by date).

import pandas as pd

LOW_ANCHOR_MEAN = 5  # below this, integer rounding error gets amplified


def rescale_groups(groups: list[pd.DataFrame], anchor: str):
    """Put several Google Trends requests (each with its own 0-100 scale)
    onto one shared 0-100 scale using an anchor term present in every group.

    Returns (combined DataFrame, factors dict, warnings list).
    """
    warnings, factors, scaled = [], {}, []

    # Reference = first group where the anchor has any signal at all.
    ref_idx = next((i for i, g in enumerate(groups) if g[anchor].sum() > 0), None)
    if ref_idx is None:
        raise ValueError("anchor is zero in every group; cannot rescale")
    ref = groups[ref_idx]
    ref_sum = ref[anchor].sum()

    for i, g in enumerate(groups):
        # Groups must share the exact same date axis to be comparable.
        if not g.index.equals(ref.index):
            warnings.append(f"group {i}: date axis differs from reference; skipped")
            continue
        s = g[anchor].sum()
        if s == 0:
            warnings.append(f"group {i}: anchor is all zeros; skipped")
            continue
        mean = s / len(g)
        if mean < LOW_ANCHOR_MEAN:
            warnings.append(
                f"group {i}: anchor mean {mean:.2f} < {LOW_ANCHOR_MEAN}; "
                "rounding error is amplified, consider a more popular anchor"
            )
        factors[i] = float(ref_sum / s)
        cols = g.columns if i == ref_idx else g.columns.drop(anchor)
        scaled.append(g[cols] * factors[i])

    combined = pd.concat(scaled, axis=1)
    peak = combined.to_numpy().max()
    combined = (combined * (100 / peak if peak > 0 else 0)).round(2)
    return combined, factors, warnings


# ---- EXAMPLE DATA (made up for illustration, not real Google Trends values) ----
dates = pd.date_range("2026-01-04", periods=6, freq="W")

# Request 1: anchor + two big terms. "term_a" sets this request's 100.
group0 = pd.DataFrame(
    {"anchor": [20, 22, 21, 19, 23, 20],
     "term_a": [80, 90, 100, 85, 95, 88],
     "term_b": [40, 42, 45, 41, 39, 44]},
    index=dates,
)

# Request 2: anchor + two small terms. With no giant term here,
# the anchor itself is stretched toward 100.
group1 = pd.DataFrame(
    {"anchor": [87, 96, 91, 83, 100, 87],
     "term_c": [30, 35, 33, 31, 36, 34],
     "term_d": [10, 12, 11, 9, 13, 10]},
    index=dates,
)

combined, factors, warnings = rescale_groups([group0, group1], anchor="anchor")

print("factors:", {k: round(v, 4) for k, v in factors.items()})
print("warnings:", warnings or "none")
print(combined)

Running it (Python 3, pandas) prints:

factors: {0: 1.0, 1: 0.2298}
warnings: none
            anchor  term_a  term_b  term_c  term_d
2026-01-04    20.0    80.0    40.0    6.89    2.30
2026-01-11    22.0    90.0    42.0    8.04    2.76
2026-01-18    21.0   100.0    45.0    7.58    2.53
2026-01-25    19.0    85.0    41.0    7.12    2.07
2026-02-01    23.0    95.0    39.0    8.27    2.99
2026-02-08    20.0    88.0    44.0    7.81    2.30

In raw request 2, term_c peaked at 36 and looked like it might be close to term_b. Once it's on the shared scale you can see it's really around 7-8, a small fraction of term_b. That's exactly the kind of wrong conclusion that naive concatenation produces.

Note the output has decimals. That's expected: rescaled values are estimates, not Google's native integers.

Pitfalls (the part that actually bites)

1. An anchor with too little volume

This is the big one. If your anchor averages, say, 3 in a group, each point's ±0.5 rounding is a huge relative error, and the factor inherits it. Every keyword in that group is then multiplied by a noisy factor. If the anchor rounds to 0 across a whole group, you can't rescale that group at all.

In the implementation I base this on, a group whose anchor mean is below 5 triggers a warning, and a group whose anchor is all zeros is marked as failed instead of guessed. I'd recommend both behaviors: flag it, don't invent numbers.

2. Choosing the anchor

A good anchor is:

• Steady: no huge spikes, so it isn't the 100 in some groups and tiny in others for reasons unrelated to scaling.
• Fairly popular in your geo and timeframe, so it stays well above zero in every group.
• Not dwarfed by your biggest keyword. If one keyword is 50× the anchor, the anchor gets crushed toward 0 in that keyword's group.

A practical check: after a run, look at the anchor's mean per group and at the factors. Factors far from 1 combined with a low anchor mean are where the error lives.

3. Rounding error is real, even when everything is done right

Rescaling can't recover precision that Google rounded away. Low-volume keywords in a group with a large factor carry roughly ±0.5 × factor of uncertainty per point. Treat small differences between small keywords with suspicion.

4. Sampling noise

Google Trends is computed from a sample of searches, and recent data gets revised (the latest point is often flagged as partial). Two requests made at different times can return slightly different values for the same term. Fetch all your groups in one session, close together in time, and consider dropping partial points before comparing.

5. Timeframe, geo, and category must match exactly

The anchor only links groups if every request uses the same geo, timeframe, and category. Watch out for the date axis too: if one group comes back with a different set of dates, don't try to line it up. Exclude it. The implementation above (and the one I based it on) requires an identical date axis and skips any group that doesn't match.

6. Rate limits and HTTP 429

If you're using pytrends, you will meet TooManyRequestsError / HTTP 429 sooner or later, especially in loops. What helps:

• Send requests one at a time with a delay between them (seconds, not milliseconds).
• Retry only on 429 / 5xx / timeouts, with exponential backoff plus jitter, and honor Retry-After if present.
• Don't retry a 400: that's a bad request and retrying won't fix it.
• Cap the number of retries, and record a failed keyword as failed rather than silently dropping or zero-filling it.

7. Browser scrapers and UI changes

Google has redesigned the Trends pages more than once. Any scraper that drives a browser and reads values out of the rendered page depends on that page's layout and selectors, so a redesign is exactly the kind of change that breaks it. The approach I've had better luck with is to skip the DOM entirely and use the JSON endpoints that the Trends UI itself calls (the same data the "download CSV" button gives you). These are still unofficial and can change without notice, so build in validation either way.

A quick validation you should run

Before you trust a pipeline, compare it to ground truth: pick 5 terms, fetch them in one direct request (Google's own scale), then fetch them again split into anchored groups and rescale. The difference between the two tells you how much error your anchor choice introduces. With a weak anchor you'll see small but non-zero errors on the low-volume terms. That's the rounding issue from pitfall 1, and it's the best way I know to pick an anchor with data instead of guessing.

Wrap-up

• Each Google Trends request is scaled to its own 0-100 peak, so separate requests can't be compared directly.
• Put a shared anchor in every request, scale each group by sum(anchor_ref) / sum(anchor_group), then renormalize so the global max is 100.
• Most of the error comes from the anchor: choose a steady, popular one, warn when its mean is low, and never rescale a group where it's all zeros.
• Keep geo, timeframe, category, and date axis identical across groups, fetch close together in time, and back off politely on 429s.

Disclosure: we built a hosted version of this

We (Raspberry Rondel) make Google Trends Bulk Compare, an Apify Actor that packages this method: groups of up to 4 keywords plus the anchor, sum-based factors and a global renormalisation, with low-anchor groups flagged and the factors and warnings saved with each run.

• Pricing: USD 0.003 per keyword that returns data. Failed or empty keywords are free, and there is no start fee.
• Try it: the Store page opens with a prefilled compare run: 6 keywords (coffee, tea, matcha, bubble tea, espresso, latte; Taiwan, past 12 months) on one shared scale, for at most USD 0.018. Running it needs an Apify account.
• What we have tested: up to 30 keywords per run; we haven't tested beyond that. In our runs on the Apify platform, per-keyword mode took 12–14 s per keyword (30 keywords took about 7 minutes); compare mode sends fewer requests. In our internal test the values matched the Google Trends CSV-export endpoint for the same requests (424/424 points), fetched by our own runner. We haven't yet compared against a CSV downloaded by hand in a normal browser session.
• Calling it from code: for long lists, start the run with the Apify client's .call() (or an async run plus polling). Apify's synchronous "run and get dataset items" endpoint returns HTTP 408 once a run passes 300 s.

You don't need the Actor to use the method. Everything above works with any fetcher and the pandas function in this post.

This article was written with AI assistance. The pandas example was run before publishing, and the output shown is copied from that run.

1. Tofu boxes. Chinese, Japanese, Korean and emoji come out as empty rectangles (□□□), because the server or container has no font for them.
2. Half-empty full-page shots. The page is the right height, but images further down are grey placeholders and "load more" content never appears, because nothing scrolled the page.

There's also a subtler third one: the text renders, but Traditional Chinese is drawn with the Japanese font, so some characters and punctuation have the wrong regional shape.

This post reproduces all three on a small local test page and fixes them with Playwright for Python. Everything below was run, and the output is copied from that run.

Test setup: Linux, Python 3.13, Playwright for Python 1.64.0 with its bundled Chromium headless shell (156.0.8078.4), and the Debian/Ubuntu packages fonts-noto-cjk and fonts-noto-color-emoji.

The test page

site/index.html has a title with an emoji, one paragraph each in Traditional Chinese, Simplified Chinese, Japanese and Korean, an emoji line (including a skin-tone modifier, a flag and a ZWJ family), and then:

• two images with native loading="lazy",
• two images that use the common JavaScript pattern (data-src swapped in by an IntersectionObserver),
• an "infinite scroll" sentinel that appends a new section (with an image) three times as you reach the bottom.

The <html> tag says lang="zh-Hant", and the Simplified Chinese paragraph says lang="zh-Hans". Some real sites use script subtags like these.

<html lang="zh-Hant">
...
<p id="zh-Hant">繁體中文：臺灣的「鐵路便當」很好吃。數字 0123456789，標點「」、。！</p>
<p id="zh-Hans" lang="zh-Hans">简体中文：这是一个用于测试网页截图的页面。</p>
<p id="ja" lang="ja">日本語：ウェブページのスクリーンショットをテストします。</p>
<p id="ko" lang="ko">한국어: 웹 페이지 스크린샷 테스트입니다.</p>
<p id="emoji">Emoji: 😀 🎉 👍🏽 🇹🇼 👨‍👩‍👧 ❤️</p>
...
<img class="js-lazy" data-src="img/3.svg" alt="js lazy image 3">

The full page, the images and a script that runs every case are in the companion folder (site/, run_demo.sh).

The script

shoot.py does four things in this order:

1. Scrolls down in 600 px steps with a 400 ms pause, until the page height stops growing for two rounds at the bottom (or a time/height limit is hit), then scrolls back to the top so sticky headers sit where a visitor would see them.
2. Waits for images that are still loading (with a timeout) and for document.fonts.ready.
3. Reports fonts. It asks Chromium, through the DevTools Protocol method CSS.getPlatformFontsForNode, which installed font actually drew each element you list, and how many glyphs came from each.
4. Captures with full_page=True and animations="disabled".

"""Full-page screenshot with headless Chromium (Playwright for Python).

- scrolls the page first so lazy-loaded images and infinite-scroll content load
- waits for images and web fonts before capturing
- reports which real font Chromium used for each element you name (via CDP),
  so you can catch tofu boxes before you look at the PNG

Usage:
  python shoot.py URL out.png [--no-scroll] [--locale zh-TW] [--fallback-lang zh_TW.UTF-8]
                  [--check "#id1,#id2"]
"""
import argparse
import os
import time

from playwright.sync_api import sync_playwright

SCROLL_JS = """
async ({ step, pauseMs, maxSecs, maxHeight }) => {
  const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
  const height = () => Math.max(document.documentElement.scrollHeight,
                                document.body ? document.body.scrollHeight : 0);
  const start = Date.now();
  let y = 0, lastHeight = -1, stableRounds = 0;
  while (Date.now() - start < maxSecs * 1000) {
    y = Math.min(y + step, height());
    window.scrollTo(0, y);
    await sleep(pauseMs);
    const h = height();
    if (y + window.innerHeight >= h) {          // reached the bottom
      stableRounds = (h === lastHeight) ? stableRounds + 1 : 0;
      if (stableRounds >= 2 || h >= maxHeight) break;   // nothing new loaded
    }
    lastHeight = h;
  }
  window.scrollTo(0, 0);                         // sticky headers back in place
  return { height: height(), seconds: (Date.now() - start) / 1000 };
}
"""

WAIT_ASSETS_JS = """
async (timeoutMs) => {
  const imgs = [...document.images].filter((i) => !i.complete);
  const each = imgs.map((i) => new Promise((r) => { i.onload = i.onerror = r; }));
  await Promise.race([Promise.all(each), new Promise((r) => setTimeout(r, timeoutMs))]);
  if (document.fonts) await document.fonts.ready;
}
"""

IMG_STATS_JS = """
() => {
  const imgs = [...document.images];
  const loaded = imgs.filter((i) => i.complete && i.naturalWidth > 0).length;
  return { images: imgs.length, loaded };
}
"""


def platform_fonts(page, selectors):
    """Ask Chromium which installed fonts it actually used to draw each element."""
    cdp = page.context.new_cdp_session(page)
    cdp.send("DOM.enable")
    cdp.send("CSS.enable")
    root = cdp.send("DOM.getDocument")["root"]["nodeId"]
    report = {}
    for sel in selectors:
        node = cdp.send("DOM.querySelector", {"nodeId": root, "selector": sel})["nodeId"]
        if not node:
            report[sel] = "not found"
            continue
        res = cdp.send("CSS.getPlatformFontsForNode", {"nodeId": node})
        fonts = [f"{f['familyName']} ({f['glyphCount']})" for f in res["fonts"]]
        report[sel] = ", ".join(fonts) or "(no text)"
    return report


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("url")
    ap.add_argument("out")
    ap.add_argument("--no-scroll", action="store_true")
    ap.add_argument("--locale", default="zh-TW")
    ap.add_argument("--width", type=int, default=1280)
    ap.add_argument("--height", type=int, default=800)
    ap.add_argument("--max-height", type=int, default=20000)
    ap.add_argument("--fallback-lang", default="",
                    help="LANG for the browser process, e.g. zh_TW.UTF-8; picks the CJK "
                         "fallback face for text that has no lang= of its own")
    ap.add_argument("--check", default="", help="comma-separated CSS selectors to font-check")
    args = ap.parse_args()

    with sync_playwright() as p:
        env = {**os.environ, "LANG": args.fallback_lang} if args.fallback_lang else None
        browser = p.chromium.launch(env=env)
        context = browser.new_context(
            viewport={"width": args.width, "height": args.height},
            locale=args.locale,          # navigator.language / Accept-Language (not the fallback font)
            device_scale_factor=1,
        )
        page = context.new_page()
        t0 = time.time()
        resp = page.goto(args.url, wait_until="load", timeout=60_000)
        print("status:", resp.status if resp else None)

        if not args.no_scroll:
            info = page.evaluate(SCROLL_JS, {"step": 600, "pauseMs": 400,
                                             "maxSecs": 15, "maxHeight": args.max_height})
            print(f"scrolled: height={info['height']}px in {info['seconds']:.1f}s")
        page.evaluate(WAIT_ASSETS_JS, 10_000)
        print("images:", page.evaluate(IMG_STATS_JS))

        if args.check:
            for sel, fonts in platform_fonts(page, args.check.split(",")).items():
                print(f"font {sel}: {fonts}")

        page.screenshot(path=args.out, full_page=True, animations="disabled", timeout=120_000)
        print(f"saved {args.out} in {time.time() - t0:.1f}s")
        browser.close()


if __name__ == "__main__":
    main()

Install and run:

python3 -m venv .venv && . .venv/bin/activate
pip install playwright==1.64.0
playwright install chromium
python shoot.py https://example.com/ out.png --check "h1,p"

Problem 1: tofu boxes in a slim container

To simulate a slim image without having to build one, the demo gives Chromium a private fonts.conf that only lists one Latin font (DejaVu Sans), via the FONTCONFIG_FILE environment variable:

FONTCONFIG_FILE=$PWD/slim-fonts/fonts.conf \
  python shoot.py "$URL" a-slim-noscroll.png --no-scroll \
  --check "#title,#zh-Hant,#zh-Hans,#ja,#ko,#emoji"

== A: slim fonts, no scrolling ==
status: 200
images: {'images': 4, 'loaded': 2}
font #title: DejaVu Sans (15)
font #zh-Hant: DejaVu Sans (39)
font #zh-Hans: DejaVu Sans (21)
font #ja: DejaVu Sans (28)
font #ko: DejaVu Sans (23)
font #emoji: DejaVu Sans (25)
saved a-slim-noscroll.png in 0.2s

In the screenshot every CJK character is a box, and so is most of the emoji line. The two emoji DejaVu Sans happens to cover (😀 and ❤) come out as black-and-white outlines instead of colour emoji. Note what the font report says: the glyphs are still "counted" against DejaVu Sans, which has no CJK glyphs at all. So the check isn't "is there a font?", it's "is CJK text being drawn by a font that can't contain CJK?" If your Chinese paragraph reports only a Latin family, you have tofu, and you can fail the job before anyone looks at the PNG.

Fix: install CJK and emoji fonts in the image and rebuild the font cache.

RUN apt-get update && apt-get install -y --no-install-recommends \
      fonts-noto-cjk fonts-noto-color-emoji \
    && fc-cache -f && rm -rf /var/lib/apt/lists/*

With the normal system fonts:

== B: CJK + emoji fonts, no scrolling ==
status: 200
images: {'images': 4, 'loaded': 2}
font #title: Arimo (1), Noto Sans CJK JP (13), Noto Color Emoji (1)
font #zh-Hant: Arimo (11), Noto Sans CJK JP (28)
font #zh-Hans: Noto Sans CJK JP (21)
font #ja: Noto Sans CJK JP (28)
font #ko: Arimo (6), NanumGothic (17)
font #emoji: Arimo (12), Noto Color Emoji (6)
saved b-fonts-noscroll.png in 0.3s

No more boxes, and the emoji (including 👍🏽, 🇹🇼 and the ZWJ family) come from Noto Color Emoji. But look at the Chinese lines: both the Traditional and the Simplified paragraph were drawn with Noto Sans CJK JP. That's problem 3, which we'll come back to. First, the missing images.

Problem 2: lazy-loaded content is missing

Run B didn't scroll. The screenshot is 3,207 px tall: the two native lazy images near the top loaded, the two data-src images are grey boxes showing their alt text, and none of the three "infinite scroll" sections exist. full_page=True captures the whole document, but it doesn't scroll the way a visitor does, so IntersectionObserver callbacks for content further down never fire.

With the scroll step:

== C: CJK + emoji fonts, scroll first ==
status: 200
scrolled: height=5358px in 3.6s
images: {'images': 7, 'loaded': 7}
...
saved c-fonts-scroll.png in 4.0s

All 7 images loaded, and the page grew from 3,207 to 5,358 px as the three batches were appended. The loop doesn't stop at the first time it reaches the bottom: it waits until the height has stayed the same for two rounds, because infinite-scroll pages add content a moment after you arrive (the test page waits 300 ms).

Things that matter on real pages:

• Always cap it. Some feeds never end. The script stops after 15 seconds or 20,000 px, whichever comes first. Pick limits that fit your pages.
• Scroll back to the top before capturing, or sticky headers and "back to top" buttons end up in odd places.
• networkidle is not a substitute. Lazy content only starts loading after the scroll, so an idle network before scrolling tells you nothing about it.
• Disable animations (animations="disabled"), so carousels and fade-ins are captured in a stable state.

Problem 3: Chinese drawn with the Japanese font

Chinese, Japanese and Korean share a lot of code points (Han unification), but each region has its own preferred glyph shapes, and punctuation is placed differently: in Traditional Chinese fonts ， and 。 usually sit in the centre of the square, while Japanese fonts put them in the lower left. Noto Sans CJK ships TC, SC, HK, JP and KR versions of the same characters, and Chromium has to pick one.

In run B, Chromium picked JP for both lang="zh-Hant" and lang="zh-Hans". We then tried a few variations on the same setup (runs B and D to G in run_demo.sh):

This was one Chromium build on one Linux font setup, so treat the table as "check yours" rather than a rule, but it shows two useful things. In this setup, script-only tags (zh-Hant, zh-Hans) didn't pick the Chinese faces, while region tags (zh-TW, zh-CN) did. And Playwright's locale option changes navigator.language and Accept-Language, but not the fallback font.

If you own the page, use region tags or name the faces explicitly. site/fixed.html does both:

:lang(zh-TW) { font-family: "Noto Sans CJK TC", "Noto Sans TC", "PingFang TC", "Microsoft JhengHei", sans-serif; }
:lang(zh-CN) { font-family: "Noto Sans CJK SC", "Noto Sans SC", "PingFang SC", "Microsoft YaHei", sans-serif; }
:lang(ja)    { font-family: "Noto Sans CJK JP", "Noto Sans JP", "Hiragino Sans", "Yu Gothic", sans-serif; }
:lang(ko)    { font-family: "Noto Sans CJK KR", "Noto Sans KR", "Apple SD Gothic Neo", "Malgun Gothic", sans-serif; }

== D: fixed page (lang=zh-TW / zh-CN + :lang() CSS), scroll first ==
status: 200
scrolled: height=5358px in 3.6s
images: {'images': 7, 'loaded': 7}
font #title: Noto Sans CJK TC (14), Noto Color Emoji (1)
font #zh-Hant: Noto Sans CJK TC (39)
font #zh-Hans: Noto Sans CJK SC (21)
font #ja: Noto Sans CJK JP (28)
font #ko: Noto Sans CJK KR (23)
font #emoji: Noto Sans CJK TC (12), Noto Color Emoji (6)
saved d-fixed-page.png in 3.9s

Each language now gets its own face. (The stack also lists the macOS and Windows names, so real visitors get the right fonts too.)

If you don't own the page and it has no lang attribute, the browser process's LANG decides which face fills in. shoot.py --fallback-lang zh_TW.UTF-8 passes that to chromium.launch(env=...):

== E1: page without lang=, browser started with default LANG ==
font #zh-Hant: Arimo (11), Noto Sans CJK JP (28)
== E2: page without lang=, browser started with LANG=zh_TW.UTF-8 ==
font #zh-Hant: Arimo (11), Noto Sans CJK TC (28)

This only helps text that has no lang of its own. A page that declares lang="zh-Hant" still got JP in our test, and we deliberately don't inject CSS into pages we don't own: for monitoring and archiving, the screenshot should show the page as its authors shipped it.

Checklist

• Install fonts-noto-cjk and fonts-noto-color-emoji (or equivalents) and run fc-cache -f
• Font-check a few CJK elements with CSS.getPlatformFontsForNode; fail the job if CJK text is drawn only by a Latin font
• For your own pages: use zh-TW / zh-CN / zh-HK region tags or :lang() font stacks
• For pages without lang: start the browser with the LANG of the region you expect
• Scroll in steps until the height stops growing, with a time and height cap, then scroll back to the top
• Wait for pending images and document.fonts.ready; capture with animations="disabled"
• Respect robots.txt and the site's terms, and screenshot for QA, monitoring or archiving, not to republish other people's content

Disclosure: we built a hosted version

We (Raspberry Rondel) make Website Screenshot, an Apify Actor that takes these screenshots for a list of URLs, for visual QA, page monitoring and archiving. It ships with CJK and emoji fonts, captures full-page or viewport shots as PNG or JPEG, and respects robots.txt by default.

Pricing: USD 0.004 per screenshot stored for a page that loaded (HTTP status below 400). Error pages, blocked pages, failed loads and duplicates are free. Each run also has a start event of USD 0.0005 per GB of run memory, charged once per run (about USD 0.002 at the default 4 GB).

You're responsible for how you use screenshots of pages you don't own. Everything in this post works without the Actor.

This article was written with AI assistance. The code was run and the output above is copied from that run.

This post shows how to automate that responsibly: follow robots.txt, read only a few pages per site, collect only public business contact info, and keep the source of every value. A runnable Python example is included.

1. Decide what you are collecting (and what you are not)

The target is company contact points: role-based addresses (info@, sales@), main lines, fax and toll-free numbers, and the company's own Facebook page and LINE official account. A named employee's email or a sole trader's mobile is personal data under GDPR and Taiwan's PDPA, even when it's on a public page.

• Minimise. Keep only the columns you'll use.
• Keep provenance. Store the URL each value came from, so you can answer "where did you get my address?" and re-check stale data.

2. Crawl politely: robots.txt and Crawl-delay

Before you fetch anything, read /robots.txt and follow it (RFC 9309):

• Skip every path that is Disallowed for your user agent.
• Honour Crawl-delay if it's set. urllib.robotparser exposes it as crawl_delay(). Without it, still wait at least a second per host.
• If robots.txt returns a 5xx error, don't crawl that site. RFC 9309 says to treat a server error as "complete disallow". A 404 means there are no rules.
• Send an honest User-Agent that says who you are. Don't log in, don't rotate proxies to look like someone else, and stop if you hit a CAPTCHA or bot challenge. A site that blocks you has answered your question.

3. Find the contact and about pages

Contact details usually sit in the homepage header or footer, a contact page and an about page, so you don't need a full crawl. Fetch the homepage, pick same-site links whose URL or anchor text matches contact, about, 聯絡, 關於 or お問い合わせ, and visit a few. About four pages per domain covers most small company sites. If nothing matches, try /contact and /about directly.

4. Extract emails, phones, LINE and social links

Emails. Take mailto: links first, then scan visible text, and filter obvious junk: asset names (banner@2x.png) and placeholders (name@domain.com). If a site deliberately obfuscates its address, respect that and don't decode it.

Phones. Regexes alone will match dates, order numbers and prices. Two rules help a lot:

1. Require context. Only parse a phone number from a tel: link or a line that has a label nearby: Tel, Phone, 電話, 客服, 傳真, Fax.
2. Let libphonenumber do the parsing. The phonenumbers package (a Python port of Google's libphonenumber) understands Taiwanese formats such as (02)2345-6789, 02-2345-6789, +886 2 2345 6789 and 0800-000-123. It normalises all of them to E.164 (+886223456789), which is what makes de-duplication work.

In Taiwan, capture extensions (分機 123, 轉 123) separately, and strip the 8-digit tax ID (統一編號) that often sits next to the number in the footer.

LINE. Businesses in Taiwan and Japan often publish a LINE official account instead of an email. Look for line.me/R/ti/p/@account, lin.ee short links and page.line.me.

Social profiles. Keep the company's own profiles and drop share buttons (/intent/tweet, /sharer).

5. De-duplicate with provenance

The same number often appears on three pages in three formats. Key phones by E.164 (plus extension) and emails by their lower-cased form, and keep a list of source URLs per key: one record per domain, with an audit trail.

6. A runnable example

The script below puts sections 2–5 together using only the standard library plus phonenumbers. To test it safely, I ran it against a fictional company site served from my own machine. All addresses use example.com, and the numbers are made up. Don't point a test script at real third-party sites.

The test site has a robots.txt that blocks /private/ and sets a crawl delay:

User-agent: *
Disallow: /private/
Crawl-delay: 1

The homepage links to an about page, a contact page and a page under /private/. Its footer puts a phone number and a tax ID on the same line:

<!doctype html><html lang="zh-Hant"><head><meta charset="utf-8"><title>Example Trading Co.</title></head>
<body>
<header><a href="/">Home</a> <a href="/products.html">Products</a> <a href="/about-us.html">關於我們</a> <a href="/contact.html">聯絡我們</a></header>
<p>Fictional demo company.</p>
<img src="/img/logo@2x.png" alt="logo">
<a href="https://twitter.com/intent/tweet?url=https://example.com">Share on X</a>
<footer>
  <a href="https://www.facebook.com/exampletrading">Facebook</a>
  <a href="https://lin.ee/AbCdEfG">加入 LINE 好友</a>
  <a href="/private/contact-staff.html">Staff only</a>
  電話 (02)2345-6789｜統一編號 12345678
</footer>
</body></html>

The contact page mixes contacts with things that must not be extracted (an order number, a date, a placeholder email, an image name):

<!doctype html><html lang="zh-Hant"><head><meta charset="utf-8"><title>聯絡我們</title></head>
<body>
<h1>聯絡我們</h1>
<p>業務洽詢：<a href="mailto:sales@example.com">sales@example.com</a></p>
<p>客服信箱：info@example.com</p>
<p>電話：<a href="tel:+886223456789">(02)2345-6789</a> 分機 123</p>
<p>傳真：02-2345-6790</p>
<p>免付費客服：0800-000-123</p>
<p>Order no. 2026101700123, updated 2026-10-17</p>
<p>Example form text: name@domain.com (image: banner@2x.png)</p>
<p>LINE 官方帳號：<a href="https://line.me/R/ti/p/@exampletrading">@exampletrading</a></p>
<footer><a href="https://www.facebook.com/exampletrading">Facebook</a></footer>
</body></html>

The extractor, extract_contacts.py (pip install phonenumbers):

"""Polite public-business-contact extractor (tutorial demo).

Reads a site's robots.txt, honours Disallow and Crawl-delay, visits the homepage
plus up to 3 contact/about pages, and returns one de-duplicated record per domain.
Only run it on sites you are allowed to crawl. Demo data is fictional.
"""
import json, re, sys, time, urllib.error, urllib.request, urllib.robotparser
from html.parser import HTMLParser
from urllib.parse import urljoin, urlsplit
import phonenumbers

UA = "ContactListTutorialBot/0.1 (+https://example.com/bot)"
MAX_PAGES = 4
PAGE_HINTS = re.compile(r"contact|about|聯絡|聯繫|關於|お問い合わせ", re.I)
EMAIL = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,24}")
JUNK_EMAIL = re.compile(r"\.(png|jpe?g|gif|svg|webp)$|^(you|name|user|email)@(example|domain)\.", re.I)
PHONE_CONTEXT = re.compile(r"tel|phone|電話|客服|傳真|fax|免付費", re.I)
FAX = re.compile(r"傳真|fax", re.I)
EXT = re.compile(r"^\s*(?:分機|轉|ext\.?|#)\s*(\d{1,6})", re.I)
LINE = re.compile(r"https?://(?:lin\.ee|line\.me|page\.line\.me)/[^\s\"'<>]+", re.I)
TAX_ID = re.compile(r"統一編號\s*[:：]?\s*\d{8}")
SHARE = re.compile(r"/(?:intent|share|sharer)", re.I)
SOCIAL = re.compile(r"https?://(?:www\.)?(facebook|instagram|linkedin|x|twitter|youtube)\.com/[^\s\"'<>]+", re.I)


class Page(HTMLParser):
    """Collects visible text (one line per block) and link targets."""
    def __init__(self):
        super().__init__(); self.links, self.lines, self._buf = [], [], []
    def handle_starttag(self, tag, attrs):
        href = dict(attrs).get("href")
        if href: self.links.append(href)
        if tag in ("p", "br", "li", "div", "footer", "header", "h1", "h2", "tr"): self._flush()
    def handle_data(self, data): self._buf.append(data)
    def _flush(self):
        line = " ".join("".join(self._buf).split())
        if line: self.lines.append(line)
        self._buf = []


def fetch(url):
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    with urllib.request.urlopen(req, timeout=10) as r:
        return r.status, r.read().decode("utf-8", "replace")


def load_robots(base):
    rp = urllib.robotparser.RobotFileParser()
    try:
        _, body = fetch(base + "/robots.txt")
        rp.parse(body.splitlines())
    except urllib.error.HTTPError as e:
        if e.code >= 500:
            return None          # server error: treat as "do not crawl"
        rp.parse([])             # 404 etc.: no rules, everything allowed
    return rp


def extract(start_url, region="TW"):
    parts = urlsplit(start_url); base = f"{parts.scheme}://{parts.netloc}"
    record = {"domain": parts.hostname, "status": "ok", "pages": [], "skippedByRobots": [],
              "emails": {}, "phones": {}, "line": {}, "socials": {}, "junkFiltered": 0}
    rp = load_robots(base)
    if rp is None:
        record["status"] = "robots-unavailable"; return record
    delay = rp.crawl_delay(UA) or 1.0           # at least 1 s between requests
    queue, seen = [base + "/"], set()
    while queue and len(record["pages"]) < MAX_PAGES:
        url = queue.pop(0)
        if url in seen: continue
        seen.add(url)
        if not rp.can_fetch(UA, url):
            record["skippedByRobots"].append(url); continue
        if record["pages"]: time.sleep(delay)
        try:
            status, html = fetch(url)
        except urllib.error.URLError as e:
            record["pages"].append({"url": url, "error": str(e)}); continue
        record["pages"].append({"url": url, "http": status})
        page = Page(); page.feed(html); page._flush()
        collect(page, url, region, record)
        if url == base + "/":                    # discover contact/about pages from the homepage
            for href in page.links:
                full = urljoin(url, href).split("#")[0]
                if urlsplit(full).netloc == parts.netloc and PAGE_HINTS.search(href):
                    queue.append(full)
    if not (record["emails"] or record["phones"] or record["line"] or record["socials"]):
        record["status"] = "no-contacts-found"
    for key in ("emails", "phones", "line", "socials"):    # sets -> sorted lists for JSON
        record[key] = {k: sorted(v) if isinstance(v, set) else v for k, v in record[key].items()}
    return record


def add(bucket, key, source):
    bucket.setdefault(key, set()).add(source)


def collect(page, url, region, rec):
    for href in page.links:
        if href.startswith("mailto:"):
            add(rec["emails"], href[7:].split("?")[0].lower(), url)
        elif LINE.match(href):
            add(rec["line"], href, url)
        elif SOCIAL.match(href) and not SHARE.search(href):
            add(rec["socials"], href.rstrip("/"), url)
    for line in page.lines:
        for e in EMAIL.findall(line):
            if not JUNK_EMAIL.search(e):
                add(rec["emails"], e.lower(), url)
            else:
                rec["junkFiltered"] += 1
        line = TAX_ID.sub(" ", line)                # 統一編號 (8-digit tax ID) is not a phone
        if not PHONE_CONTEXT.search(line):
            continue                                # phones need context (Tel, 電話, 傳真 ...)
        for m in phonenumbers.PhoneNumberMatcher(line, region):
            e164 = phonenumbers.format_number(m.number, phonenumbers.PhoneNumberFormat.E164)
            ext = EXT.match(line[m.end:])
            key = e164 + (f";ext={ext.group(1)}" if ext else "")
            entry = rec["phones"].setdefault(key, {"kind": "fax" if FAX.search(line) else "phone", "sources": []})
            if url not in entry["sources"]: entry["sources"].append(url)


if __name__ == "__main__":
    print(json.dumps(extract(sys.argv[1]), ensure_ascii=False, indent=2))

Serve the site and run it:

python3 -m http.server 8000 --bind 127.0.0.1 --directory site &
python3 extract_contacts.py http://127.0.0.1:8000/

Real output (the demo script run_demo.sh picks a free port and appends the server log):

{
  "domain": "127.0.0.1",
  "status": "ok",
  "pages": [
    {
      "url": "http://127.0.0.1:48023/",
      "http": 200
    },
    {
      "url": "http://127.0.0.1:48023/about-us.html",
      "http": 200
    },
    {
      "url": "http://127.0.0.1:48023/contact.html",
      "http": 200
    }
  ],
  "skippedByRobots": [
    "http://127.0.0.1:48023/private/contact-staff.html"
  ],
  "emails": {
    "info@example.com": [
      "http://127.0.0.1:48023/about-us.html",
      "http://127.0.0.1:48023/contact.html"
    ],
    "sales@example.com": [
      "http://127.0.0.1:48023/contact.html"
    ]
  },
  "phones": {
    "+886223456789": {
      "kind": "phone",
      "sources": [
        "http://127.0.0.1:48023/",
        "http://127.0.0.1:48023/about-us.html"
      ]
    },
    "+886223456789;ext=123": {
      "kind": "phone",
      "sources": [
        "http://127.0.0.1:48023/contact.html"
      ]
    },
    "+886223456790": {
      "kind": "fax",
      "sources": [
        "http://127.0.0.1:48023/contact.html"
      ]
    },
    "+886800000123": {
      "kind": "phone",
      "sources": [
        "http://127.0.0.1:48023/contact.html"
      ]
    }
  },
  "line": {
    "https://lin.ee/AbCdEfG": [
      "http://127.0.0.1:48023/"
    ],
    "https://line.me/R/ti/p/@exampletrading": [
      "http://127.0.0.1:48023/contact.html"
    ]
  },
  "socials": {
    "https://www.facebook.com/exampletrading": [
      "http://127.0.0.1:48023/",
      "http://127.0.0.1:48023/contact.html"
    ]
  },
  "junkFiltered": 2
}
--- server access log (shows robots.txt first and the Crawl-delay spacing) ---
[17/Oct/2026 16:31:02] "GET /robots.txt HTTP/1.1" 200 -
[17/Oct/2026 16:31:02] "GET / HTTP/1.1" 200 -
[17/Oct/2026 16:31:03] "GET /about-us.html HTTP/1.1" 200 -
[17/Oct/2026 16:31:04] "GET /contact.html HTTP/1.1" 200 -

What to notice:

• robots.txt was fetched first, and /private/contact-staff.html was skipped even though it matched the "contact" hint.
• Requests are about one second apart, as Crawl-delay: 1 asks.
• (02)2345-6789 and +886 2 2345 6789 became one E.164 key with two sources. The 分機 123 version kept its own key, and the fax line was labelled fax.
• The tax ID, order number, date, placeholder email and image file name produced no contacts (junkFiltered: 2 counts the two email-like strings).
• The share-on-X link was dropped, and both LINE link styles (lin.ee and line.me/R/ti/p/@…) were kept.

It's a teaching script. It reads phones from labelled text (not tel: links), doesn't try /contact when nothing matches, and has no JavaScript rendering, retries or parallelism.

7. GDPR and PDPA: your responsibilities

This section is general information, not legal advice.

• You decide which sites to read and how to use the results, so compliance is your responsibility, not the tool's.
• GDPR (EU/EEA and UK). You need a lawful basis, usually legitimate interest (Art. 6(1)(f)) backed by a documented balancing test. If any personal data ends up in the list, you have to tell those people within a reasonable time (Art. 14) and honour objections (Art. 21). National ePrivacy rules often require prior consent before you send marketing email to individuals.
• Taiwan PDPA (個人資料保護法). Data the person made public themselves, or that was lawfully made public, may be collected, but only for a specific purpose (Art. 19). The notice obligation can be waived for such data (Art. 9). If you use it for marketing, you must offer a way to refuse at first contact and stop as soon as the person objects (Art. 20).
• In practice: prefer role-based addresses, keep source URLs, honour "no solicitation" notices, offer an opt-out in every first message, delete what you don't need, and never build profiles of individuals.

Checklist

• Company contact points only
• Fetch robots.txt first; skip disallowed paths; 5xx means don't crawl
• Crawl-delay (≥1 s per host), honest User-Agent, no logins or proxies, stop at challenges
• Homepage plus 3 contact/about pages per domain
• Phones only with context; normalise to E.164; keep extensions; strip 統一編號
• Filter junk emails; don't decode obfuscated ones
• Keep LINE official account links; drop share buttons
• De-duplicate by normalised key and keep source URLs
• Document lawful basis and purpose; offer an opt-out

Disclosure: we built a hosted version

If you'd rather not maintain a crawler, we built Website Contact Extractor, an Apify Actor that does the steps above for a whole list of company websites, so you don't have to open each contact page or tidy the spreadsheet yourself. You get one row per domain with business emails, E.164 phone numbers (Taiwan formats and 分機 included), social profiles and a LINE column, each with its source page. It obeys robots.txt and Crawl-delay, reads 4 pages per domain by default, doesn't log in or try to work around anti-bot protection, and filters obvious junk. It doesn't find contacts that appear only in images, PDFs or contact forms.

Pricing: Charged per domain with results (USD 0.003), not per page crawled; domains with nothing found are free. Apify also charges its standard start event of USD 0.00005 per run per GB of memory.

Its README sets out your GDPR and PDPA responsibilities in more detail. In our internal test (30 sites, Oct 2026), our re-check of the 378 contact items it returned against their source pages found 0 errors. Results on other sites may differ.

This article was written with AI assistance. The code was run and the output above is copied from that run.

This post covers what goes wrong, why converting to Markdown (not plain text) fixes most of it, how the common open-source libraries compare, and a small Python script you can run locally to see the difference.

Three things plain-text extraction breaks

1. Reading order in two-column layouts. Most extractors work line by line, top to bottom. On a two-column paper, line 1 of the left column is followed by line 1 of the right column. Every chunk then mixes two unrelated passages, and the embedding ends up halfway between them.

2. Tables. A table flattened into text keeps the values and loses the structure. "Q1 Q2 Q3 / 12 15 9" no longer tells the model which number belongs to which column. A Markdown table (| Q1 | Q2 | Q3 |) keeps the header attached to each row, and LLMs read that format well.

3. Headings. Headings are the best chunk boundaries you have, and they're also useful metadata ("this chunk is from 3.2 Limitations"). In plain text they look like any other line. In Markdown they're #/## lines, so a splitter such as LangChain's MarkdownHeaderTextSplitter or a 10-line function can cut on them.

CJK documents add one more problem: justified Chinese text is often stored with a space between every character, and line breaks land mid-sentence. Both get in the way of keyword search and chunk-size math.

The options, briefly

All of these are real, maintained projects. Check licenses yourself before you ship; the ones below are from PyPI/GitHub metadata as of October 2026.

A rough rule of thumb: if your PDFs have a text layer (born-digital reports, papers, government documents), a layout-aware rules approach on pdfminer.six/pdfplumber is light, deterministic and cheap to run. If they're scanned or very visual, you need OCR or an ML pipeline (Docling, Marker, MinerU), and you pay for it in compute.

A runnable example: two columns + a table → Markdown → chunks

We'll generate a small sample PDF so you don't need any files, then compare extract_text() with a ~90-line converter.

python -m venv .venv && source .venv/bin/activate
pip install pdfplumber==0.11.10 fpdf2==2.8.9

make_sample.py builds a page with a title, two columns, and a ruled table:

# make_sample.py - build a small two-column PDF with headings and a ruled table
from fpdf import FPDF

LEFT = ("Retrieval quality depends on chunk boundaries. When one chunk mixes "
        "two sections, its embedding blurs both topics. Splitting at headings "
        "keeps each chunk about one thing.")
RIGHT = ("Tables are worse. Flattened into plain text, a row of values "
         "loses its column headers, so the model cannot tell which number "
         "belongs to which metric.")

pdf = FPDF(format="A4")
pdf.add_page()
pdf.set_font("Helvetica", "B", 20)
pdf.cell(0, 12, "Chunking PDFs for RAG", new_x="LMARGIN", new_y="NEXT")
y = pdf.get_y() + 4
for x, title, body in [(10, "Why boundaries matter", LEFT), (110, "Why tables break", RIGHT)]:
    pdf.set_xy(x, y)
    pdf.set_font("Helvetica", "B", 14)
    pdf.multi_cell(90, 8, title)
    pdf.set_x(x)
    pdf.set_font("Helvetica", "", 10)
    pdf.multi_cell(90, 5, body)
pdf.set_y(150)
pdf.set_font("Helvetica", "B", 14)
pdf.cell(0, 8, "Formats compared", new_x="LMARGIN", new_y="NEXT")
pdf.set_font("Helvetica", "", 10)
with pdf.table(col_widths=(60, 40, 40)) as table:  # fpdf2 draws ruled borders
    for row in [("Output", "Headings", "Tables"), ("Plain text", "lost", "flattened"), ("Markdown", "# / ##", "pipe table")]:
        r = table.row()
        for cell in row:
            r.cell(cell)
pdf.output("sample.pdf")
print("wrote sample.pdf")

pdf_to_md.py does four things: groups words into line fragments (splitting at the column gutter), orders them with a recursive XY-cut (split at a vertical gutter if one exists, otherwise at the widest horizontal gap), marks headings by font size relative to the page's median size, and swaps ruled tables for Markdown tables using pdfplumber's find_tables().

# pdf_to_md.py - minimal text-layer PDF -> Markdown -> heading-aware chunks
import statistics, sys
import pdfplumber

def lines_of(words, gap=12):
    """Group words into line fragments; a wide horizontal gap (a column gutter) starts a new fragment."""
    rows = {}
    for w in words:
        rows.setdefault(round(w["top"]), []).append(w)
    frags = []
    for ws in rows.values():
        ws.sort(key=lambda w: w["x0"])
        cur = [ws[0]]
        for w in ws[1:]:
            if w["x0"] - cur[-1]["x1"] > gap:
                frags.append(cur); cur = []
            cur.append(w)
        frags.append(cur)
    return [{"x0": f[0]["x0"], "x1": f[-1]["x1"], "top": min(w["top"] for w in f),
             "bottom": max(w["bottom"] for w in f), "size": f[0]["size"],
             "text": " ".join(w["text"] for w in f)} for f in frags]

def gaps(intervals):
    intervals = sorted(intervals); out = []; end = intervals[0][1]
    for a, b in intervals[1:]:
        if a > end:
            out.append((end, a))
        end = max(end, b)
    return out

def xy_cut(items, min_gutter=7):
    """Recursive XY-cut: split at a column gutter if there is one, else at the widest horizontal gap."""
    if len(items) <= 1:
        return items
    width = max(i["x1"] for i in items) - min(i["x0"] for i in items)
    cols = [g for g in gaps([(i["x0"], i["x1"]) for i in items]) if g[1] - g[0] >= min_gutter]
    for a, b in sorted(cols, key=lambda g: g[0] - g[1]):          # widest gutter first
        left = [i for i in items if i["x1"] <= a]; right = [i for i in items if i["x0"] >= b]
        if min(max(i["x1"] for i in left) - min(i["x0"] for i in left),
               max(i["x1"] for i in right) - min(i["x0"] for i in right)) > 0.2 * width:
            return xy_cut(left) + xy_cut(right)
    rows = gaps([(i["top"], i["bottom"]) for i in items])
    if not rows:
        return sorted(items, key=lambda i: (i["top"], i["x0"]))
    a, b = max(rows, key=lambda g: g[1] - g[0])
    return xy_cut([i for i in items if i["bottom"] <= a]) + xy_cut([i for i in items if i["top"] >= b])

def page_to_markdown(page):
    tables = [t for t in page.find_tables() if len(t.extract()) >= 2]   # ruled tables only
    def in_table(w):
        return any(t.bbox[0] <= w["x0"] and w["x1"] <= t.bbox[2] and
                   t.bbox[1] <= w["top"] and w["bottom"] <= t.bbox[3] for t in tables)
    words = [w for w in page.extract_words(extra_attrs=["size"]) if not in_table(w)]
    body = statistics.median(w["size"] for w in words)
    items = lines_of(words)
    for t in tables:
        rows = [[(c or "").replace("\n", " ").strip() for c in r] for r in t.extract()]
        md = ["| " + " | ".join(rows[0]) + " |", "|" + "---|" * len(rows[0])]
        md += ["| " + " | ".join(r) + " |" for r in rows[1:]]
        x0, top, x1, bottom = t.bbox
        items.append({"x0": x0, "x1": x1, "top": top, "bottom": bottom, "size": body,
                      "text": "\n".join(md), "table": True})
    out = []
    for it in xy_cut(items):
        if it.get("table"):
            out.append(it["text"])
        elif it["size"] >= body * 1.8:
            out.append("# " + it["text"])
        elif it["size"] >= body * 1.3:
            out.append("## " + it["text"])
        elif out and not out[-1].startswith(("#", "|")):
            out[-1] += " " + it["text"]                  # re-join wrapped lines into one paragraph
        else:
            out.append(it["text"])
    return "\n\n".join(out)

def chunk_by_heading(md):
    chunks, current = [], []
    for block in md.split("\n\n"):
        if block.startswith("#") and current:
            chunks.append("\n\n".join(current)); current = []
        current.append(block)
    return chunks + ["\n\n".join(current)]

if __name__ == "__main__":
    with pdfplumber.open(sys.argv[1] if len(sys.argv) > 1 else "sample.pdf") as pdf:
        page = pdf.pages[0]
        print("=== naive extract_text() ===")
        print(page.extract_text()[:240], "...\n")
        md = page_to_markdown(page)
        print("=== markdown ===")
        print(md, "\n")
        for i, c in enumerate(chunk_by_heading(md)):
            print(f"--- chunk {i}: {c.splitlines()[0]!r} ({len(c)} chars)")

Run it:

python make_sample.py && python pdf_to_md.py

Here's what extract_text() gives you. Look at lines 3-5: each one runs straight across both columns.

Chunking PDFs for RAG
Why boundaries matter Why tables break
Retrieval quality depends on chunk boundaries. When Tables are worse. Flattened into plain text, a row of
one chunk mixes two sections, its embedding blurs both values loses its column headers, so the model cannot
topics. Splitting at headings keeps each chunk about tell which number belongs to which metric.
one thing.
Formats compared
Output Headings Tables
Plain text lost flattened
Markdown # / ## pipe table

And here's the converter output (tested with Python 3.13, pdfplumber 0.11.10, fpdf2 2.8.9):

# Chunking PDFs for RAG

## Why boundaries matter

Retrieval quality depends on chunk boundaries. When one chunk mixes two sections, its embedding blurs both topics. Splitting at headings keeps each chunk about one thing.

## Why tables break

Tables are worse. Flattened into plain text, a row of values loses its column headers, so the model cannot tell which number belongs to which metric.

## Formats compared

| Output | Headings | Tables |
|---|---|---|
| Plain text | lost | flattened |
| Markdown | # / ## | pipe table |

--- chunk 0: '# Chunking PDFs for RAG' (23 chars)
--- chunk 1: '## Why boundaries matter' (196 chars)
--- chunk 2: '## Why tables break' (170 chars)
--- chunk 3: '## Formats compared' (134 chars)

Each column comes out as one paragraph under its own heading, the table keeps its header row, and the chunker splits on headings. In a real pipeline you'd merge tiny chunks (like the title-only chunk 0) into the next one and add the heading path as metadata.

Where this toy version breaks

It works on this page, but real documents will break it quickly:

• Paragraphs: it joins every non-heading line until the next heading. Real code needs to detect paragraph breaks (indents, vertical spacing) and hyphenation.

• Borderless "three-line" academic tables aren't detected by find_tables() with default settings.

• Rotated margin text (arXiv side stamps), headers/footers and page numbers end up in the body.

• Symbol fonts map glyphs into Unicode's Private Use Area, and some fonts give (cid:NN) garbage. You want to detect those pages, not embed them.

• Scanned pages have no text layer at all, and you should flag them rather than index empty strings.

Disclosure: we built a hosted version

We built PDF to Markdown – Text, Tables & Chinese/CJK for LLMs, an Apify Actor that handles the cases above so you don't have to maintain them. Under the hood it uses the same approach as this post, built out further: pdfplumber + pdfminer.six (both MIT), recursive XY-cut reading order, font-size headings, ruled tables to Markdown, re-joined wrapped lines, CJK inter-character space removal, ligature and Symbol-font cleanup, and dropping rotated stamps. It checks each page for garbled glyphs and leaves bad pages out, and it reports scanned/image-only pages instead of returning junk. We tested it on Traditional and Simplified Chinese government reports and two-column papers (ResNet, BERT). It does not do OCR, and borderless tables come out as text.

Pricing is pay-per-use: USD 0.002 per run + USD 0.0005 per converted page. A 10-page PDF costs USD 0.007. Scanned, blank and garbled pages and failed downloads aren't charged. A scheduled change on 2026-10-31 lowers the price for pages 21+ of each document to USD 0.00015.

If you'd rather run it yourself, the script above is a reasonable starting point.

This article was written with AI assistance. The code was run and the output above is copied from that run.