I ran a structural audit on my own site, trying to quantify a vague feeling: do these articles all look the same?
The first measurement said 92%, and I wrote that straight into my conclusions. It was wrong. The error was that I had counted the theme’s template as part of “article structure”. Corrected, the figure is 12%.
Nearly eight times apart, separated by a single line of extraction code. This is about the measurement itself: how to do it, how I got it wrong, and why the second number is not the end of the story either.
Why measure shape rather than length
When content quality is in question the first instinct is to check word counts. I checked: median body around three thousand characters, the shortest still over fifteen hundred, not a single filler post. Length explains nothing.
So try a different angle. If a body of articles all share the same shape — the same section rhythm, a comparison table in the same position, the same kind of ending — both readers and reviewers register “this came out of a template”, even when every word is original. Shape can be quantified, provided you define it as a set of detectable structural features.
I defined nine binary features: presence of a code block, a table, a list, an h3, a blockquote, an image, numbered sections, the word “step”, and an FAQ. Each article then maps to a feature set such as code+list+table, and you study how those sets are distributed.
Where the first measurement went wrong
I began by profiling the source files in my repository, and got an absurd result: not one article contained a “related reading” block.
The reason is simple. Those blocks are not in the source files. A plugin wraps every post at render time — reading-progress bar, difficulty badge, generated table of contents, related links at the end. The source holds only the body; readers and crawlers see the wrapped page.
So I switched to fetching the rendered HTML. That part was right. The next step was not: I used the <article> element to delimit “the article”.
That produced figure at 100% — every post has an image. The number looked entirely reasonable at the time; technical posts have diagrams. So I accepted it, folded it into the “shares the same skeleton” test, and arrived at 92%.
Then I added three new articles with no images in the body at all, re-measured, and figure was still 100%.
Which finally prompted me to look at what that <img> actually was:
<img alt='' src='.../personal-site-community/avatars/user-1/default-v3.png'>The author avatar. The theme renders an author box at the foot of every post, and the avatar sits inside <article>. I had been counting it as article content the whole time.
The correction
The right boundary is not <article> but the container holding only the author’s own markup. On this site it carries a data-article-primary attribute; on another theme it will be called something else, but the principle holds: find the line where template injection stops and authored content begins.
Re-measured against that boundary, the same 56 articles: largest single skeleton drops from 92% to 12%, and figure from 100% to 36%.
The entire difference was the wrapper — and every WordPress site has a wrapper. Judging whether content is templated by including it is like judging content quality by whether the site has a footer.
What the script looks like
The whole measurement is under eighty lines and comes down to three functions. Fetching has to deal with a practical problem: the link is unreliable and returned HTML is frequently truncated, while a truncated response looks exactly like a healthy one.
The test is that a page must end in </html>. Feeding incomplete responses into the statistics produces conclusions like “this article has no table” that are simply artefacts:
def fetch(slug, tries=4):
for _ in range(tries):
r = subprocess.run(["curl", "-s", "--max-time", "100",
f"{BASE}/{slug}/?nc={random.randint(1, 10**9)}"],
capture_output=True, text=True)
if "</html>" in r.stdout: # only accept complete responses
return r.stdout
return NoneThe random query parameter defeats CDN caching. Without it you change the site, re-measure, and get the old page back — a trap I fell into once. The symptom is “I changed the config and the metric did not move”, which is indistinguishable from a metric that genuinely does not work.
Feature extraction is a handful of regexes, each returning a boolean:
def skeleton(h):
f = set()
if '<table' in h: f.add('table')
if re.search(r'<pre|<code', h): f.add('code')
if re.search(r'<(ul|ol)', h): f.add('list')
if re.search(r'<blockquote', h): f.add('quote')
if re.search(r'常见问题|FAQ', h): f.add('faq')
...
return sorted(f)Crude regexes, but precision is not the point here. What matters is applying one ruler to every article; systematic error cancels out in the comparison.
Finding the body boundary
This is the only part requiring thought, and the part I got wrong. The method is look first, then write code: pick an article, print the fragment your extractor returns, and read the whole thing.
What you are checking for is concrete: any navigation links, any author box, any related-posts list, any footer. If any of those appear, the boundary is wrong.
The end marker I settled on is whichever of three positions comes first:
ends = [x.start() for x in [
re.search(r'related|site-experience__next', b),
re.search(r'<footer', b),
re.search(r'</article>', b),
] if x]
body = b[:min(ends)] if ends else bInitially I cut only at the related block. The consequence: articles that switch related off have no such marker, so the body ran on into the page footer and their length came back six times too large. One article went from 22234 characters to 3407 once corrected.
This error and the avatar error are the same species: delimiting a boundary with a marker that is usually present, rather than a set that is necessarily present.
The second number is not the end either
A 12% “largest single skeleton” reads as healthy, but it hides something. Change the lens:
Code blocks appear in 96% of articles, lists in 95%, tables in 91%. All three together appear in 88%.
There are 29 distinct skeleton combinations and a structural entropy of 4.44 bits against a ceiling of 5.81. So the articles do differ — but the differences live in secondary features like whether there is an FAQ or a pull quote. The trunk is stable: a passage of explanation, a code block, a list, a comparison table.
“Largest single combination” is blind to this, because it treats code+list+table and code+list+table+quote as two different skeletons. Taxonomically correct; as a reading experience they are the same thing.
So one metric is not enough. You need at least three at once: the share held by the largest single combination, the individual rate of each high-frequency element, and the co-occurrence rate of those high-frequency elements. The third is the one that exposes the problem.
Doing this on your own site
The whole thing is three steps and needs no special tooling.
First, fetch rendered pages, not source files. The source is missing exactly the injected parts, and injected parts are the most likely to be uniform.
Second, separate authored content from theme chrome. Pick one article, print the fragment your extractor returns, and read it — confirm there is no navigation, no author box, no related-posts list in there. I skipped this step, and it cost me a conclusion that was off by a factor of eight.
Third, judge on co-occurrence, not on the distribution of combinations. Find which elements exceed a ninety percent rate, then compute how often they appear together. A high number there means the trunk is fixed.
One useful by-product: you end up with a list of structural outliers, the articles that do not fit the dominant skeleton. I assumed those were the ones to fix. It is the opposite — those are the ones you need more of.
The third one: the duplicate-detection metric never worked for Chinese
After writing this, the same self-measurement setup produced another error — one that had been running silently for a long time. It belongs here because its failure mode differs from the first two.
Alongside structural uniformity I run cross-article duplicate detection: cut each article into sliding 5-token windows and compute pairwise Jaccard similarity. It consistently reported a site-wide maximum of 0.059, which looked healthy.
The tell came when I ran it on four new articles and two of them were English, yet their closest match was the same Chinese article. An English article matching a Chinese one most closely is not possible.
The cause was tokenisation. The script split with re.findall(r'\w+', text) — correct for English, where spaces separate words, and completely wrong for Chinese: Chinese has no spaces, so an entire run of consecutive characters becomes one token.
t = "更糟的是它会冻结同一个循环上所有正在流式输出的对话"
re.findall(r'\w+', t)
# -> one token, 25 characters longA “5-token window” therefore spanned five complete clauses. For two articles to share one window, five consecutive clauses would have to match character for character — essentially impossible. So the metric returned near zero for any pair of Chinese articles. It was never detecting duplication; it was reliably printing “fine.”
The correct approach tokenises per script: character n-grams for Chinese, word n-grams for Latin text, unioned.
def shingles(t, n=5):
out = set()
for run in re.findall(r'[一-鿿]+', t): # Chinese: by character
for i in range(len(run) - n + 1):
out.add(run[i:i+n])
lat = re.findall(r'[A-Za-z0-9_]+', t.lower()) # Latin: by word
for i in range(len(lat) - n + 1):
out.add(tuple(lat[i:i+n]))
return outRe-measuring the whole site with this, the genuine maximum similarity is 0.0488, between two deployment articles in the same series — a sensible result, and the metric can finally distinguish “related” from “unrelated.” The earlier 0.059 came entirely from English pairs; the Chinese half was never participating.
The conclusion did not change — there really is no duplicated content on the site. But until this point, that conclusion had no evidence behind it.
Supporting evidence: I did once write a duplicate article, and I noticed only because its cover image filename collided with an existing one — while this metric reported it as fine. A duplicate check that a filename collision catches and it does not deserved suspicion from the start.
What the three errors have in common
Taken together the three failed in different ways, with one thing in common: each produced a plausible-looking number, and not one of them raised an error.
- Measuring source files gave “0% contain related reading” — a precise, wrong number.
- Measuring the whole
<article>gave 92% — a precise number that counted the theme’s own shell. - Splitting Chinese with
\w+gave 0.059 — a precise number reflecting only English.
So “the metric runs and the result is in a plausible range” carries very little information. What actually exposed each one was a coincidence that should not have occurred: adding three image-free articles moved the metric not at all; the figure rate was exactly 100%; an English article’s nearest neighbour was Chinese.
The takeaway is procedural: give the measurement script a control input with a known answer. Feed in a text deliberately duplicated from an existing article — detection must report similarity near 1. Feed in something entirely unrelated — it must report near 0. Once both endpoints pass, the numbers in between are worth believing. The cost is trivial and it catches all three classes of error above at once.
On measuring yourself
The part worth remembering is not the method. It is how long the error survived.
92% looked completely reasonable: it matched my intuition, it explained what I was seeing, and it handed me a clear course of action. Its only flaw was being wrong, and I ran no check before using it.
What eventually killed it was not a review. It was an accidental control: I added three articles with no images, and the metric did not move. Had those three articles happened to include diagrams, the error would still be sitting in my conclusions.
So when measuring your own work, the highest-value move is to deliberately construct a sample whose answer you already know, and check whether the metric responds. If it does not move, it is not measuring what you think it is.