After reading Matt Brailsford's post about Umbraco AI Search I wanted to give it a test drive. Unlike typical searches that work by matching keywords, Umbraco AI Search uses semantic search that understands the meaning of the search terms and then looks for content it deems most relevant to what the user is looking for. This is helpful when a person mistypes a word, or is searching for information but doesn't use the "right" keyword. It also alleviates the need for content admins to exhaustively add synonyms, broad keywords, and other metadata to ensure that content is discoverable. 

Breaking SEO

Coming up with a good test was the challenge. The AI semantic search works in conjunction with the keyword search, and I wanted to test the AI search without using any keywords that might influence the results. A fun goal emerged: write a blog post that breaks every SEO best practice in the book. 

Test-Driven Content Writing 

I worked with Ella M to map out a blog post I had been meaning to write. It's about an effect that using AI is having on people—specifically the topic of skill atrophy. It addresses the question "is AI making us dumber?". We created an outline following our normal co-written blog structure: I'd open up with an anecdote and some observations. Ella would chime in with additional perspective. Then, before drafting any content, we worked in reverse, starting with the tests.

1. We generated a list of questions that people might ask, in which we would want this post to come up. 

2. Based on those questions, we generated a list of associated keywords and key phrases—with the intention to avoid all use of them.

3. Then we developed a screener to review our post and flag any uses of those key words or phrases. 

We then wrote the post, heavily relying on anecdotes, metaphors, talking about the topic, while also talking around it—the post is called The Reps we Stopped Doing. Ironically, despite the complete absence of any keywords, I think it does a pretty good job of conveying the premise. AI search seems to think so too. I entered our test questions into the site search and the post consistently came up as the top result. In fairness, there's not a ton of content on this site to sift through, but overall it looks promising that the AI search will be a great addition to sites, and surface the most relevant results for users. 

Test the search here.

For those curious, here are the guardrails we used: 

Guardrails: Semantic Search Test Article

Article topic (internal only, never stated in text): The phenomenon of cognitive skill erosion resulting from over-reliance on AI tools.

Target query set (what the article must be findable by, without using their language):

1. Is AI making people worse at thinking?

2. Does relying on AI cause skill loss?

3. What happens to your brain when you outsource thinking to AI tools?

4. Are students losing critical thinking skills because of AI?

5. Can using AI too much make you dependent on it?

Tier 1 — Hard Bans (must not appear, in any form)

These are the direct lexical targets of the test queries. If any of these appear, the test is compromised.

• dumb, dumber, stupid, stupider

• atrophy, atrophied, atrophying

• skill loss, losing skills, lost skill

• dependent, dependence, dependency, reliant, reliance

• critical thinking

• cognitive decline, cognitive atrophy, cognitive loss

• brain rot, brain drain

• intelligence (when used as a human attribute being lost)

• thinking skills, thinking ability

• outsource / outsourcing (when paired with thought, brain, thinking, judgment)

Tier 2 — Near-Miss Bans (must not appear; these would leak the concept to keyword search)

• weaken, weakening, weakened (when applied to mental faculties directly)

• deteriorate, deteriorating (when applied to mental faculties directly)

• erode, eroding, erosion (when applied to mental faculties directly)

• decline (when paired with mental, mind, ability, skill)

• crutch (used metaphorically for AI)

• lazy, laziness (as applied to the mind or thinking)

• dumb down, dumbing down

• cognitive (in any construction)

• mental muscle (too on-the-nose)

• forget how to (think / reason / write / etc.)

• lose the ability to

• can't think for themselves

Tier 3 — Permitted Metaphorical Territory (use freely)

These are the vehicles we want to use. They carry the meaning through analogy without naming it.

• Physical metaphors: muscles, fitness, training, practice, rust, stiffness, calluses, reflexes

• Navigation metaphors: maps, landmarks, turn-by-turn directions, GPS, knowing the way home, getting lost on purpose

• Tool-history metaphors: calculators and mental math, spellcheck and spelling, handwriting in the age of keyboards, memorized phone numbers

• Domestic metaphors: recipes we used to know, groceries we used to remember without a list

• Craft metaphors: the apprentice who never had to do the hard parts, a chef who only microwaves

• Observational framing: "you can usually tell," "there's a certain look," "the kind of pause that wasn't there before"

Tier 4 — Structural / Tonal Rules

• Never state the article's topic explicitly. No thesis sentence that says what it's about.

• Never use the word "AI" paired with any Tier 1 or Tier 2 word. In fact, minimize direct references to AI — prefer "the thing," "the tool," "the help," "the shortcut," "what we all started using a couple of years ago," etc.

• Tone: two friends in a coffee shop being terrible at discretion. Nudgy, knowing, a little conspiratorial.

Screening Protocol (for the final pass)

1. Grep the full text for every Tier 1 and Tier 2 term.

2. Read aloud and ask: "Would a journalist writing this topic reach for a word I'm using?" If yes, check whether it's a near-miss.

3. Confirm that at least 3 of the 5 target queries would semantically match the piece (human judgment).

4. Confirm that a keyword search for any single Tier 1 term returns nothing.

— Dennis Kardys