The first tool this lab tests is the obvious one: an AI chatbot, running on this site, answering questions about it. The tool is Chatvox, the bubble at the bottom right of this page. This post is the full record: the setup, the training mistake, the test questions, and the grades.
Setup
An agent named WPLLMs Assistant was created in Chatvox and given three things. Instructions: answer from the site’s content, be plain-spoken, never invent results, and say so when it does not know. A look: deep teal widget at the bottom right, with the greeting “Questions about the lab or our AI-on-WordPress tests? Ask me.” And knowledge: this site’s own pages. On WordPress it installs two ways: a script tag from the dashboard, or the official Chatvox plugin, where you paste in an Agent ID. This site runs the plugin. The configuration took minutes. Sorting out account access took considerably longer, which is a lesson about admin friction, not about the chatbot.
The training mistake
The first training run happened while the site was still in maintenance mode. The crawler did its job faithfully. It fetched the pages and learned the holding page: 385 characters of “under maintenance” for every URL. A chatbot is only as good as what it actually read, and a green “Ready” in a dashboard does not tell you what got read. After launch, the stale sources were deleted and re-added. The day-one post came in at about 3,800 characters, the About page at 1,300, the homepage at 1,200. If this experiment teaches one thing, it is to check the size and content of what your bot trained on.
The tests
Three questions, asked as a visitor.
“What is WPLLMs?” Correct and complete: an independent test lab for AI tools built for WordPress, experiments on the Blog page, background on the About page. Pass.
“What was the first experiment on this site?” Correct: this one, Experiment 001, the chatbot itself. Pass.
“How much did the plugins on this site cost?” This is where it got interesting. The day-one post states the answer outright: total spent on plugins so far, $0. The first attempt hedged: the bot said it had no verified breakdown. Not wrong, and importantly not invented, but not the answer sitting in its own training data. After retraining on the real pages it named the free plugins and cited its sources, but still hedged on the total. After one Q&A entry was added stating the fact plainly, it answered: “The plugins on this site cost nothing. We’ve spent $0 on them so far.”
The pattern across all three: the bot retrieves well and shows its sources. Its weakness is synthesis. Given a total stated plainly in a document, it preferred a cautious summary over the number. One targeted Q&A entry fixed the specific fact.
Verdict
| Area | Finding |
|---|---|
| Setup effort | Low. The official plugin plus an Agent ID, minutes to configure. |
| Training | The crawler works, but verify what it fetched. A maintenance page will poison it quietly. |
| Answer accuracy | Three of three correct after tuning. Two of three straight out of the box. |
| Failure style | Hedges instead of inventing. That is the failure mode you want. |
| Cost | Runs on the existing Chatvox team plan. No extra spend for this test. |
The bot stays on the site. It gets retrained as new experiments publish, and its misses will keep appearing in these logs. Next up is Experiment 002: AI site search.
