A wrong practice question is worse than no practice question at all.
Sit with that for a second. If you skip a topic, you know you skipped it. But a plausible, confident, subtly incorrect question teaches you the wrong thing and then hands you a checkmark for learning it. You cannot catch the error (you do not know the material yet; that is the entire reason you are studying). The question that looks right and is quietly wrong is the one that follows you into the testing center.
So when we set out to build a bank of Salesforce certification practice exams, the interesting engineering was never the generating. Any model will cheerfully write you ten thousand questions before lunch. The interesting engineering was the checking: building a system that treats its own output as guilty until a second, differently-minded process proves it innocent.
The theme underneath everything below is trust nothing, including yourself. The generator does not trust the source page. The reviewer does not trust the generator. The code does not trust the reviewer's arithmetic. And when we finally audited the thing that fixes bad questions, it turned out we should not have trusted that one either. It had been shipping bugs the whole time.
Refuse to write from a bad page
The cheapest quality control is refusing to start.
Before a single question gets written, every source document is scored 0 to 30 by a model on relevance, depth, and clarity. Anything under 15 is dropped on the floor. This is not glamorous work, and it catches an embarrassing amount: login walls, maintenance notices, error pages, the navigation chrome that a crawler scoops up along with the real content. Of 6,055 documents we pulled across 141 exam topics, 750 (12.4%) never produced a question at all.
You cannot write a good question about a login wall. A model asked to try will write one anyway (and it will look perfectly fine, which is the problem). Better to never ask.
The questions that do get written are aimed at principles, not pages. A question about the worked example in some doc's section 12 tests whether you read section 12. A question about the concept that example illustrates transfers to the exam. We instruct the generator to ask about the second kind.
The grader must not be the author
This is the load-bearing decision.
Questions are written by one vendor's model and graded by another's. As of July 2026 the author is a Google model and the reviewer is an OpenAI model; different vendor, different training data, different blind spots (the specific names will roll; the principle will not).
A model reviewing its own work agrees with itself, warmly and confidently, because it shares the exact blind spot that produced the mistake in the first place. Ask it to grade its own homework and it gives itself an A. Hand the same question to a model from a different family and, at minimum, the reviewer's mistakes and the author's mistakes are now uncorrelated. The reviewer also gets the original source document and has to check every claim in the question against it. No source, no valid review.
Salesforce is blunt about what these exams measure. The official Platform Developer I exam guide describes its candidate as someone with roughly one to two years of developer experience and at least six months on the Lightning Platform. That is a bar about applied judgment rather than trivia recall, and it is exactly the kind of bar a self-grading model will lower for itself.
Each review scores four criteria, 20 points each, 80 total:
- Factual accuracy. Does every claim match the source, including the ones that sound true in general?
- Groundedness. Is this traceable to the source, or did the model reason its way there from outside knowledge it happened to have?
- Correct-answer validity. Is the keyed answer unambiguously the best, or could a well-prepared candidate mount a real case for another option?
- Explanation quality. Does the explanation justify the key and rule out the distractors?
The reviewer has to quote the source passage behind each score. Handwaving is not a review.
Two of those criteria catch the real damage. Groundedness catches the question that is perfectly true but not taught in the material, which is accurate and useless as practice. Correct-answer validity catches the ambiguous item, and the ambiguous item is the cruelest thing you can do to a candidate: reason correctly, arrive at a defensible answer, and get marked wrong. Do that to someone twice and they stop trusting the whole product (which is the right call on their part).
Let Python do the arithmetic
The reviewer returns four numbers. The total is summed in code, never by the model.
It would have been easy to let the model return the total too. It would also have been wrong. Models are good at judgment and mediocre at addition, and a total that silently disagrees with its own four components is an unfalsifiable data-quality bug that lives in your database forever, smiling. Let the model do judgment. Let Python do sums.
Averages hide the failure
A question is pulled for rewriting if the total falls below threshold, or if any single criterion drops below 15 out of 20. That second condition is the one that earns its keep.
A question can score 68/80, comfortably fine on paper, while hiding a 9/20 on correct-answer validity. Average the four and it looks healthy. What those numbers describe is a well-written, well-explained, factually accurate question with two defensible answers, which is precisely the item that teaches a candidate to distrust you. Gate on the total alone and you have built a machine that launders one serious flaw behind three good scores.
Across four full verification passes over the bank we wrote 19,413 review records. The mean score climbed from 67.9 on the first pass to 71.6 on the latest, out of 80. The correction agent rewrote 3,660 questions along the way, roughly one in four.
We audited our own fixer and found 408 bugs it had introduced
This is the part I did not see coming.
The correction agent rewrites a flagged question as one JSON object: the stem, four options, the answer-key letter, and the explanation, all produced together. In a subset of rewrites the model settled on one ordering for the options array, then wrote the explanation's inline letter references against a different ordering. The stored key says C. The explanation says "Option B is correct." Both minted in the same breath, by the same model, inside the same object. 408 questions carried this contradiction, and every one of them had already been "improved" by the corrector.
We did not catch it with another model. The defect is deterministic, so the detector is too: a regex that parses every letter verdict in each explanation and flags any row where the keyed letter is called wrong or a non-keyed letter is called right. Read-only. No model in the loop (and so no judgment to be talked out of).
The fix is the better half. We did not go fix the letters. We stripped every letter reference out of the explanations and rewrote them to name options by their content instead. A letterless explanation cannot name the wrong letter, which kills the whole defect class rather than patching 408 instances of it. It turned out the app never shows answer letters to users anyway, so those references were meaningless even when they happened to be right. The rule now lives permanently in the correction prompt.
The unglamorous safety work around it deserves a line: every original explanation was snapshotted to an append-only log before the update, a deterministic gate rejected any rewrite that still contained a letter reference, the job was resumable and idempotent, and the production patch was content-keyed (so it was structurally incapable of clobbering a row that had drifted). Letter-key contradictions in the bank today: zero.
The lesson generalizes. Your quality machinery is only trustworthy if you also point it at the quality machinery. The reviewer found bad questions. It took a separate, deliberately paranoid audit to find the bugs in the thing fixing the bad questions.
Publish the numbers you don't like
Now for the part a marketer would delete, and I am leaving it in.
1,113 questions still sit below 60 out of 80. They are queued rather than shipped, and we say so on the site along with the mean and the distribution (and a note on which exams still have thin banks). These questions are generated from public Trailhead and documentation material, not leaked or real exam content; the gap between the 67.9 first-pass mean and the 71.6 latest-pass mean is the measured value of running verify-and-correct at all. It is real, it is not enormous, and I would rather show it than hide it. A quality report with no bad numbers in it is not a quality report. It is a brochure.
One more number I like. After balancing the answer key: A 25.0%, B 25.2%, C 25.0%, D 24.8%, a spread of four tenths of a point across 14,432 questions. You cannot beat this test by always guessing C.
What this gets you
"Independently verified" here means one specific thing: a separate AI reviewer, from a different model family, that reads the source and grades every claim against it. Not a human panel, not an expert board, not Salesforce. Just a genuinely adversarial second pass, run four times and counting.
If you want to use it: the core exams are free, no signup, anonymous, with results that expire after four hours. A free account adds exam history, Previously Wrong, and New Questions Only. You can work through the verified practice exams for the certification you are chasing without handing over an email.
The generating took days. The verifying is still running. That ratio is the summary of the whole project: with AI-generated study material, the writing is the easy part; the adversarial process around it is the work.
Trust nothing, including the thing that checks the thing. Especially that one.