A fairer personal comparison

Give both approaches a reasonable chance to succeed.

Suppose you finish a task with AI and then repeat it without AI. The second attempt is faster.

What did you learn?

Maybe the unaided approach was better. Maybe you remembered the sources, the structure and the answer from the first attempt. The comparison contains both possibilities.

In the illustration, the first attempt finds a deadline change from Friday to Thursday. By the second attempt, you may already know to look for Thursday. Removing AI has not removed what you learned. Those are fictional inputs, not recorded trial results.

Yesterday we decided what to count. Today we need to think about which work gets counted on each side.

Keep the question small

You don't need to prove whether AI is useful for everyone. You need enough evidence to decide whether a particular arrangement helps with your recurring task.

For our example, that might be: does an AI-assisted comparison help me complete source checks with less active effort while preserving accuracy and unresolved questions?

The outcome has two parts. Faster unsupported answers don't satisfy it. Better-supported answers that take longer may still be valuable, but they answer a different version of the question.

Compare similar work without pretending it is identical

Choose several tasks of roughly similar difficulty before you know which approach will handle them. Give both routes the same relevant material and the same finish criteria. If you can, assign the route by a simple random choice instead of reserving the easy tasks for AI.

Random assignment reduces your ability to choose favorable cases, but a tiny personal trial can still be unbalanced by chance. Keep obvious differences visible: one source had a dense table, another had a missing comparison copy, a third was a short paragraph.

For a small constructed pair, task A could compare a Friday-to-Thursday deadline change; task B could compare a meeting-location change from Room 3 to Room 4. Supply both versions and a one-sentence note for each. Both ask for one change and its supporting passage. That makes the intended difficulty similar, not identical. Use more than one pair and vary which route goes first where feasible; you may still learn the checking method as you go.

Use the normal tools you would use without AI. An artificially inconvenient baseline answers the wrong question. If you usually search your notes, use a checklist or run a familiar script, those belong in your ordinary route.

Record failed attempts too. Removing the tasks where one approach struggled would change the question from “Does this help with my work?” to “How well does it do on the work where it succeeds?”

Review the result before you read the timing

Set a few concrete quality criteria. Did the result identify the relevant source? Preserve the condition that matters? Leave missing evidence unresolved? Explain the next action?

If another person can review the results without knowing the route, that can reduce some expectations in the judgment. If you review them yourself, say so. A personal trial doesn't have to disguise its limits to be useful.

We learned something similar from our comparison of AI working arrangements. The later contributions received qualified memos that the first response hadn't received. Some final copies had fewer problems, but that couldn't fairly establish that more contributors caused the improvement. Earlier corrections and uneven inputs were part of the result.

That experience gave us a useful question for future comparisons: are we comparing the arrangement, or comparing how much help each arrangement received?

Make a modest decision

A few comparable tasks can reveal a recurring repair, a clear inconvenience or an encouraging pattern. They won't isolate every cause or settle the effect for all future work.

Keep the individual results beside any average. “Usually faster, but failed on the table case” gives you more direction than a single number that buries the exception.

You might keep AI for the initial comparison and do the final source check yourself. You might improve the input and try again. You might decide the ordinary route is already good enough.

The point of a fair comparison is to make those choices easier. Tomorrow we'll apply the same questions to published AI research, where a striking percentage also depends on what was compared.