AI tool testing methodology

AI tools change quickly. A useful test must be bounded by a specific task and date, not treated as a permanent verdict.

Every test records

1. The task and success criteria.
2. The tool, plan, model or version when available, and test date.
3. The input material and key settings needed to reproduce the result.
4. The observed output, failure modes, and human-review effort.
5. Who the result is likely—and unlikely—to help.

What we avoid

We avoid unsupported “best” claims, invented benchmark results, and recommendations that conceal important trade-offs. When evidence is incomplete, we say so.