Suppose an agent skill improves the answers on your test set but also uses more tokens. Is that a win? Only if the extra quality is worth the resource cost for that task. A higher evaluation score is not evidence of a lower API bill.
NVIDIA’s SkillEvaluator report provides a useful example of why those measurements need separate columns.
Read the units before the headline
The report describes a controlled comparison: run the same prompt, model, task inputs and grading criteria in separate sandboxes, with and without the skill. Skill Lift is the with-skill score minus the without-skill score, in points.
For the August 12, 2026 benchmark snapshot, macro-averaged Correctness rose from 46 to 87: a gain of 41 points. That is not a 41% improvement, and 87 is not an estimated 87% chance of passing your application’s tasks.
The label “Efficiency” also needs care. It scores behavior such as avoiding wasted steps or redundant tool calls; the report tracks token usage separately. An improved Efficiency score is not a token discount.
Keep resource measurements separate
The report gives two single-attempt examples:
| Skill | Tokens, without → with | Execution time, without → with |
|---|---|---|
jetson-optimize-memory |
617,306 → 142,540 | 474.9 → 220.0 seconds |
cuopt-install |
25,227 → 55,582 | 34.0 → 41.1 seconds |
One used fewer tokens and finished faster; the other used more tokens and took longer. Neither row establishes a repeatable savings rate. Nor does the catalog-wide correctness gain tell us that either of these individual skills improved correctness by that amount.
Make the adoption decision on your tasks
For an API-backed agent, I would use the following evaluation procedure—not treat the catalog score as an adoption threshold:
- Freeze the comparison. Record the skill revision, model version, harness configuration, task inputs and grading rubric. Change skill availability, not the model and instructions at the same time.
- Include tasks where the skill should stay out of the way. Grade answer correctness separately from relevance: did the agent use the skill when appropriate and leave it unused on unrelated requests?
- Keep one row per attempt. Record task ID, condition, correctness, relevance, input/output token usage, elapsed wall time and failure status. Retain failed attempts rather than calculating resource use only for successful runs.
- Repeat both conditions. Inspect paired differences by task and report their spread, not just a catalog-style average. The article says 85% of skills with published results had one attempt per task and 15% had two; it reports no confidence intervals. Small one-run differences are a reason to repeat the experiment, not declare a winner.
- Price usage separately. Use the billing categories and rates applicable to your API calls, distinguishing cached usage where relevant. Keep elapsed time alongside that estimate: neither total tokens nor a rubric score alone is a measured bill.
Before running the comparison, set the minimum acceptable correctness and relevance, plus your cost and latency budgets. If a skill clears the quality bar but exceeds a budget, inspect the traces before deciding: are the extra calls doing necessary verification, or repeating unproductive work? Those are diagnostic possibilities, not conclusions supplied by the benchmark.
The useful outcome is not “skills save tokens.” It is a task-specific decision: accept the quality/resource tradeoff, revise the instructions, or keep the baseline. Which would block adoption in your workflow first—an incorrect answer, irrelevant skill activation, extra API cost, or a slower result?