If you mostly use chat-style AI and you are just starting to follow agent tools, this is where people waste money fast. You see a new demo, hear that AI can now do multi-step work, and think the smart move is to upgrade the model, add more tools, and sort out the process later. That is the wrong order.
Without an eval set, agent budgets are guesswork. An eval set is a small test pack: representative tasks, known good answers, and a simple way to track failures. If you do not have that, you cannot tell whether the money is going into the model, the connected tools, or the workflow itself. You only know that the system feels busy.
That is the real issue inside How to manage AI investments in the agentic era. Agent systems are not the same as a one-shot chat reply. They are multi-step, they drift, and the same task can vary across runs. The three sources behind this brief all point in the same direction. OpenAI's cookbook warns against "prompt-and-pray" and pushes a measure-improve loop. OpenAI's eval guide says test data needs representative samples and human-labeled ground truth, or you cannot judge quality. OAgents adds the uncomfortable part: many agent tasks are hard to reproduce, and results can swing between runs when evaluation is weak.
A new AI update is not worth your attention because it lists more features. It is worth your attention if it changes your next decision. That is the filter most people skip. If you keep chasing surface-level upgrades, you burn time, budget, and attention in the wrong place, and you still do not know what actually improved.
The next step is boring on purpose: write a first golden set, meaning a small list of common tasks with known right answers, a pass bar, and a short failure list. Then decide whether you need a better model, more tools, or a different workflow. If this matches someone you know who is about to spend on agents without a test set, share it with them first.