Chapter 9 — Tool-Using and Agentic AI Systems

Metadata

Opening Quote

MSQE principle: When an AI system can act, correct language is insufficient evidence; authority, state transition, confirmation, and audit boundaries become part of quality.

Opening Story

The following is an illustrative scenario. Atlas Support Assistant can retrieve an order, create a refund proposal, and submit a refund after explicit customer confirmation. It correctly tells the customer that a refund requires review, but first submits a refund proposal for the wrong order after an ambiguous reference. In a second run, it chooses the correct order but sends a submission request before confirmation is recorded.

The system is not merely answering. It is selecting tools, forming arguments, interpreting state, and potentially changing it. Tool success responses and fluent explanations do not prove that the action was authorised or correct.

Why This Chapter Matters

Tool-using and agentic systems combine AI behaviour with API contracts, permissions, workflows, and side effects. An agentic system is one that may plan or select actions across steps; the term does not imply autonomy without controls. Quality engineering must make action authority and stop conditions explicit.

This chapter does not advocate unrestricted agents, autonomous refunds, or a particular orchestration framework.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish tool selection, argument formation, execution, and state-transition claims;
• define least-authority and confirmation boundaries;
• test multi-step workflows without triggering real effects;
• specify traces for safe diagnosis and audit; and
• produce a Tool-Use and Agent Workflow Safety Record.

Action Boundaries Are Product Contracts

Idempotency means that repeating the same authorised request has the same intended effect as performing it once. It is an API/workflow property, not a promise that a model will avoid repetition.

Model Planning Does Not Grant Authority

Treat model-generated tool calls as untrusted proposals. A deterministic policy layer should validate permissions, schemas, confirmation state, amount and policy constraints, and allowable sequence. The application, not the model, owns irreversible execution. When an input is ambiguous, the safe action may be clarification rather than tool use.

Worked Reasoning: A Proposal Is Not a Refund

The correct outcome is not a more persuasive apology. It is prevention of an unauthorised state transition plus evidence sufficient to repair the responsible boundary.

A Complete Synthetic Action Trace

The following trace is a compact, illustrative evidence packet. It is deliberately incomplete: it has enough information to make an engineering decision, not enough to expose real customer data or operational controls.

The final sentence is not sufficient evidence of correct agent execution. Step 3 targeted the wrong order, and step 5 produced an invalid proposal. The independent policy layer prevented a state transition, which reduces harm but does not make the agent’s planning or entity-resolution behaviour acceptable. The decision is to retain the guard, investigate the application/entity boundary, and add this trace class to regression evaluation.

Trace B: Retry, State, and Termination Failure

The following illustrative trace exercises a different failure mode. The customer has selected order 412-B and explicitly confirmed a permitted refund proposal.

Trace B demonstrates that a retry is not automatically a recovery action. A known rejected request may be retried under documented conditions. An ambiguous side effect requires idempotent reconciliation using the same stable request identity, an API status check, or controlled human intervention. Generating new keys can defeat the very duplicate-prevention mechanism intended to make retries safe.

Evaluate Five Separate Qualities

The learner should produce a trace review that identifies the first violated boundary, not merely the last visible error. This supports targeted correction: entity resolution for Trace A; session scoping, reconciliation, and idempotency design for Trace B; deterministic policy enforcement for both.

Trace C: Cross-Interaction Memory Contamination

Trace C makes memory a testable quality boundary rather than a convenience feature. The review must ask whether information is necessary, current, actor-scoped, authorised, and safe to retain. It also shows why a denied tool call does not fully resolve harm: the final answer can still leak an unsupported inference or create an unneeded trace.

Agent Evaluation Matrix

Use the matrix for a professional Action-Trace Review: name the first failed boundary, affected state, evidence limitation, correction owner, and regression trigger. It avoids an agent-framework tutorial while making process quality inspectable.

For an ambiguous side effect, use the sequence verify → reconcile → stop or escalate. Do not retry merely because a timeout occurred: the tool may have acted after the client lost the response. Idempotency keys reduce duplicate risk only when the same stable key and server-side semantics are applied; they are not a universal exactly-once guarantee. An audit trail should retain the safe action identity, policy decision, result state, versions, and handoff without indiscriminately retaining private prompt content.

State, Memory, Retries, and Termination

An agent workflow needs explicit state ownership. Customer-visible session state, model-provided memory, retrieved facts, tool results, confirmation tokens, and policy decisions are not interchangeable. The application should decide which state persists, how it expires, whether it is revalidated, and how it is associated with an actor and order.

Retries deserve separate tests. A transient tool failure may justify a bounded retry; an ambiguous action result may require reconciliation rather than repetition. Termination means that the workflow stops or hands off when it has no permitted safe action, not merely when a model emits a confident answer. Test loops, repeated proposals, stale confirmation, partial failure, and human takeover using resettable synthetic state.

Action-Trace Review Questions

Review an action trace in order: Was the goal interpreted within scope? Was the selected tool permitted at that step? Were arguments valid, current, and associated with the correct subject? Did a deterministic policy layer validate authority and confirmation? Did the retry preserve idempotency and avoid duplicated side effects? Did the workflow terminate or hand off when information was insufficient? Could a reviewer safely reconstruct the answer without retaining more personal content than needed?

The answer to one question cannot compensate for another. A correct final sentence with an incorrect tool target remains an unsafe execution path. A denied submission can demonstrate a guard’s value while revealing that planner quality, entity resolution, or memory handling still needs correction. This is why tool-using evaluation must include intermediate artefacts rather than a final-response-only rubric.

Multi-Step Tests Need Controlled State

Use a synthetic environment with resettable state, safe stubs, explicit preconditions, and observable effects. Test rejected calls as seriously as successful calls. Include retries, timeouts, partial tool failures, stale state, conflicting user instructions, and human handoff. Do not let a test run against a production refund interface merely because the system is “only proposing” an action.

ReAct illustrates a research pattern combining reasoning and actions, but product use still requires explicit authority, safety, and observability controls.¹

Engineering Perspective

The primary oracle for a consequential action is often deterministic: was the tool available, was its input valid, did the policy layer permit it, was confirmation present, and did the system record the result? Generative evaluation complements—not replaces—those checks by assessing explanations, clarification, and action selection.

Industry Perspective

The NIST AI RMF frames risk management as socio-technical and context-specific.² For tool use, that context includes user authority, workflow design, API semantics, operational recovery, and people affected by an erroneous action.

Common Misconceptions and Pitfalls

“The model chose the right tool, so the action is safe”

Selection is only one boundary. Arguments, permission, confirmation, state, and execution remain to be checked.

“Human confirmation makes all agent actions acceptable”

Confirmation must be informed, bound to the intended action, resistant to stale context, and enforced by the application.

“Agent trace means record every prompt and private value”

Trace the decision-relevant, safely redacted facts and versions. Retention and access limits remain necessary.

QA → QE Transition

QA tests tool responses. Quality Engineering defines the permitted action graph, independent policy enforcement, state preconditions, recovery behaviour, evidence, and release gates for side effects.

Summary

Tool-using AI systems are workflow systems. Their quality depends on bounded authority, deterministic enforcement, safe state handling, testable failure paths, and proportionate audit evidence.

Key Takeaways

• Treat model tool calls as proposals, not authority.
• Enforce permissions, confirmation, and state transitions outside the model.
• Test denials, retries, and recovery paths in controlled synthetic environments.
• Trace action-relevant evidence safely and proportionately.

Review Questions

1. Why is tool selection insufficient evidence for safe action?
2. Define idempotency in the Atlas workflow.
3. Which boundary owns refund submission permission?
4. Why should rejected calls be tested deliberately?

Interview Questions

1. How would you test an AI workflow that can issue a refund?
2. What belongs in an agent-action trace?
3. How would you prevent a model from bypassing a confirmation step?

Practical Exercise

Create a Tool-Use Contract and Action-Trace Review for Atlas’s order lookup, refund proposal, and refund submission path. Define tool authority, preconditions, confirmations, denied paths, resettable synthetic state, trace fields, and revision triggers. Do not access any live service.

Further Reading

• ReAct: Synergizing Reasoning and Acting in Language Models
• NIST AI Risk Management Framework

References

Chapter Checklist

• I can distinguish planning, tool-call, execution, and state-transition claims.
• I can define an independent confirmation boundary.
• I can test a safe denial and recovery path.
• I can specify proportionate agent-action evidence.

Chapter Navigation

Previous: Chapter 8 — Human Evaluation and Model-Based Evaluators · Next: Chapter 10 — Safety, Fairness, Privacy, and Responsible Quality Boundaries