The model update nobody evaluated
Earlier evaluation evidence may no longer support reliance on a changed model or configuration during an offensive security engagement.
Fictional illustrative situation
A team changes the model used to propose checks and summarise evidence in a SaaS penetration-test workflow. It keeps the earlier readiness decision. The new configuration proposes additional targets and omits missing evidence from a draft finding. The consultant cannot justify relying on those outputs. This situation is fictional.
What has gone wrong
The team has carried evidence about one configuration into reliance on another. It cannot show whether the changed capability respects engagement limits or distinguishes a demonstrated issue from an unsupported explanation. A familiar interface does not resolve those gaps. The client report could inherit a claim whose supporting evidence was never inspected.
Evidence the lead should inspect
The technical lead compares the recorded and current workflow version, model identity and exposed version, prompts, tool permissions and settings. They check earlier evaluation conditions and unresolved issues, then identify the affected engagement activity, conclusions and the model’s actual role. They should establish which activities remain usable without it and record unavailable configuration details as uncertainty.
A proportionate correction
The owner should pause reliance on the changed model and evaluate a small set of sanitised records with known conclusions. For this proposal and summary role, include an evidenced weakness, a clean result, failed authentication, missing artefacts and an out-of-scope lead. The technical lead checks whether proposals are justified and authorised, and whether summaries preserve limitations. These cases evaluate the model’s use of records without giving it application access.
Until that evaluation supports resumption, the lead can use an evaluated manual workflow or hold the affected activity. If the model still proposes leads, each remains unexecuted until a consultant verifies its rationale and authority. That contains its use without accepting its reliability. The owner records the available recovery option and report limitations; restoring the previous version may be impossible.
Check whether the correction worked
Another reviewer should connect the current configuration to the evaluation evidence and readiness decision, confirming that unsupported claims and scope extensions were rejected. The owner repeats affected cases after corrections and keeps unresolved limits visible. This evaluation supports the model’s defined role under tested conditions; it does not establish engagement completion or that the assessed environment is secure.