How Regulated Firms Fixed Their AI Systems to Pass Review

Chart of practices that worked and failed at 33 regulated firms: recorded human sign-off led the fixes with 10, and retrofitted records led the failures with 7 of 26

Recording human sign-off as a workflow step was the most common AI audit readiness fix, named by 10 of 33 regulated firms. Adding records after the build was the most common failure, with 7 of the 26 failures coded in the study.

Study methodology: An open-response survey ran through Connectively and direct outreach between June and August 2026, and 33 of 67 submissions passed screening. Each respondent was asked for a practice that worked and a practice that failed. The coding produced 33 working practices and 26 failed ones.

How regulated firms fixed their AI systems to pass review

Practice that workedFirmsPractice that failedFirms
Human sign-off recorded as a step10Records retrofitted after the build7
Evidence captured at decision time9Periodic reports where live evidence was due5
Deterministic core, AI at the edge5Supplier certification as own validation5
Controls at data and identity layer3Accuracy metrics as compliance evidence4
Evidence package per use case3Prompt or written policy as a control3
Change control written before go-live2Generic checklist for every use case1
Evidence pushed to auditor’s system1Synthetic tests in place of production data1
Total coded practices33Total coded practices26

Essential statistics: AI audit readiness practices that worked and failed

  • Human sign-off recorded as a workflow step was the most common working practice, at 10 of 33.
  • Evidence captured at the moment of decision followed, with 9.
  • A deterministic core with generative AI confined to the edges worked for 5.
  • Controls at the data and identity layer worked for 3, as did a single evidence package per use case with a named owner.
  • Records retrofitted after the build were the most common failure, at 7 of 26.
  • Periodic reports sent to auditors who wanted live evidence failed for 5, and so did supplier certifications treated as internal validation.
  • Accuracy metrics offered as compliance evidence failed for 4, and a prompt or written policy used as a control failed for 3.

Key takeaways

  • The 2 leading fixes both create evidence while the system runs. Recorded sign-off and decision-time capture account for 19 of the 33 working practices.
  • Evidence that arrived late caused 12 of the 26 failures. 7 firms added records after the build, and 5 sent periodic reports to auditors who wanted live evidence.
  • Borrowed assurance failed as often as periodic reporting, with 5 firms treating a supplier’s certification as their own validation.
  • 3 firms failed review by using a prompt or written policy as a control. At SynSphere, an auditor rejected a prompt that told a model to withhold a data category.
  • 4 firms offered accuracy scores as compliance evidence and failed. CyberMed’s founder noted that the FDA expects traceability from requirement through risk to test evidence.

Actionable insights

  • Record each human sign-off as a step the system logs, since 10 of 33 firms named it, more than any other AI audit readiness practice.
  • Capture evidence at the moment of each decision, because 9 firms named it as their fix and 7 failed by rebuilding records after the build.
  • Keep figures in a deterministic engine and let generative AI write the language around them. CoinLedger and ProAI worked this way, among 5 firms that used the split.
  • Run your own validation even when a supplier holds a certification, since 5 firms failed by accepting the supplier’s paperwork as their own.
  • Give each use case a single evidence package with a named owner, a setup that worked for 3 firms.

“What did not work was trying to reconstruct that sequence afterward from transcripts, screenshots, application logs, ticketing systems, and separate approval records. Each artifact could be accurate, but the collection still failed to establish a clear chain of custody.” — Ken Herron, Co-founder, VCONify

SumatoSoft logo
If you have any questions, email us info@sumatosoft.com

    Please be informed that when you click the Send button Sumatosoft will process your personal data in accordance with our Privacy notice for the purpose of providing you with appropriate information.

    Vlad Fedortsov, IT Sales Expert at SumatoSoft
    Vlad Fedortsov
    Account Manager
    Book an intro call
    Thank you!
    We've received your message and will get back to you within 24 hours.
    Do you want to book a call? Book now
    SumatoSoft clients logo