Case study: An AI proof of concept (PoC) for an online learning platform
A UK online learning platform delivers academic and professional courses built around open-text assignments. Manual grading had become a bottleneck as enrollment grew, and every learner followed the same fixed path regardless of level. Before funding a full AI rebuild, the platform asked SumatoSoft to prove the idea on its own data. In four weeks, SumatoSoft fine-tuned a model that graded open answers within one point of human scores about 89% of the time, and built a knowledge-tracing model that lifted completion on the pilot cohort from about 68% to 79%. The client took the result and moved into a full production build.
Project details:
About the Client:
A UK online learning platform delivers academic and professional courses built around open-text assignments and quizzes.
Location: United Kingdom
Industry: EdTech, online learning and professional upskilling
Team size: 4 specialists (a lead data scientist, an ML engineer, a data engineer, and a project manager)
Project duration: 4 weeks
Business challenge:
The platform’s enrollment was growing faster than its graders could keep up, and open-text assignments piled up waiting for manual scoring. Every learner followed the same fixed path, and completion rates stayed flat. Leadership wanted to rebuild the product around AI but wouldn’t commit the budget until someone proved that grading and personalization would work on their own data.
Additional requirements
- Test on real, anonymized student data
- Measure auto-grading against the platform’s human graders
- Keep student data private and compliant
- Deliver a clear go or no-go decision with cost projections
Our solution
SumatoSoft scoped the proof-of-concept to one course and one cohort. For grading, the team fine-tuned a transformer model on the platform’s past graded answers and measured how closely it matched human scores, routing low-confidence cases to a person.
For personalization, a knowledge-tracing model adjusted the order and difficulty of content from each learner’s diagnostic results. Everything ran in a private sandbox on anonymized data, with agreement metrics agreed up front so the result would be a clear go or no-go rather than a demo.
Additional info about the case
The four-week scope kept the investment small and the decision fast. SumatoSoft delivered an evaluation report, a working sandbox, a costed production roadmap, and a clear go-or-no-go recommendation, so leadership could judge the result against a funded plan rather than a promise.
Additional features:
- Fine-tuned transformer grader with human review on edge cases
- Knowledge-tracing model for adaptive content
- Anonymized data pipeline with privacy controls
- Agreement metrics and a costed production roadmap
Business value
Before:
- Open-text assignments waited in a manual grading queue
- Every learner followed the same fixed path, and completion stayed flat
- Leadership had no evidence that AI grading would be accurate or fair
- A full AI rebuild was a large, unproven bet
After:
- The grader matched human scores within one point about 89% of the time on the pilot course
- On the pilot cohort, module completion rose from about 68% to 79%
- Leadership had measured proof, a clear view of the edge cases, a costed plan, and the confidence to invest
- The client signed on for the full production build
