We switched Nonsuri's explanation model: costs fell 41% and the payment rate rose 18%

Nonsuri is a humanities essay explanation service run by the ABTO team. When a user solves an essay problem, AI adds an explanation, and we changed how we pick the model for that feature from benchmark scores to user behavior. Comparing a new model setting side by side with the existing one, model costs fell by 41% and the payment rate rose by 18%.

How did we pick a model at first?

We used a model known for strong performance. The only numbers we could look at were coding and math benchmarks, which had little to do with people reading essay explanations.

What we wanted to know was whether users solved the next problem after reading an explanation, and whether they went on to pay. Model costs rose every month, and we had no way to tell whether that spending led to payments.

What did we decide to look at?

We chose to track the retake rate, the payment rate, and the problem-solving rate together. We kept using benchmark scores to pick candidates and made only the final call on user behavior.

The blocker was that the records lived in separate places. Call costs and payment records were stored apart, so we could not tell which model’s explanation a paying user had seen.

For this comparison, we set the payment rate as the single deciding metric. We only watched the retake and problem-solving rates.

How did we split the comparison in ABTO?

We connected Nonsuri’s explanation feature to ABTO and split users of that feature into two groups. One group kept the existing setting, and the other received a new setting that used GPT-6 Luna.

As success metrics, we set up the payment action rate, grading scores, the rate of additional actions after solving a problem, and average cost per call.

How did we roll out the new setting?

We started by giving the new setting to 10% of users. After a few days of checking error rates, response times, and success metrics, we slowly raised the share to 50%. Expanding in small steps kept the number of affected users small in case something went wrong.

In the first few days there was little data and results flipped often. We kept comparing until the numbers settled.

What did the numbers show after a week?

After more than a week of tracking, the new setting cut model costs by 41% and raised the payment rate by 18%. More users also read explanations to the end and moved on to the next problem.

The setting with the higher payment rate was the cheaper model. That is why costs and the payment rate improved at the same time.

This result covers one explanation feature, judged on the payment rate. We do not report figures for the other metrics we watched, such as the retake rate.

How do we pick models now?

When a new model comes out, we use benchmarks to pick candidates, give them to some users first, and look at the payment rate and costs together. An expensive model may win on another feature, so we check each feature separately.

The steps we followed for Nonsuri are in How to choose the right AI model for your product (Korean). ABTO is an AI performance analytics tool that shows AI call costs and user responses side by side for each feature. Install the ABTO Skill in your coding agent and ask /abto Integrate ABTO to start the integration.