Tutorial 30 · Metrics: what JMAX costs and how well it answers

What the video says

1. Metrics answers two questions: what JMAX costs, and whether what it answers is worth it. Spending and budget at the top; effectiveness indicators below.
2. Every model answer is counted: input tokens, output tokens and their cost at the going rate. At the top you see what you have spent this month, and below how many tokens and how many calls that took.
3. Set a monthly budget and save. From then on a bar appears with the percentage used, and JMAX warns you as it approaches the limit. With no budget you only see the spend, with nothing to measure it against.
4. The table breaks it down by provider and model: calls, input tokens, output tokens, and what it cost. Sorted by cost, with the share each one represents of the total.
5. Look at the first row: the local server does by far the most work and costs nothing. What you pay for is the cloud, and here you see exactly how much and on which model. This is the figure that justifies having your own server.
6. The host sits next to the model on purpose: the same model can run on your machine and in the cloud, and the host is what tells them apart beyond doubt. And if a provider does not report usage, it says so as not counted.
7. Below, the tokens per day over the last month. It is for reading the curve: if spending goes up, here you see from which day, and that usually explains why better than the monthly total.
8. The rates are editable, in dollars per million input and output tokens. They ship with list prices, but if you have a different agreement, change them here and the maths follows. Local ones sit at zero.
9. The second half is the effectiveness indicators, and they all come from one thing: the stars you give the answers. Every star trains JMAX and feeds these numbers.
10. Overall effectiveness, average stars, satisfaction, which is the four and five star answers, rated answers, dissatisfaction, which is the one and two star ones, and the trend against the previous week.
11. And the same broken down by channel: written chat, WhatsApp, Telegram. Here you see whether JMAX answers equally well on all of them, or whether one channel is dragging the score down.
12. The star distribution shows the shape, not just the average. An average of four spread evenly is not the same as an average of four with many fives and a few ones: that second one points at a specific problem.
13. And the daily average over the last two weeks tells you whether you are improving. It is the chart to check after changing a brain, adjusting a skill or correcting an agent.
14. The advice that makes all of this useful: rate the answers. If nobody gives stars, these indicators stay empty and JMAX has nothing to learn from. Rate what reaches you and you get trend, weak channel and cost, on one screen.