Bridging the AI Demo-to-Production Gap: A Sentiment Analysis
Case Study

Most sentiment integrations I am asked to review look complete in a slide. A frontier model reads a message and, in one structured call, can often identify targets, aspects, emotion, stance, intent and supporting language. That capability is real. The failure begins when the application treats that judgment as a decision: negative at 0.91, therefore be more empathetic. That is a demonstration. It is not a system you can staff.

Production work starts at the boundary between what a model can infer and what an application can trust. A confident JSON field is not calibrated. A claim without exact evidence cannot be audited. A document-level label cannot tell policy what to act on. And a system with no abstention path converts ambiguity into fluent guesses. Without those controls, a model change can silently change the behaviour of the product.

Realistic speech is rarely about one thing. A customer can praise the agent and still be waiting for a refund. An article can be positive about NVIDIA’s revenue and negative about next quarter’s margins. A user can admire a company and refuse to buy at the current valuation. Collapse any of those into a single polarity and the downstream policy will act on the wrong object.

True sentiment analysis is not a better positive/negative prompt. It is a measured intelligence layer that turns model capability into an auditable, bounded decision: targets, evidence, calibrated uncertainty and a gate before anything consequential happens.

A label is not an analysis

Do not ship a production decision system whose entire contract is a polarity and a verbalized confidence. A simple binary classifier can be enough for narrow, low-risk reporting. But when the result can alter a response, route work or justify an action, the output only looks precise: its confidence is not calibrated, its polarity is not attached to a target, and it gives the application nothing to verify.

{
  "sentiment": "negative",
  "confidence": 0.91
}

A production analysis has to answer a longer list. Who or what is the sentiment about? Which aspect of that target is being judged? Is the language positive, negative, neutral or genuinely mixed? How intense is it? Is the speaker frustrated, satisfied, anxious or confused? Is this an evaluation, an emotion, an investment stance or an action they want taken? Which exact words support the conclusion? How uncertain is the model? Should it abstain or escalate?

Recent aspect-based work treats the problem as target extraction, opinion evidence and continuous valence–arousal, not a document-wide stamp. Finance makes the failure mode obvious: one sequence can carry opposing sentiment for different names, or for different aspects of the same name. Sequence-level labels then become the wrong unit.

Separate sentiment, emotion, stance and intent

These four jobs are routinely mixed, which is why the classifications go bad.

• Sentiment is an evaluation of a target or aspect: “The interface is beautiful.”
• Emotion is the affect in the language: “I am getting really frustrated.”
• Stance is a position toward an idea, company, asset or action: excellent business, would not buy at this valuation.
• Intent is what the speaker wants done: “This is the third outage. Cancel my account.”

A person can be anxious and still think well of the company. A company can look excellent and still be considered overvalued. If the schema cannot hold those distinctions, the assistant will apologize for the wrong thing, or worse, execute the wrong next step.

Score the target, not the document

“Your support agent was kind, but the refund still hasn’t arrived.” A basic classifier calls this negative or mixed and stops. The useful cut is: agent / helpfulness / positive; refund process / completion / negative; overall experience / mixed. The policy can now thank the person and still escalate the refund.

The same cut applies in markets. “NVIDIA beat revenue expectations, but gross margins may contract next quarter.” Revenue, present, positive. Margins, future, negative. That is not automatically a bullish stance. Positive company sentiment and a buy recommendation are not the same claim, and the system should refuse to invent the second from the first.

Keep intensity, then calibrate confidence

“The button is slightly confusing,” “this product is nearly unusable” and “this is the worst service I have ever experienced” are all negative. They are not the same event. Keep a categorical label for rules, and keep continuous valence and arousal for intensity. Neutral means no meaningful evaluative polarity. Mixed means materially opposing evidence. Neither word is a synonym for “the model is unsure.”

{
  "label": "negative",
  "valence": -0.84,
  "arousal": 0.91,
  "emotions": ["anger", "frustration"]
}

A generated confidence score is a verbalized probability, not a measured one. Treat it as a raw signal. Calibrate on held-out human labels in your domain. If the text is too short, the target cannot be resolved, sarcasm is likely without enough context, or quoted speech makes attribution unclear, the model should abstain, not guess more fluently.

The path that can be staffed

The operating sequence is not “call a larger model.” It is a pipeline with a cheap first pass and a human-readable stop.

1. Run a fast structured analysis on the current message.
2. Validate the result deterministically: evidence quotes must be exact substrings; canonical IDs must be copied from known targets, never invented.
3. Apply calibrated confidence, not the model’s self-score.
4. Escalate to a stronger model only when ambiguity, sarcasm, opposing aspects or abstention make the cheap pass unsafe.
5. Send the accepted analysis through a response policy.
6. Only then let the assistant speak or act.

Structured outputs give you the shape of the object. They do not give you a correct interpretation. OpenAI’s own structured-output guidance says the payload can still be wrong. Provider-side schema constraints also differ: keep one Zod contract in the application and validate again after the SDK returns. Pin a schema version. Do not let a model invent ticker symbols, product IDs or current prices from memory.

Give models jobs, not a beauty contest

Start with a fast model for the common path, then reserve a stronger model for offline hard-case labeling or failures that cross a measured cost threshold. The threshold comes from latency, accuracy and the cost of being wrong, not from a provider hierarchy.

That is an architecture, not a claim that one vendor is more accurate on your ontology. Benchmark both against your domain, languages and failure costs. Implement one provider first against a shared schema, then add the second as an adapter. The schema is the product. The model is a replaceable worker.

The mistakes that keep shipping

• One sentiment for the entire text. Extract targets and aspects.
• Treating neutral as uncertainty, or mixed as confusion. Those are different states.
• Treating emotion as sentiment, or positive company language as a bullish stance.
• Trusting a generated confidence number. Calibrate it.
• Letting the model invent entities. Resolve them in your application and pass candidates in.
• Using sentiment as moderation. Toxicity, abuse and self-harm need their own classifiers.
• Acting automatically on the result. Consequential steps go through a policy and, when the cost of error is high, a person.

Start with a layer you can measure

If I were attaching this to a live assistant, I would not begin with a dashboard of every emotion. I would begin with a provider-neutral schema, a fast structured first pass, exact-evidence validation, one escalation path, a small labeled suite, and a policy that can refuse to act. Aggregation by target, calibration curves and a second-provider adapter come after the first job is boring.

That is the same standard I apply to the rest of production AI. Give the model a defined job. Bound what it may conclude. Observe when it should stop. Keep a person accountable for the action that follows.