How "AI follow-up" decides whether and when to ask, and why that makes it faster and cheaper

New to AI follow-up? Read how to add AI follow-up questions first to see how to set it up in your survey, or see the AI follow-up questions overview. This post explains what happens behind the scenes.

Two jobs: deciding and writing

AI follow-up reads what a respondent just wrote and asks a relevant next question. A creator can also limit it, for example "only follow up on negative answers". So every request has two parts: decide whether a follow-up should be asked at all, and write the follow-up.

The two parts need different tools. Writing a good question needs a language model and a long prompt, since the prompt carries the framework, tone, previous questions and the sentiment rules. Deciding is different. It is a narrow question with a short answer, such as "is this answer negative?" or "does this person want to stop?", and it needs none of that context.

So we split the two. A small decision step runs first. If it is sure no follow-up should be asked, the long GPT prompt is never sent, and those tokens are never spent. The decision step also runs on a lighter, cheaper model than the one that writes the question, so even a request that is not skipped pays little for it.

It is faster too, and the respondent is waiting on the screen. A skipped request is answered in about half a second, where a GPT call takes about one to one and a half seconds. A request that passes the decision step waits that extra half second before GPT runs, so the speed-up applies to the requests that are skipped.

Where Jev fits

Jev is a decision model. We give it some text and a typed question about it, and it answers with a choice or a yes/no probability. It does not write text. We use it for one reason: it is a good fit for narrow classification questions, and those are the questions that come before a follow-up.

• If Jev is sure no follow-up should be asked: stop here, no GPT call.
• If Jev says yes, or is unsure: GPT writes the follow-up.

GPT still writes every follow-up the respondent sees. Jev only answers whether one should be written. We run three checks, called gates.

Jev gives a probability with every answer, not just a verdict. A gate acts only when that probability is very high or very low, so an unsure answer always goes on to GPT.

Gate 1: Sentiment

The job. A creator can say "follow up only on negative answers". The gate reads the answer and labels it positive, neutral or negative.

Why Jev. This is classification with fixed labels, which is what Jev's Choice question does. Jev labels the answer first. If the sentiment cannot pass the filter, there is nothing to write, so the GPT call never happens.

How it stays safe. We skip only when Jev is highly confident and gives the sentiments the creator allowed a very low probability. An unsure label does not skip. It goes to GPT, which still checks the sentiment itself.

The criteria are the same rules the GPT prompt carried, written as definitions. For example, an answer that names something the respondent wants added, fixed or improved counts as negative, and sarcasm about a bad experience is also negative.

Gate 2: Junk answers

The job. Catch keyboard mashing, test entries like asdf, repeated characters, and text that has nothing to do with the question.

Why Jev. It is one yes/no question: is this a genuine attempt to answer? Jev's Noul question gives a probability for exactly that. We treat the answer as junk only when the probability that it is genuine is very low.

What it does not catch. Short real answers such as "No", "idk", "n/a", a selected option, a number or a name are genuine. A respondent who declines to say more is a job for the stop gate.

What happens next. A junk answer is not a silent skip. GPT is asked for a short question saying the answer seems unrelated or unclear. For that call the sentiment rules are dropped from the prompt, because a junk answer has no sentiment worth filtering on. If GPT gives nothing usable, the follow-up is skipped.

From the second level of follow-up on, the check also receives the respondent's first question and answer. A short follow-up like "Can you elaborate?" cannot tell by itself whether a reply is off-topic, and the original context fixes that.

Gate 3: Respondent wants to stop

The job. End the chain when the respondent says they are done, wants to skip, or refuses to add more. It runs from the second level of follow-up, since that is the first time the answer is a reply to a follow-up.

Why Jev. People say they are done in many ways. "That's all I have" and "nope, nothing else" mean the same thing, and no fixed list of phrases can cover every wording. Jev reads for meaning, not for exact words. It also reads the question together with the answer, which matters: a bare "No" answering a yes/no question is an answer, while "no" in reply to "Can you tell me more?" is a refusal.

How it stays safe. We end the chain only when the probability that the respondent wants to stop is high. Words like "done", "all" or "stop" inside a longer real answer do not count.

Jev decides. GPT writes. Neither does the other's job.

Three answers, three outcomes

Say a creator turns on AI follow-up for the question "How was your checkout experience?" and asks for follow-ups only on negative answers. Here is what happens to three different answers.

Every one of these decisions is made before GPT is asked to write anything.

One request for all gates

All three gates read the same answer, so they share a single Jev request that carries all of their questions. Only the sentiment gate can read a different answer, when the creator limits sentiment to one follow-up chain, and then it gets its own request.

If more than one gate is sure, the first verdict wins in a fixed order: stop, then junk, then sentiment. A respondent who asked to stop is not asked anything else, and junk is not read for sentiment.

state     = { question, answer, problem? }
questions = { stop: Noul, genuine: Noul, sentiment: Choice }

responses = system_one(state, questions)     # one request, many verdicts
decision  = first sure verdict in [stop, genuine, sentiment]

Failing safe, and privacy

A gate that blocks real follow-ups is worse than no gate, so every gate does nothing when in doubt. If the gate is switched off, the API key is missing, Jev times out or fails, or the verdict is not sure, the decision is "do not skip". GPT then runs as normal and writes the follow-up.

Nothing from the question or the answer is ever logged.

The result is that GPT is only paid for when there is a real question worth writing.