The SignalOS methodology

AI answers are not deterministic. Ask an engine the same question twice and it can name different brands, in a different order, citing different sources. One answer is a single draw from a range of possible answers, not a fact about your brand, and a tool that asks once reports that draw as if it were the whole picture.

SignalOS asks each question five times on each engine in every measurement run, and stores every answer separately with its full text. A plan can be configured for deeper sampling, up to 20 answers per question. Every answer to the same question is collected with identical settings, including a fixed sampling temperature, so the spread we record is the engine's own variation rather than variation we introduced.

Budget and volume limits can reduce that depth, but never below 3. When a run samples fewer answers than usual, the account owner is told how many and why. If a run can afford fewer than 3 answers per question on an engine, that engine is not measured in that run at all, and the owner is told which engine, why, and the date measurement resumes.

Visibility is reported as a rate with a 95% Wilson confidence interval and the sample size behind it. For that interval, one sample is one question measured on one engine in one run, and it counts as a mention when the brand appears in at least one of that run's answers to the question. Answers that could not be evaluated are left out of the rate entirely.

Every answer SignalOS collects comes from a new session with its own session ID. Nothing carries over between calls: no memory, no earlier turns, no personalisation, and not even the previous answer to the same question. An answer shaped by an earlier conversation measures that conversation, not how the engine describes your brand to someone asking for the first time.

The rule is enforced before sampling starts, not left to good intentions. Each engine integration declares whether it can guarantee a clean session, and a measurement run on an integration that cannot is stopped with an error rather than sampled.

Each stored answer keeps its provenance: the engine, the model version the provider reported, the session it ran in, the sampling settings, the language and region, and the raw text. In the product, you can open the answers behind a figure and read them.

AI providers update their models, sometimes without announcing it. A change in the model can move your numbers as much as a change in your content, and the two call for completely different responses. So SignalOS records which model produced every answer.

Each answer stores the model version the provider's API reported. When a completed answer reports a different version from the last one observed for that engine, SignalOS records a model-version change, draws it as a marked line on the perception timeline, and raises an alert. The first version ever seen for an engine sets the baseline and is not reported as a change. If a provider reports no version, none is recorded; we don't guess one.

To divide history into model eras, a transition is called only when the 5 requests before it and the 5 requests after it are each at least 80% one model, and those two models differ. A single stray answer never splits a trend. When impact analysis asks whether your work moved a number, a model change in the same period is flagged as a possible confounder.

A measured number is something SignalOS observed: an engine was asked, its answer was stored, a classifier scored it, and you can read the answer. A modelled number is estimated from measured history, such as a forecast of where a metric is heading. It describes something that hasn't happened, so it cannot be observed.

Modelled values carry a visible ~ Modelled marker: on charts, alongside a dashed line, and in generated report narrative wherever a modelled result appears. The tilde is a shape rather than a colour, so the marker survives greyscale printing. A projection is never presented as a measurement.

For what each part of SignalOS does, and which parts measure rather than model, see all features.

SignalOS classifies every answer: whether your brand appears, how prominently and how favourably, which sources are cited, and why a competitor was recommended instead of you. Those classifications are rules, built from cue words, position in the text, and matching against your domain and your competitors'. The same answer gets the same label every time, and any label can be audited.

One classification can be refined by an AI model: the reason a competitor was recommended instead. The model must quote the passage that justifies its reason, and its result is kept only if that quote appears word for word in the answer. Otherwise, or when no model is available, the rule-based result stands.

Accuracy is measured, not asserted. A person labels real answers with the correct classification, the classifier runs on the same inputs, and precision, recall and F1 are computed from where the two agree. Three classifiers are evaluated this way: brand presence, mention sentiment, and the rule-based reason a competitor was recommended. Results appear on the methodology page inside SignalOS for team members with evaluation access. Below 20 labelled examples for a classifier, that page says there are not enough examples instead of showing a number.

A scheduled check compares each classifier's measured accuracy against a floor of 80% and raises an alert when one falls below it. This page does not print an accuracy figure: the figure that matters is the current one, computed from the current labelled set, and that lives in the product.

Don't take the numbers on trust. Pick a question from your question set and ask it five times yourself, each time in a new conversation with no history. Note which brands appear in each answer, then compare that with what SignalOS reports for the same question and engine.

Expect the pattern to match, not the exact words. Your answers will vary for the same reason ours do, and a consumer app can search the web or draw on your account in ways an API call does not. If the pattern differs from what we report, open the answers behind our figure and compare them side by side: each one shows the engine, the model version and when it ran.

Every change to a formula, threshold or decision rule gets a new methodology version with the date it took effect, and each stored answer keeps the version that produced it. A number can always be traced to the rules that were in force when it was measured.

Research we publish on Insights states its method, and what it cannot conclude, in the same way. If anything on this page doesn't match what you see, ask us.