Measurement
Part of Measuring social listening honestly, from metric cards to causal evaluation
Keeping a records register behind social listening figures
A records register for social listening figures: what to file for each metric, who supplied it and which change makes the record stale.
A social listening figure should never travel without a filed record. Keep one register that matches every reported figure to the record behind it, the person who supplied it and the change that makes it stale. A figure with no entry is unsupported.
The register does not define the metrics. Metric cards do that. Its job is narrower: showing at a glance which figures can be defended today and which cannot. No value in this text is an observed result, a supplier benchmark or a target.
What to take away
- One register row per reported figurethe record, the supplier, the date received and the refresh trigger.
- Five metrics need five different records, held by five different people, so one shared folder is not enough.
- A record goes stale on a defined trigger, such as a change of query, model, endpoint or protocol.
- A figure whose record cannot be produced is marked unsupported, whatever the dashboard shows.
- The five metrics cannot be combined into one performance score.
What one register row holds
One row per reported figure keeps the register short enough to be read. Each row has six fields, and a blank field is itself a finding that someone should chase before the figure is quoted again.
| Field | Entry | Why it is there |
|---|---|---|
| Figure | The wording used in the report, with its period | A changed definition cannot slip through unnoticed |
| Record | File name, version and storage location | Anyone can retrieve it without asking again |
| Supplier | The person or team who produced it | There is a named person to ask |
| Received | The date the file arrived | Shows how old the evidence is |
| Refresh trigger | The change that makes the record stale | Says when to request it again |
| Status | Current, stale or unsupported | Tells the reader what to do with the figure |
A small team may hold several roles at once. What matters is that one named person answers for each row. How the roles split decides who can be asked, and how quickly.
File the records in the order the report is built
Each metric has its own record and its own owner. File them as they arrive and record the refresh trigger the owner names.
- Returned-item countthe immutable ingestion log from the data custodian. Refresh when the source, query, identifier, geography or ingestion changes.
- Sample relevance proportionthe return frame, selection log or random seed, codebook and adjudications from the analytical assurer. Refresh when the question, query, population, codebook or sample design changes.
- Positive-label precisionthe model version, test set, human labels and confusion matrix from the classifier assurer. Refresh when the model, threshold, language, source mix or label definition changes.
- X impression count for owned poststhe authorised API file from the channel analyst. Refresh when the endpoint, authentication, definition or extraction process changes.
- Intention-to-treat outcome differencethe assignment, intervention and outcome records plus analysis code from the independent evaluator. Refresh on a protocol amendment, allocation failure, contamination or outcome revision. Without randomised assignment there is no row to file.
Two records that are easy to misread
The ingestion log and the selection log each support a narrower claim than the figure beside them suggests. Read them before the figure is trusted.
Returned-item count essentials
- Countdelivered item identifiers
- Populationone authorised source and query
- Periodstated UTC start and end
- Ownerdata custodian
In the ingestion log, look for duplicates, later deletions and unknown locations marked as such, and for test records identified before the run. Delivery failures and deduplication can move the count without moving the conversation. Reading volume as coverage is one of the common measurement mistakes.
Sample relevance proportion
Numerator
- What counts
- sampled records adjudicated relevant
- Unit
- per cent
- Source
- adjudications
- Owner
- analytical assurer
Denominator
- What counts
- all records selected by sampling
- Unit
- per cent
- Source
- selection log
- Owner
- analytical assurer
The selection log shows how records were sampled and who decided what counted as relevant. Sampling error, reviewer differences and incomplete source access all sit inside the share. It describes the returns that came back, not the items the source failed to return, so it is not recall.
What a filed record cannot support
A record supports only the claim it was built for. The test set and labels bound positive-label precision: it holds for that model on that test set, not for other languages, sources or later versions. Each of the AI jobs in listening needs its own evidence.
Impressions are X-defined, and repeat appearances are counted by the platform. Deleted posts, access windows and metric classes unavailable under the authentication used all change what comes back. X sets its definitions on its first-party metrics page. The count is not unique reach, which needs a different method and a different source.
Government guidance on analytical quality assurance says assurance effort should match the risk of the intended use. A figure headed for a client pitch therefore deserves a firmer record than one in an internal note. The guidance does not validate a commercial classifier or set a sentiment threshold, so it cannot replace the test set.
Marking a figure current, stale or unsupported
A figure leaves the team only with its row complete. Run these checks each time a number is reused in a report, a pitch or a plan, then pair them with the release checks.
- Name the record behind the figure and the person who holds it.
- Ask for the record before the figure is quoted, not after a challenge.
- Check whether a refresh trigger has fired since the record was received.
- Mark a figure whose record cannot be produced as unsupported.
- Do not call a sample relevance proportion recall or treat impressions as unique reach.
- Do not collapse the five metrics into a single performance score.
Common questions
What goes in the row when a record cannot be produced?
Leave the record field blank and set the status to unsupported. Without the record, nobody can show what was delivered, sampled or labelled. Ask the supplier named in the row, and keep the figure out of reports until the file arrives.
When does a row become stale?
When its refresh trigger fires. For the model behind positive-label precision that means a new model, threshold, language, source mix or label definition. For the ingestion log it means a change of source, query, identifier, geography or ingestion. Each record has its own trigger.
Does a filed record make the figure correct?
No. It makes the figure checkable. Sampling error, reviewer differences and platform definitions still sit inside the number, and the register only shows that someone can inspect the file behind it.
