How to test social listening tools in Newcastle

What to take away

What Geordie dialect and humour do to keyword matching

Social listening starts with text capture and classification. That much is standard across the industry, as the background on Social media analytics - Wikipedia sets out. The trouble starts when the text is Geordie.

Newcastle upon Tyne speech and writing use forms that national dictionaries and lexicons rarely include. "Gan" for go, "canny" for good or careful, "howay" for come on, "pet" as a friendly address, "netty" for toilet. A tool tuned on national English may score "mortal" as grief or death, and "canny" as cautious rather than positive.

Humour compounds it. Geordie self-deprecation and deadpan phrasing read as negative to a generic sentiment classifier. A post that says a match was "class, like" may be scored neutral or negative because "class" is ambiguous and the intensifier is unknown.

Spelling variants matter too. Local writers may use "Toon" for the city and club, "Mags" for Newcastle United supporters, "Geordies" for residents. A query that only lists "Newcastle" will miss posts that never name the city.

When assessing supplier claims and source questions, ask which lexicons, regional seed lists and languages or dialects the classifier was trained on.

Named Newcastle media and civic sources to test against

A coverage check needs a ground truth. In Newcastle upon Tyne, that means named regional media and civic sources, not a generic national feed.

Use these as your seed set:

Civic sentiment is not the same as brand sentiment. A council announcement about a clean air zone or a bus lane generates anger, relief and jokes in the same thread. National tools that classify by keyword will flatten that mix.

Use population and community data from the People, population and community - Office for National Statistics to weight your sample. If your tool reports 40 mentions of a civic topic in Newcastle, you need to know whether that is plausible against the city's size and online population.

Judging how those figures are produced matters as much as the figures. The Methodology - Office for National Statistics pages explain how official statistics are built, and the same discipline applies to your own coverage counts: define the denominator, state the window and show the query.

A repeatable method for checking national tool coverage

You can run this in an afternoon and repeat it quarterly. It gives you a benchmark for comparability across tools and over time.

The table below is a hypothetical example of the summary format to produce, not results from a named supplier or a measurement of Newcastle coverage.

There is no universal capture-rate or sentiment-agreement threshold established by this example. Interpret your results against the test design, seed set and sources used, and state those limits when reporting.

The Market Research Society publishes resources on AI in research: resources and support | Market Research Society, which are useful when you need to justify your test design to a client or an internal reviewer.

Where national tools miss local civic sentiment

National tools may prioritise high-volume sources and accounts, leaving local sources that carry Newcastle civic sentiment less visible. Test which sources each tool actually includes rather than assuming coverage.

Three potential gaps to test are source coverage, query design and civic context.

First, source coverage. A national index may include the BBC and major national titles but omit ChronicleLive comments, local radio phone-in summaries or council meeting coverage. Check whether each source is available in the tool and included in your test; otherwise its sentiment will not appear in your results.

Second, query design. A national query for "Newcastle" pulls in Newcastle-under-Lyme, Newcastle upon Tyne and Newcastle in other countries. Without a geographic filter, your Newcastle upon Tyne social listening is diluted by unrelated places.

Third, civic context. Local anger about a road closure or a school place allocation is often expressed without the words a national tool expects. It appears as a reply, a nickname or a joke. This is the fake coverage problem in miniature: the tool reports a clean picture because it is not looking where the sentiment lives.

Social grade matters here too. The Market Research Society's Social Grade | Advertising & Market Research | Census Data definition helps you check whether your sample skews toward one audience segment, such as ABC1 professionals, and misses others.

Testing sentiment accuracy on Newcastle-specific posts

A sentiment accuracy test is the only way to know whether labels are trustworthy on local text. Run it on posts you have labelled by hand.

Build a test set of at least 100 posts from the named Newcastle sources. Include a mix: football reaction, council service complaints, transport, local business and general chat. Label each as positive, negative, neutral or mixed, and note any sarcasm or dialect.

Then compare tool output. Report agreement overall and by category. Expect the lowest agreement on sarcasm, self-deprecation and dialect-heavy posts.

Watch for these specific failure modes:

Document each failure with the post URL and the tool's label. That evidence is what makes your report credible, and it feeds directly into the next procurement round.

If you are comparing providers, remember that coverage is only a claim until you have run this test on your own seed set.

Reporting coverage limits rather than claiming completeness

A truthful report says what the tool captured and what it missed. That is more useful to a North East brand team than a single confidence score.

Structure your report like this:

This approach also keeps you aligned with UK expectations. Under UK GDPR and the Data Protection Act 2018, you need a lawful basis for processing personal data in posts. The Information Commissioner's Office publishes guidance on this.

Ofcom and the Advertising Standards Authority shape what you can claim publicly about coverage and accuracy. The Online Safety Act 2023 adds duties for some platforms, which can change what data is available at all.

National social listening tools can differ in their source coverage and classification. This article does not establish how any particular tool performs in Newcastle upon Tyne. Run the seed-set test, show what it captures and misses, and base your recommendation on that evidence.

Common questions