Are psychometrics past their peak? Rethinking organisational surveys in the age of voice and AI

As an organisational psychologist practising for over 25 years, I've used my share of organisational surveys to help get insights into things like culture, team dynamics, leadership attributes, and change management. And why wouldn't I? They are backed by theory, rigorously validated, and easy to scale. When they were developed they were somewhat revolutionary. People were fascinated by the way they quantified and showed patterns related to thoughts, attitudes, behaviours, and experiences.

I've been recently reflecting, though, on whether the very reasons we relied on psychometrics may no longer hold. I get this may be challenging for some. I appreciate that many companies are built on reselling 'validated' psychometrics. My reasons, though, are to help add to the thinking and practice which is at the heart of organisational psychology - to help uncover meaning, then act on it.

The origin of organisation survey tools

It is probably helpful to remember that, before the tools, there was the research to develop these tools. It came primarily from qualitative methods - observation, interviews and ethnography. These insights built theory/ frameworks, which were then used to create the measures.

This saw the proliferation of Likert scales. They were easy to administer, easy to analyse at scale, and easy to crunch the numbers. This meant we had global scores in things like employee engagement, staff satisfaction, leadership domains, etc. This was far easier than interviewing everyone, for example, when trying to determine the sorts of cultural values that were spread across the organisation.

Strengths and weaknesses of organisation survey tools

Let's be clear. Validated psychometrics still play an important role:

1. Efficiency: It is now standard practice for vendors to provide systems to automate the distribution of validated measures and then provide an interactive reporting dashboard. This makes it super easy to look and trends and patterns across time and across groups.

2. Validity and Reliability. One of the reasons people use these measures is that they have been tested for reliability and validity. That said, most measures have not been tested for the critical aspect of predictive validity. This is because it is generally too hard to do so.

3. Standardisation and Benchmarking. Because many of these tools have been used for such a long time, it is possible to benchmark against things like industry norms, or to map against historical trends. While this is helpful from a descriptive perspective, I'm not sure what insight it might bring. I've been in many meetings with executives when they look at the data and the graphs. The conversation tends to stall when someone asks 'what does this mean we should do differently'.

4. Scalability: Because these tools can be deployed quickly across thousands of people they provide a scalable way to capture feedback. This is helpful when a broad perspective is needed.

Despite these strengths, psychometrics have a number of limitations.

1. Reductionism: These measures look at complex aspects of human behaviour and experience, but ask people to answer a predefined set of domains and response options. They assume the areas respondents are asked about are the ones key to them. They also assume organisational life is static, rather than dynamic.

2. Cultural bias: A lot has been written about the cultural bias of psychometrics. They are often developed by researchers from a dominant cultural context, with the measures refined by surveying people within a dominant cultural context. Even if the language is translated, the domains and constructs may not fully resonate with people from culturally different communities.

3. Social desirability: While there are techniques to limit people from answering in a way that may be perceived as 'correct', these tools are still at risk of people responding in ways they perceive as expected. This is particularly true when anonymity is unclear or results are tied to team-level KPIs.

4. Changing work context: The world of work has changed significantly in the last couple of years. Things like the pandemic, social media, and AI are all impacting how people interact in the workplace, and what they are looking for in their work life. However, the psychometrics used today were primarily developed pre-pandemic (or longer ago). As such, what the theory/frameworks indicated as insightful then, may now be tone-deaf, outdated, or just unhelpful.

Enter voice and natural language processing: a new frontier

With recent advances in Natural Language Processing (NLP), we can now capture the complexities and nuances of people's experiences in organisational life - as described by the people who have experienced them. In many ways, then, we have come full circle back to the methods originally used to develop some of those foundational organisational theories of things like leadership, culture, engagement, etc. Further, if the feedback is captured via voice, we are able to get an insight into the sentiment, emotion, and intent. This all gives decision makers deeper insight, and is simply not possible with the standard Likert scale.

We still have the ability to look at the pattern across the key domains, the ability to slice and dice across different groups, and across time. The strength, though, is that we are no longer limited to the domains that the traditional psychometrics have determined as important. Rather, what comes to the surface is what is important for that specific organisation and the people within it.

While this is an emerging area, we are seeing a number of ways that natural language processing is being used in organisational surveys.

Five ways natural language processing is used in organisational surveys

NLP has been used in survey data capture and analysis for a number of years. As illustrated below, these developments have happened over time, as computational power has increased, and the algorithms have become more advanced.

1. Thematic analysis of open-ended comments.

   It is becoming common practice for survey platforms to offer AI solutions for the thematic analysis of open-ended comments. This is generally at a high level though, with little to no ability for the user to customise the model. The vendor will typically use the NLP models that are accessed via cloud providers like Amazon or Microsoft Azure.

2. Sentiment analysis to detect emotional tone

   NLP algorithms for sentiment analysis have been around for some time. Typically, they score keywords on a scale of -1 to +1. The model will have a dictionary behind it with words/phrases that are positive and negative. This will help organisations understand emotional intensity or frustration signals across large data sets. Some providers (e,g. InMoment) have even started to include algorithms that label the emotions.

3. Voice native data capture using NLP pipelines

   Some platforms (e.g. Talkscape) use NLP to power real-time dashboards that aggregate the qualitative feedback into interactive insights. These are used in contexts such as employee culture surveys, 360 degree feedback, change management tools, and DEI insights. The NLP highlights the themes, and will identify when there are trends in themes. Because the voice is retained, the emotion and intent is retained for people to hear.

4. Diversity, equity and inclusion (DEI) insights from language patterns.

   NLP models can be used to capture bias, inclusion gaps, or inequitable treatment within free-text surveys and data capture models. For example, when used to train machine learning models for recruitment, NLP can ensure that the bias that is often in standard machine learning models can be offset through a human-centred approach to train the NLP model.

5. Predictive analytics based on text input

   Finally, NLP uses thematic analysis and sentiment analysis to predict outcomes like attrition, burnout risk, or team performance. They do this by learning words/phrases that correlate with critical outcomes and generate early warning signals. For example, in Talkscape's Voice of Employee platform, organisations can track feedback on domains that are likely to lead to increased burnout. If these themes increase in frequency, the platform will automatically trigger a prompt for action. This means the issues can be addressed before they have an impact on performance. In the research space, the efficacy of NLP approaches has been explored in a range of areas (e.g. selection, job analysis, work engagement).

   
   

Is NLP as 'valid' as a survey?

Validity is, obviously, central to the conversation of the quality of a data capture tool. In the context of a psychometric the relevant issues are things like face validity, content validity, predictive validity, and construct validity. These are obviously important, but they all assume that the measure is looking at the 'right thing'. By this, I mean the things that matter to the cohort in that particular organisation, at that point in time. With NLP you will know that you are looking at things that are valid for that cohort.

1. It captures the themes that people are actually talking about and are important to them at a particular point in time. This is the essence of content validity

2. It allows people to express themselves in their own words. This is the ultimate in face validity.

3. It can detect meaningful sentiment and emotion. This supports predictive validity because the NLP model is able to highlight factors that are impacting positively and negatively on things like employee engagement.

   
   

So...where does this leave organisational psychologists?

Where I see this heading is that the future of measuring things like employee engagement, organisational culture, or leadership effectiveness is not an either/or choice. Rather, it is about integrating validated psychometrics and voice-based NLP insights. Essentially, we are looking at a hybrid model. In this model we have psychometric tools to continue to do what they do best:

• Provide quantitative benchmarks and trend data over time

• Support statistical modelling and predictive analytics when aligned to clear constructs

Meanwhile, NLP-powered voice or text analytics enriches this picture by:

• Revealing the stories, emotions and nuances behind the numbers

• Uncovering emergent themes that are not in the survey framework, but matter to employees

• Providing real-time adaptability by capturing language shifts, culture change, and lived experiences as they evolve.

Together, this allows for triangulation. The quantitative data tells us what is happening, and the qualitative tells us why.

Embracing these new technologies has some implications for organisational psychologists.

1. A move from measurement designers to insight facilitators

   Traditionally, organisational psychologists have invested significant time in designing survey items, validating scales, and interpreting statistical scores. In a hybrid model there is an emphasis shift. One in which there is more emphasis on sensemaking to help leaders interpret both the structured scores, and the unstructured narratives.

2. Developing new technical and analytical skills.

   While survey design will remain important, organisational psychologists will need to develop skills in: data literacy in NLP models (e.g. sentiment score, and linguistic pattern detection); the ability to audit AI outputs for accuracy, bias, and cultural fit; and, competence in integrating qualitative and quantitative dashboard for clients.

This is exciting for the future of organisational psychology. It is not about choosing one method over another. It is about reimagining our toolset to better reflect how people actually experience work.

What if the most valid data......is the voice itself.