24 August 2021

How to predict and improve employee retention with data

How can data and predictive analytics help HR teams manage and improve employee retention and shape culture? Rob Robson of The People Experience Hub shares his thoughts


Rob Robson

Rob Robson


Employee engagement HR transformation Recruitment and retention


How can data and predictive analytics help HR teams manage and improve employee retention and shape culture? Rob Robson of The People Experience Hub shares his thoughts

Employee retention is one of the most widely sought-after ‘people’ outcomes, irrespective of organisation type or industry. Losing employees can cause disruption to employers in many ways – the loss of skills means existing employees may need to be upskilled and the cost of replacing employees stands at an average of £11k in the UK.

As a result of the pandemic and Brexit, staff shortages in a range of industries has fuelled a new ‘war for talent’. In such an environment, it’s understandable that businesses will seek to minimise turnover across the board, but it’s important to consider whether that’s a desirable goal. Many organisations treat a certain level of ‘churn’ as natural or even desirable, in order to keep freshening up the talent pool, but of course, there’s no magic number.

And of course, you must be careful where to invest your time and effort. It’s important to have a clear idea of what skills you require now and in the future, in order to fulfil tactical and strategic plans or minimise business disruption.

In order to be efficient and effective at retaining employees, data is critical. If you can use data to clearly articulate the problem to be solved, potential solutions will become apparent. The problem that many organisations have is understanding the problem in the first place. When they struggle with that, they often jump to solutions and find them ineffective.

Here, we share the ways that HR teams can gather and use data effectively – rather than suggesting strategies – to improve employee retention.

Start by understanding your workforce needs

Short-term (resource) and strategic workforce planning is critical, in many businesses, to ensure that products and services can be supplied in the right quantity, at the right time, to the right quality in the right markets.

Effective resource planning requires a blend of metrics from across the business, while strategic workforce planning involves factoring in a variety of long-term scenarios or assumptions and analysing the impact on that data.

When it comes to understanding who (what skills) and where you need to focus your retention efforts, both are important. In the short-term, your aim may be demand fulfilment, but in the longer term you may have a broader range of goals, such as improving diversity.

Predicting turnover (and intentions to stay) using historical data

Predicting turnover means understanding patterns and trends, to better understand current issues and what’s most likely to influence turnover in the future.

Descriptive versus diagnostic versus predictive analytics

At the most basic level, you can observe patterns in your data, looking for trends and areas of concern. You can see differences and, to a certain extent, relationships in the data.

However, if you are reliant on descriptive analytics (percentages, graphs and charts) you may be misled. Differences between groups may look more significant than they are, meaning that you might act on chance effects, or attribute cause and effect to relationships that simply doesn’t exist.

Using statistical analysis helps to drive more trustworthy observations. However, you still have to be careful. Simple statistics such as correlations can also be misinterpreted. Take, for example, US spending on science, space, and technology, which correlates with suicides by hanging, strangulation and suffocation.

Or, perhaps, per capita cheese consumption and the number of people who died by becoming tangled in their bedsheets.

These examples, courtesy of Tyler Vigen’s spurious correlations, highlight the problem when two unrelated variables are observed to be trending in the same direction.

Diagnostic analytics use more sophisticated tests

In order to get beyond these kinds of problems, it is important to use more sophisticated analysis, specifically multivariate analysis that are often described as diagnostic analytics. Furthermore, because correlations are a form of statistical analysis (with the equivalent for differences being a t-test) it is important to start using multivariate statistics.

Multivariate statistics analyse relationships or differences between numbers of variables, at greater or lesser degrees of complexity, taking into account the effect of the other variables.

For example, your employee engagement survey may measure a range of ‘drivers’ as well as an engagement index. The individual correlations between each driver and engagement might tell you one story. When you combine them into a multiple regression equation, you might well see that some drivers are not really driving engagement at all.

True prediction requires machine learning, but do you need it?

The difference between diagnostic and truly predictive analytics is the addition of machine learning, which uses large historical datasets and algorithms to better understand what might happen in the future, rather than what has happened.

However, for most businesses, an accurate diagnosis of the issue represents a big step forward.

What data should you use for diagnosis?

Employee survey data

Given that I’ve already mentioned employee surveys, let’s start there. Many employee surveys include questions such as “I intend to work here in twelve months time”, which is fine as a transactional, short-term measure of retention, but it doesn’t explore why they intend to stay. Perhaps there’s a recession or a pandemic, and employees will look to leave when it blows over.

This is where emotional commitment to the organisation (a key element of employee engagement), its purpose and mission might be more helpful indicators of long-term likelihood to stay or leave.

Combining with actual turnover

Survey questions that aim to understand people’s experiences of the organisation can also be combined with actual data. In an annual survey, for example, you can compare demographics or business areas that have higher or lower turnover levels. Although analysing these things at group level may be less statistically powerful than at the individual level, it is still useful.

Exit interviews and surveys

Another option is to ask those people who have left, directly or indirectly, why they left. On one hand, directly asking people their reasons for leaving in an exit survey can be a rich source of information if they are willing to open up in an exit interview.

Qualitative or quantitative?

While handling qualitative data from interviews can be challenging (though technology can certainly help), perhaps the biggest problem with exit interviews is getting people to sit down for long enough and be truthful enough to draw meaningful conclusions.

However, although automated exit surveys may lack the potential for rich insights that an interview might provide, they have many potential advantages. In around ten minutes, an online exit survey can ask about a range of experiences and provide quantitative data that enables more generalisable insights.

Stay interviews

Many organisations, in addition to or instead of exit interviews or surveys, are now using stay interviews. Stay interviews allow a completely different perspective by focusing on those people that actively choose to stay in the organisation and understand what drives loyalty rather than the decision to leave.

While both perspectives have their uses, the reasons for staying may be more powerful, and are currently less commonly used at present.

Segmentation is important

It is important, when using any form of survey data – on its own or in combination with actual data – to segment the data. You might have a demographic, for example females in their 30s, that you know to be leaving at higher rates, and therefore the ability to drill down into the data and perform analysis is important. At the same time, you might not have a group that you know is leaving in greater numbers, but your analysis might suggest that a particular group is a higher risk.

Combining demographics can provide a little more sophistication. For example, in one organisation we had a particular problem with people in one city that were long-serving. With a lack of alternative employers in the city, they felt trapped and unwanted. In this case, the client had a choice to try and engage them more effectively or provide a ‘soft-landing’ to those people that didn’t really want to stay. It was the opposite of a retention problem!

In this case, we combined demographic data with qualitative, psychological data, to create a rich picture and create more compelling options for the company.

Accessing analytical capability

Without data, any efforts to retain staff are like using an elephant gun blindfolded and hoping that at some point it will hit the target. They will be ineffective, wasteful or both. By adding layers of analysis, from descriptive analytics (what you can see on the page) to predictive analytics (using statistical analysis) you can add accuracy to your efforts.

But what if you don’t have access to sophisticated analytics capabilities? While it’s not realistic for many outside the largest companies to have extensive in-house data science capabilities, and while good people analytics or data science specialists are at a premium, the tools and technologies available (such as IBM Watson or Microsoft Power BI) mean that analytical tools are more accessible and usable for non-experts than ever before.

It is important, therefore, to consider your people analytics requirement as part of your people or HR strategy, and to start building what capability you can by recruiting, developing your own people, or drawing upon your HR tech suppliers.

Rob Robson is Director of People Science for The People Experience Hub, which helps organisations transform the experience of their people through their flexible employee feedback platform and expert support.