Predictive analytics not predictable techniques: What's needed in L&D
Evaluation as traditionally envisaged in L&D is no longer fit for purpose, John McGurk says
Whenever I hear people talking about "evaluation" in L&D, I feel like Bill Murray in the film Groundhog Day. In the film, things become so predictable for him he nearly loses his mind. Mention of Kirkpatrick levels and ROI leaves me feeling as trapped in the past as Bill Murray does. Eventually he manages to move the clock forward by learning. That is what we need to do in L&D. Why?
- Because the challenge for learning metrics is sharper than ever, as more organisations are maintaining investment in learning, meaning the business value of learning is more likely to come under the spotlight
- The trends towards both talent analytics and big data are challenging what we do. The answer to the impact and value of learning lies outside L&D interventions.
- Access to socially networked data and so called unstructured data unlocks the richness of new insights and removes the excuse that it's not possible to measure less tangible issues like collaboration and knowledge sharing. All fresh and exciting horizons for learning.
Analyse this! Learning metrics and analytics under the spotlight
Learning is the key value driver for organisations. That's because learning builds skills, enables change and should increase productivity. The CIPD/Cornerstone Learning and Talent Development survey in 2013 showed that L&D professionals understand the importance of tracking learning metrics. About half always or frequently use HR metrics such as absence, retention, talent and 360 degree feedback. Just under two fifths record the same for business and organisational metrics like profitability, revenue, market growth or service targets (see Figure 1 below). It's encouraging that more L&D specialists are using wider HR human capital and business metrics, and CIPD will be working with the Chartered Institute of Management Accountants (CIMA), to develop common metrics. An easily overlooked point is that while metrics are the raw ingredients of analysis, they are not "analytics". These are linked metrics normally expressed as ratios or percentages. We will look at some examples in future articles.
Figure 1: L&TD metrics in practice
Source CIPD/Cornerstone Learning and Talent DevelopmentSurvey 2013http://www.cipd.co.uk/hr-resources/survey-reports/learning-talent-development-2013.aspx
However, as learning strategy consultant Andrew Mayo was saying back in 2004, the majority of L&D professionals are still pretty much focused on measuring the internal impact of learning as a process rather than as an impact generating change programme(Mayo 2004). For example, the use of ROI after the fact and without a baseline is still widespread (see Kearns 2005 for the right way to use it). In our 2013 survey, about a quarter reported that they always and frequently do this. Just under a quarter use the Kirkpatrick model, yet the proportion who regularly use development data such as 360 degree feedback, psychometrics and learning performance data also amounts to a quarter. This is skewed, and the fact that our analysts were pointing this out a decade ago is why it really does feel like Groundhog Day. There are productive ways to look at how we measure learning using our traditional techniques (see Bellinger's risk focused approach to Kirkpatrick and ROI, for example), but we really need to do more.
Using internal people, performance and business data against business metrics is known as talent analytics. Getting to this stage is overdue because the big data train is already bearing down on us. Boudreau and Jesusathan provide a compelling framework for talent analytics, based on segmentation, optimisation, risk, integration and other key strategic aspects.
Mine that! "Big data": Amplifying the impact of learning
Big data is the generic term for the practice of mining insights from diverse streams of data. It's the foundation of companies like Google and Amazon1. A great example is how Public Health England's (PHE's) Longer Life database provides data on cause of death, links it to regions and cross correlates a lot of other data2. This uniquely impactful source doesn't just give numbers, it gives actionable advice. How powerful could our learning be if we could do that?
Box 1 Big data scenario
Take a bank tracking sales and looking for customer insight. It will be using its data to look for anything from the number of times employees log on to a learning programme on data protection and the number of customers who ticked the box on a company survey which asked about the same issue or a related issue like "I trust big banks with my personal details". That correlation could give us a valuable piece of insight. This would be predictive, in that we would be able to imply from the insight that doing something would lead to a measureable change. That, in essence, is big data.
Big data, however, is not just numbers and it's not always precise or well-defined. Let's look now at what that means.
Capture that! Unstructured data
According to various consultancies, about 80 percent of all organisational data isunstructured. That is, it is text heavy, relates to documents, discussions, conversations and connections. Such data in its raw form can't be stored in the relational database format which number crunchers use. Figure 2 shows how technology company Oracle is developing tools to capture and "interrogate" unstructured data. CIPD is working in partnership with Oracle Corporation to investigate the talent analytics and big data challenge for HR.
These massive flows of data are present in any workplace where ICT is used. This is as true for the biggest global consumer company as it is for an SME employing one hundred people. Unstructured data are loaded with learning insight. Another key aspect of the big data picture is knowledge shared across webinars, devices, websites and the traffic flowing across online platforms such as Twitter and LinkedIn. With integration tools these are currently harnessing in real time the learning dialogues we previously thought impossible. Capturing this is our next big challenge.
Hack that! Social and collaborative learning
In our 2013 Learning and Talent Development survey, we asked practitioners to rate interventions in terms of their social and collaborative impact. Group webinars were rated highest, mentoring and peer to peer learning was also seen as highly social. The most informal type of learning, taking place around central social hubs such as the water cooler or the coffee point came close behind. "Learning on the job" also figured near the top. Social learning is bringing insights previously rooted in leading edge learning theory and academic research to the forefront of practice. Such interactional learning is where the formal or instructional learning and the experiential learning such as on-the-job training (OTJ) and knowledge management comes together and clicks.
Capturing and channelling these social learning flows holds big promise. Such data is not easy to capture but a reflective awareness of it can help practitioners to link to other forms of learning data. Already many applications and platforms seek to tap into social and collaborative data. For example, Chatter, SharePoint and Yammer are widely used and new tools are coming on the market. Integrated talent management systems which link these all together in one place are increasingly used in large organisations too. Innovation and competition will drive the costs down. Collaborative initiatives like CIPD's Hackathon with Gary Hamel, where a community of practitioners put forward solutions and develop ideas, is an example of this. So L&D professionals have to develop a smart and sophisticated appreciation of the new world of rich data led insight.
Evaluation as traditionally envisaged in L&D is no longer fit for purpose. Metrics will come from how learning interacts with a whole range of people, performance and business indicators. This trend will be helped by the greater and more systematic use of talent analytics. None of this will stand on its own, and that should logically demonstrate to us the futility of either "happy sheet" evaluation or baseline free ROI as two extremes of poor current practice.
Make metrics our learning challenge
Learning how to measure the value of what we do is for me our biggest challenge in L&D. It outranks all others. The availability of talent analytics, big and unstructured data and social and collaborative learning have shifted the ground beneath us. Many embrace this but some L&D professionals feel uncomfortable with it. Yet Bill Murray's miserable know-all weatherman only makes progress out of the endless loop when he realises the need to shift his thinking. He enjoys learning to play the piano, making ice sculptures and learning French. Shifting our focus towards our own key challenges will make them opportunities for us too.
This is the first article in a series as we develop our various metrics projects. We will be working with global technology company Oracle Corporation to look at talent analytics and big data across HR. I will pull out some of the key insights for L&D as we progress this work, particularly the obstacles to addressing the three priorities I have outlined: namely silos, skills and suspicion. Watch this space and please tell us what you think.
1. See Mayer-Schoenberger V, and Culier, K Big Data.
2. Thanks to Nigel Paine and Learning Technologies Conference for this example.
Bersin, Josh (2012) Big Data in HR http://www.slideshare.net/jbersin
Bellinger, A (2011). Analytics Beyond Kirpatrick Training Journal June 2011
BOUDREAU, J.W. and JESUSATHAN, R. (2011) Transformative HR: how great companies use evidence-based change for sustainable advantage. San Francisco: Jossey-Bass.
Mayo A, (2004) Return on Investment One Stop Guide (2004) Personnel Management
Kearns P; (2005) Evaluating the RO from learning; How to develop value based training CIPD books
MANYIKA, J., CHUI, M., and BROWN, B. (2011) Big data: the next frontier for Innovation, competition and productivity. McKinsey Global Institute
Mayer Scoenberger, V and Cukier, K (2013) Big Data; A revolution that will transform how we work, live and think John Murray
McGurk J, (2012) Data Smart HR; the new Must Have. CIPD Impact ( Quarterly Journal of Policy and Practice insight) No 40 September 2102)http://www.cipd.co.uk/impactmagazine/40/pdf/impact_40.pdf
Silverman Research (2013) Data Visualisation Seminar London June 27th
A look at the best stories, research and news in HR, talent, learning and organisational development as selected by the TJ editorial team.
Cass Coulston explores recent research into ways of leading and thriving in a hybrid work environment
Natalia Ramsden looks at neuroscience research and provides her view of the best science that supports improved brain function and learning
Vincent Belliveau, Senior Vice President & General Manager EMEA at Cornerstone OnDemand, explores the benefits of internal recruitment
We need to do a better job of preparing young people for the world of work, so they can make informed choices and build fulfilling careers.
Anthony Santa Maria on how personalised learning builds future-ready workforces