Building a Training Data Architecture with Learning Analytics

Hello and welcome to today's Training Industry Leader talk on measurement and data to support the training function sponsored by Administrate realize it and a Lego. I'm Elizabeth Parker, director of Marketing Training Industry, and I'm so happy you all decided to take some time out of your day to be with us before we dove into our first session. I would just like to review our housekeeping items so that you are getting the most out of your time with us. Throughout today's event. Please feel free to use your chat window to collaborate with other attendees or to chat with myself and our speakers. And use the Q&A window to direct your questions for our speakers. I'll be monitoring your comments and saving questions for Q&A. Towards the end of the session, I do encourage you to share the information you received today via social media by following our handle training industry.

While we will be engaging in that Twitter conversation throughout the event. So please feel free to join in there at the end of your time with us. You'll notice a short survey has popped open in your browser We greatly appreciate your thoughts on today's event. And lastly, all of our sessions are being recorded and they'll be archived on training industry account. You'll receive a follow up email from us with a link to On-Demand sessions as soon as they are available. If this is your first event with training industry, a special welcome goes out to you Training industry exists to support the learning leader. We offer timely and insightful information on the business of learning through live events like today's virtual conference as well. As through website and magazine articles, research reports, referral services and our podcast, just to name a few.

You can find all of the ways that we connect you with expert perspectives in the industry app training industry icon. Now, of course, today's event is made possible by our sponsors, and before we get started with our first session, here's a brief message from straight

The role of training is changing. No longer is training simply fulfilling business objectives. In many organizations, it is helping to guide the direction of the business. And for that kind of impact. You're going to need more than an LMS That's where Administrate comes in. And ministries. Holistic edtech infrastructure supports your entire training operation with resource management, task automation system integrations and delivery of training. But the most important thing you need to remember about Administrate is that we help your training team drive business results

all right. And with that, it is my pleasure to introduce you to our first speaker, John Peebles. John is the chief executive officer and Data Insight Matchmaker of Administrate, an innovative training management platform for enterprise learning and development operations. Prior to Administrator John helped found Century Data Systems, a health care intelligence company based in Fort Lauderdale, where he served as chief information officer and vice president of operations. While the company grew to more than 30 million in revenue within five years. He is passionate about education, teamwork, technology and mental health in the workplace and often speaks on these subjects around the world. John, great pleasure to have you with us today for this event. I'm going to hand things off to you now.

All right. Thanks, Elizabeth. And welcome, everyone. I am just getting my screen shared here and then we'll get rolling It's really good to see everybody from all over the world in chat. I am broadcasting to you today from Europe in Scotland, sunny Edinburgh, where at least right now, it's sunny. It'll probably be raining soon and then it'll be snowing and it'll be sunny again. And that's how we roll here in Scotland. So great to have all of our international colleagues on the line and also for those of you that are in states. Obviously, my accent is American and we do a lot of business in the U.S. and get to go over there quite a bit. And I'm really excited to speak to you today. Because this is kind of a mash up of two of my favorite topics that would be learning analytics and basketball.

And this is actually kind of a bittersweet thing or topic for me to discuss today because I am a lifelong North Carolina State Wolfpack fan. If you don't know anything about college basketball, just know that within the state of North Carolina, there are four schools that are known as Tobacco Road, Wake Forest University, of North Carolina, the Duke Blue Devils and the North Carolina State Wolfpack. Our friends at Trent Industry are familiar with this because they're based in Raleigh. These schools are very close together. And I'm seeing already some, you know, annoying Tar Heels fans in the chat and some great folks from the Wolfpack. So always good to see say hi to you. But this is bittersweet because I'm a huge NC State fan and we're going to actually be talking about a Duke player and how they relate to training and learning analytics.

And these ideas that I'm going to talk to you about today aren't mine originally. This comes from an article that I actually read about ten years ago, and it was written by Michael Lewis. And you might be thinking, Hmm, data analytics applied to the game of basketball. That kind of sounds familiar. And this was an article that was written by Michael Lewis, who also wrote the book Moneyball, which was turned into the movie Moneyball and talks about how analytics were first applied to the game of baseball and basketball and many other sports followed the lead of of the baseball teams that he talks about in that book and in that movie. But the game of basketball is slightly different, and we'll kind of get into why. Before we dove in, just want to talk about the prequel to the talk that we're going to give today, and that is this idea of noise versus signal.

What questions do we want to answer as leaders of learning? We have built out a couple of resources that I'll share with you in just a second. But before we do that first poll question for the audience today, how many learning tech tools or systems does your training program use? These can be business systems. These can be learning systems. These could be things like content authoring and so forth. You know, how many do you use? A, five or less. B, six to eight. C, nine to 11 or D, 12 or more. So give you a couple of minutes. What we tend to see is many folks out there in the industry have built and bought more and more tools that the answer is often what we see, particularly at large organizations, organizations that have had a really long standing training function.

You know, just quickly, which of these best fits your environment C, D, by far this time, yes, it can get pretty crazy. The more systems that we had in All right. So while people are yeah, I'm going to go ahead. And in the poll, looks like we've got most most of everyone's voted. So here's a look at those results. All right. So it looks like we have folks in the five or less category at the top, but six to eight, nine to 11 and 12 or more are still representing a huge chunk of their answers. And that more or less lines up with what we tend to see. So the prequel to this talk about building out a training architecture is how do you find a way to get signal out of all of this data and all of these systems and all of this noise?

And on the left hand side, you can see what is probably a pretty typical environment for learning and training operations, depending on what size of company you are. You might have a few these systems maybe less than five, you might have 12 or more, but often they are kind of glued together with people and people are either moving things between systems manually or retyping stuff sometimes or moving things with spreadsheets. It can be a real mess, but that's what we tend to see when there's a lot of learning systems involved in. Actually, bizarrely, there's this correlation where the more learning systems that we add to our environments, the more difficult it becomes to deliver quality training. And so the right hand side is kind of what we would like to see within a training operation. That is, there's one system of record that everything can plug into.

And if you're curious about exactly what this means to you and to your operation, how to achieve this, we've got a guide that is a business intelligence guide for learning and training operations. And you can click and access and we'll send it out in email. But it's it's a really powerful way to look at how we can build out a system of record, capture the right piece of information, and then find the questions that we want to ask. And the talk that I did about this previously. You know, we're talking about basketball today, but before it was all about serial killers. And after that, it was made clear to me by our team that all of my talks have to be submitted for pre-approval no less than two weeks in advance. And they want me to reiterate that it's about solving crimes not necessarily committing them.

So if you're interested in that, check that out. That's the prequel to the talk that we are going to be covering today, which is a training data architecture, how we answer questions and how we can drive decisions with the answers to that score, to that question. So we like to think about key elements of learning analytics, architecture within this model. And the idea is that there will be some sort of catalyzing change that happens to your organization or to your business or to your team. And that will then kick off this idea of we need to start answering these questions and we need to start figuring out what we're going to do to respond to this catalyzing change. And so catalyzing change is the first part.

And we can start building out descriptive analytics. And we'll talk about what those are and some examples of that and why they're useful. Then element three is diagnostic analytics, predictive analytics prescriptive analytics. And finally, hopefully, if you've got your data architecture to the point where you can answer all of these different types of questions and derive these types of analytics, you will find yourself with a future proof data architecture that can respond to the next catalyzing change. So let's dove in. Let's go back to basketball. And if you rewind to kind of the mid 2000, the Houston Rockets there, an NBA team, they had a problem and their problem is maybe familiar to some of us.

All of their budget have been spent on two guys, two superstars, Tracy McGrady in Miami. So, you know, it wasn't wasted. And we find that within learning operations a lot, all of the money or the majority of the money have been spent on a couple of systems or a couple of initiatives. That is that is something that we often see and there's nothing wrong with that. But they still need to field an entire team and what they wanted to do is they wanted to find undervalued players that could help them win a championship, that could support these superstars and get them to the point where they could win championships. And so what they did is they sat there and they thought, what if we took analytics and data and started applying that to the game? Of basketball?

What would that look like and how could that make us successful? And kind of jumping back into our environment a little bit, not all of us are games of NBA teams, but many of us have problems that look a little bit like this. You know, how do we shorten and improve the employee onboarding experience? That's a problem that a customer bars Brunswick, which you may have seen in that video ad. They wanted to know how can they make sure that the employee experience from the date that they're hired to the date they become a factory worker out on the spot on the assembly line was shorter and better. And so they went they wanted to solve that problem closely related to that Brunswick, which is in particular the division that we were working with was Boston Whaler they wanted to know how they could reduce manufacturing errors.

That's a very common problem that our manufacturing customers see. And manufacturers all around the world think a lot and spend a lot of money on this. How do we reduce manufacturing errors? Maybe you want to improve employee scheduling decisions. So manufacturers, again, often want to say, look, all things being equal, maybe we should schedule the person that has done better on their training or progress further on their training. How do we do that? Because that would really help with the overall quality at the factories producing, preventing employee churn. This is one that we probably are all very familiar with. This cuts across many different industries. How do we do that? How do we prevent customer churn? We have customers that are running software, software as a service companies or a subscription based business model.

And the last thing that they want is to spend all this money getting this customer through the door, getting them up and running with the software and so forth, and then find that they may churn or not renew their contract. That's disaster for subscription business. How do we prevent that? And maybe how do you lead a strategic initiative? We have seen that many organizations out there have these very, very important initiatives they have to get done. Where is the training and learning operation in the mix for those strategic initiatives? Often it's not at the forefront, which is where we think it should be. And you'll probably notice that we skip the first two, which is decreased support tickets and increased customer net promoter score Those are actually two problems that we had here at Administrate we wanted to figure out how we could get the support ticket volume for customers to go down and how we could get customers to recommend us to a friend of our colleague more often.

And we solve that problem with training and with data that supported the training that we were doing. And it's just a nice reminder that even for us, training and learning operations and the data analytics and so forth that drive these decisions can be applied to our business. And so that was a really neat outcome where we dove into the data. We saw that customers who had been provided extensive access to Administrate University which is what we call our training program, basically reduced the support tickets by more than half and dramatically increased customer net promoter score. These are real tangible business outcomes that you can achieve if you can actually interrogate your data and you have your data architecture set up to answer these questions

So basically the Houston Rockets going back to basketball, they had this problem. They need to find undervalued players to support their superstars. And so they started crunching the data. And what they found was a guy named Shane Battier. And the reason that this is bittersweet for me is a North Carolina state Wolfpack fan is because Shane Battier played in college for one of our bitter rivals, the Duke Blue Devils. And he was a phenomenal college player. He was excellent. He one player of the year. They won a championship with Shane Battier, and he is involved in some memorable moments in college basketball, including coming back from being down by like ten points with less than a minute to go against Maryland. He was an exceptional college basketball player. He got drafted six in the first round to the NBA and then he became, well, kind of a mediocre basketball player.

And, you know, he he physically was not as tall as he needed to be. He wasn't as physically gifted as other NBA players. And that's uncommon when you go from the college level to the pros. But Shane Battier was out there. And what happened was the Houston Rockets started to try to find undervalued players. And Shane Battier is name started coming up more and more the more that they drove in the data. And I'll take you through why that was and why it was actually very difficult for other teams to figure this out. But quickly, second, Paul, question. Let's say your CEO comes in and says, what does our training data tell us about risk and opportunity for our organization next quarter? What is your reaction? Maybe you've got some great thoughts.

Give me 2 minutes to send your report. That could be one reaction. Maybe it's I have some great thoughts. Give me two days or weeks to get a report. Another option, I have a lot of thoughts, but how do I make the data tell the story? That's option number three. And then the last one, I have 99 problems accessing the training, training data. I want our 98 of them so give me a few minutes to fill that out. Your CEO comes in. What does our training data tell us about risk and opportunity next quarter? And it's kind of reminds me of what happened in our organization, where we started crunching the data that I mentioned about reducing support tickets and increasing customer NPS And immediately it was super obvious that we needed to double down and invest in that area of our business.

Whereas if we hadn't seen that data we probably would have just known that we needed training, but we wouldn't have necessarily doubled down like the data was telling us. Getting a few questions about whether the recordings and the slides and all that will be available. The answer is Yes, they will. Don't worry about that. This will be recorded. If you have to step out, answer a phone call, get a delivery, pick up a kid, whatever. It'll all be recorded. So maybe just another minute or so. Yeah. There we go. Polls closed. All right. Looks like option over one. Give me a 2 minutes. To send your report. 7%. I have some great thoughts. Give me two days or weeks. 17% pass.

Almost half of a lot of thoughts. But how I make the data tell a story And lastly, 99 problems in accessing the training. Do I want or 98 of them? That jives pretty closely to what we see out there in the market. And appreciate you taking a minute or two to answer that.

So back to these key elements of learning analytics, right? We've got a catalyzing change. We need to find an undervalued basketball player or we need to reduce our manufacturing errors or whatever it is within your organization. It's a business priority. It's an operational priority. We need to do it. How do we start down this path? And well, the first time, the first thing that we normally look at are descriptive analytics. And back to basketball, right. This is a quote out of that New York Times article. For most of its history, the game of basketball is measured not so much what is important as what is easy to measure. Points, rebounds, assists, steals, block shots. And these measurements have warped the perceptions of the game. Now, one of the interesting things about basketball is all of these stats can actually be gained by individual players.

And you can have a great game from a stat perspective. You could score a lot of points. You could get a lot of rebounds. You have a lot of assists. It doesn't matter if you're not familiar with basketball and these aren't making any sense to you. But just to say is you can do all these things and actually still lose the game. And you could have an amazing statistical career averaging huge numbers of points and your team could have finished last place. Every single game that you played or lost, every single game you played or finished last place in the conference or the season And that is unusual in the game of basketball. But I think that the game of basketball in this respect actually mirrors or marries up to the game of life. Because if you think about descriptive analytics and how they've warped our perceptions of the game of basketball, let's look at some examples of descriptive analytics in our own industry.

And that is a lot of time and attention and focus goes into things like learner attendance, satisfaction scores, completion rates, course costs, correlations. These are descriptive analytics about what happened. And they don't really answer the question of did we achieve our objective? Right. So learner attendance, very, very straightforward. We know the learner attended. We don't actually necessarily know how well the learning impacted their job. Same thing with satisfaction scores and whatnot. These are the analytics that I would contend have warped our perception of what learning means within our industry, just like within the game of basketball, moving on, getting past descriptive analytics, which are things that we often see measured in the tools that we buy.

Each tool have its own measurements and so forth. There are the diagnostic analytics and what we mean by that are you know, in the game of basketball diagnostic analytics, the Houston Rockets are saying if you want to know a player's value as a rebounder or when there's this miss shot, you want to go after and grab the ball because you might get another opportunity, or at least you can then take the ball down to your side of the court and score. You need to know not whether he got a rebound, but the likelihood of the team getting the rebound when a missed shot enters that player zone. So this is really, really important. When Shane Battier was on the floor, it was far more likely that his team would get the rebound, not necessarily that he would get the rebound. And one of the reasons for that is there are things that Shane Battier did as a player that actually were not tracked by any stat line within the game of basketball, things like deflections.

He was famous for basically getting in the right position, even though he wasn't as tall as he normally would be in that position, jumping up. He wouldn't be able to grab the rebound himself, but he could tip the ball with his hand to a teammate or at least tip the ball to an area where a teammate could run it down and grab it. And so the team, whenever Shane Battier was on the floor, would get far more rebounds than when he wasn't. And that was a key thing that the Houston Rockets picked up when they started analyzing data beyond descriptive analytics back to our industry. Here's some examples of what might be descriptive diagnostic analytics in our in learning, in operational environment. So mapping business KPIs to individual contributors and teams. How do we know that when we provide a piece of learning, when a student goes on a course, when a learner completes an objective that are actually materially manifests in both their own work and their teams in work, how do we know that as far as learning professionals, pinpointing missing elements in a course, maybe everybody is going back and doing great work and doing a really good job and they've improved across the board, except for one key area of the job or the role.

And that's not well covered within the course that we've just put them through. How do we tell that? How do we measure that? Maybe we need to find a weakness with instruction. So from time to time or instructors will get out of date, our course materials will get out of date. How do we identify that and correct it efficiently? And then showing correlation versus causation. Right. What is causing this or what is correlated with this? These are examples of diagnostic analytics that help us solve problems within our operational and business environments. Okay.

And here's an example of what this might look like. This is a customer of ours: "partnering with Administrate allow us to bring more robust data collection reporting to our internal operations. We've improved our reporting capabilities, our ability to track the entire learner and account journey throughout our training and streamline our operations to better pursue opportunities for growth in the future." That's what we'd like to see, right? We like to see the idea that learning analytics informs better learning operations, which then informs business change in organizational change. And Merav and her folks over at Data Society are really doing a great job leading the charge in this area. It's great to be partnered with them. So we do have we have catalyzing change. We have descriptive analytics, diagnostic analytics.

We know there's a problem. We want to solve it how do we get to predictive analytics and what would that look like? So here's a picture of Shane Battier guarding Kobe Bryant. And actually there's a couple of famous games from this context in the NBA that happened, which is Kobe Bryant was a massive superstar for the Lakers, one of the best players ever played the game of basketball but what happened was when the Rockets would play the Lakers, they would have Shane Battier guard Kobe Bryant, which it would actually mean that Shane Battier was playing out of position. And the reason that they would have him guard Kobe Bryant was because they could feed him data and analytics about Kobe Bryant's tendencies and dispositions and shot selection and all kinds of things that he would study before the game and then go out and implement when he was on the court.

Here's an example. We may Kobe Bryant go to his left hand. He'll score 44% of the time. He goes right hand. He's going to score 56%. So you don't need to be a math genius to understand that the defender should take Kobe to his left. And this is a really interesting example where again data that was not normally available in the stat line i.e which way to Kobe Bryant drive was going to the left? Or was he going to the right? The Houston Rockets found that data, tracked that data, put it into a system that they could mine, and then it helped them provide predictive analytics that would help them beat the Lakers. We were a much better team often when they played and and got a really outstanding result. So how does this look in our world, right?

Training surge forecasts? How do we as learning training leaders figure out when is there going to be a training surge? Sometimes we have leading indicators Sometimes it's obvious like maybe a pandemic or opening a new factory or we're acquiring a company, but it often isn't obvious. And how do we figure out when that's going to happen and get in front of that problem? Resource Impact Forecasts Similar to training surge. Many customers of ours have sophisticated resources that they need to bring to bear in order to deliver training. It's not always just e-learning. It could be simulators. It could be things like space, special equipment that needs to be around. If we have a business surge and we can see that happening, how do we then know that there's going to be a training surge and then that's going to how is that going to impact on our resources and basically just boiling all this stuff down to figuring out how do we build a predictive response model that can help us further the organizational goals and do so with confidence and be able to lay this out to our leadership and justify budget, justify investment, etc..

All right. So another poll question. When it comes to learning analytics, I would say our training program is a very prepared. Our training reports deeply connect to and inform organizational KPIs. Maybe it's on our way. We know what reports we need and we're figuring out how to get there. Three We have some reports and four, we have goals. And by the way, just in case you're sitting there and you're thinking yourself, hmm, I've got a lot of systems, but I can't do a lot of reporting and so on and so on. It's not you, right? It's not you and it's not your team's We believe we have this thesis that there's a fundamental missing piece within the learning environment today, and that is comprehensive learning infrastructure.

That's the problem we're trying to solve in order to help you and your teams do better and get access to this data that you know you want to access. You know, you want to use to make decisions. What has just been locked away in spreadsheets or manual work for years and years?

As that poll question is closing out? See what the results say. Not so many very prepared. We're on our way. That's encouraging. We have some reports, almost half, and we have goals. So that is pretty pretty much reflects what we see in our normal experience. All right. Great.

So let's go back to Shane Battier. Right. The Houston Rockets crunched this data and they put him. They signed him, put him on the team. They played him for a number of years. And Shane Battier actually holds a number of distinctions that are really, really rare. He is the only player in NBA history to have been part of two 20 game winning streaks. So there's never been another NBA player that's been a part of two 20 plus game winning streaks. And Shane Battier was there he had because he did these things that weren't immediately exposed in stats. He had this ability that when he was on the court, teams played better and opponents played far worse.

And so the Houston Rockets used him. They put him in games. They often started him. They often played him out of position. And he was deeply consumed with this idea of how analytics could transform the game of basketball. He was the rocket's only player that had access to its highly sophisticated, by the way, confidential statistical data that they compiled on all opposing players. And he used this data to become familiar with the tendencies of the basketball players that he would guard each game. He was a defensive specialist even back when he was a college player at Duke. But armed with the right data, he became a defensive superstar and he would shut these other superstars down that had far flashier stats made a lot more money. We probably bought their jerseys, not Shane Battier, and yet he was just very effective out on the court.

And he also won two championships with the Miami Heat. So after he was released by the Rockets, he went to the Miami Heat, won two championships and probably most memorable in his career is he would often score like zero points, three points, because he would select his shots. He was not a great NBA shooter. So, again, a stat line was miserable. If you looked at him, he would even probably be in the top 150 folks in the NBA. But he really drove his teams and his teammates to to success. And what I liked the most about Battier, even though he was probably one of my most hated players when he was in college because he was always doing very well against my team, was that when he was in the pros, he managed to basically score 18 points in an intense Game seven against the Spurs to win his second championship.

And so he actually had his day in glory, even though he didn't often score that many points. And he got awarded an award for that performance. And he was asked to speak after that performance. He basically said it's better to be timely than good. But I think what he really meant is it's better to be armed with data and actually focusing on the right outcomes than to be focusing on stats and not have access to that data. And just to kind of complete the Shane Battier story or saga. After you retired from Miami Heat. He actually moved into a role with the Miami Front Office to be director of their data and analytics division because they wanted to take these concepts and move them into their day to day operations.

And that's what we would really like to see for all of you, is you get the platform, you get the systems, you get the infrastructure sorted out. You can then start becoming familiar with the different types of analytics that you're going to want to run your business. And then you progress your career and you become you move up in the organization or whatnot or go to another organization and implement these strategies there. And much like Shane Battier. So if this was interesting, to you at all, we've got this resource build a learning analytics blueprint and highly encourage you to check it out. It's really extensive. We've interviewed a lot of customers, and we'll send it out after the after the event today. And I think now it's basically time for questions.

And Doug, I won't hold it against you that you said a Duke player can make things better. I really won't. We can still be friends but yeah, if you have any questions on what we've talked about today, send them in. Either be a chat or via the Q&A module, and I'll do my best to answer them. This is my email address. This is our website administrator. That's my LinkedIn. We'd love to hear from you. I'd love to get feedback on how you thought the session went today. Maybe chat about how to catch serial killers or how to improve the game of basketball or, you know, any other topic that you think might relate to analytics. We love to see kind of different ways of thinking about maybe what to some would be boring business analytics. We get really excited about that, but we also get excited about illustrating it with with different ways

All right. Thanks so much, John, for a great presentation. And everybody, we do have some time here for questions with John. So if you have those pop up in your Q&A window and ask those. Now, here's one from Audrey that just just popped in. Let's see, are there any standard metrics to look at within each of the elements It's a great question, and I would say that the answer is probably no, but maybe go a little bit deeper on that. What I really what we really want for folks at administrate for customers and so forth and for our industry is to equip training and learning teams and departments with the ability to engage with the standard metrics of their business.

So what I mean by that, maybe if you're in manufacturing things like error rates and defect rates and so forth, maybe if you're in a software company, it's things like, you know, customer churn or whatnot, whatever it is that your organization manages main line. Right? Because most of us work for organizations that aren't solely devoted to training. We want training teams to be able to take those metrics and tie them all the way back to individual learners, individual courses, individual instructors. So if you can't do that, that's a clue. Probably that you might be fighting a pretty tough battle because maybe the ability to do that is locked away in spreadsheets and so forth. And so that's why we really want to be able to work with folks to get the business metrics that they really care about at their mainline organization related all the way down to learner activity and so forth.

So hopefully that makes sense. One thing I've seen a couple questions come in about, if you all will receive the guide And just a reminder, we will be sending out an email tomorrow with a link to the session recording along with that guide and the PDF of today's slides

Next question from Sandeep. Do you see any role of descriptive data in learning analytics? Yeah, I mean, let's let's be clear, right? Just because descriptive data is the first piece of this puzzle doesn't mean that it's not important. So we need things like attendance and how well people did and what they thought of their instructors and even what they thought of their meal choice and all that type of thing. Because that helps us deliver better training and engage with our learners. And that all of that stuff is good. We just believe that we need all of those different levels to really build out a powerful training function that, again, can kind of be on the forefront and help you answer that question. If your CEO is roaming the hallway trying to find somewhere to spend more money.

I know that would never happen, but let's just say it would. You can answer that question quickly and fluently and back it up and tell a story with data and follow up question on that from Nalini. How do we identify what specific data to focus on beyond the descriptive

Yeah, great question. I think that's just part of this iterative process, right? So we start with descriptive data about the learning environment, but then it's kind of like, well, where does the business want to go? Most of us are working in companies like Administrate. We're a software company, we're not a training company. And so we have business objectives we want to achieve that often revolve around selling more software or getting our customers more engaged. How then does our training team in our training function link that business go back to whatever it is that they're doing? And so sometimes that can take a little bit of iteration. Maybe we want to measure things like support tickets, right? That meant that our training team had to go and integrate our ticketing system with Administrate which is what we use, of course, to run our own training program and be able to then query that data and use it.

And so that will that will change. And that's an iterative process. And I think there's another question here from Katie which kind of jives right in with this answer, which is, you know, with each training, with each industry's training needs being vastly different. How long would you say developing a program takes? Right. And I think the answer to that is this is hopefully always going to be an iterative process. What we see is step one is kind of just get in there and get things organized that could take a few months. It could take a year depending on the complexity of organization. A lot of very large, very, very successful multinational enterprises that we've all heard of can't even say what a unified course catalog of theirs looks like. That's an organizational problem, right?

We've got to solve that first before we can really start to see how those courses and how those learning experience are then matching up against business metrics. So that might take a while. And then it's the figuring out what questions we need to answer and figuring out which systems we need to get. Integrated integration is step two. Then we can become analytical and then we can start moving through these various questions. And, you know, the thing is, systems will change. Questions will change. Business challenges and priorities will change. That's normal. But if you've got an infrastructure that's put in place, this is our contention. Then that change, you can actually be on the forefront of it and it can become something that the training team is driving, you know, towards as opposed to maybe trying to catch up because there's just so much training that needs to be done.

One of the things that haunts us at administrate is that the larger organizations get the less efficient training operations tend to become. And we want to reverse that trend. And we want it to be you don't need to add more bodies just to do more training. We want it to be we can do more training. We can scale this operation. We don't have to invest in more bodies necessarily. Instead, we can invest where it makes sense and really dove in and push the business forward.

Wonderful. Thanks, John. How about this question from Kenyon? Any suggestions on best practices for ensuring everyone is on the same page when entering data? So there's not a lot of cleanup of the data when you're ready to go get the metrics that's great question. We have a pretty simple philosophy that I think is actually very difficult to implement. So it sounds simple, but it is not easy and that is dry, right? Don't repeat yourself deeper. Why don't repeat yourself. And what we mean by that is try your hardest to make sure that data is not duplicated and not repeated in various different places within a business.

The worst thing that can happen is you're saving data in one system and then duplicating it maybe manually, maybe every week or two in another. That can cause a whole bunch of problems. Things will get out of sync and so forth. And so that's why with Administrate with high quality training infrastructure, we look to have integrations that can make sure that you're not duplicating that data and keeps it in sync. And then when you go to actually run reports, you don't spend and this is pretty common, right? You run a report how many people that we train, you know, last year and it comes out to be a number that you might think is three times higher than what actually happened. And the reason is because data has been duplicated somewhere that's time consuming and painful. And that cleanup can often be simplified by that. That philosophy of don't repeat yourself, however, that is very difficult.

That's easy to say. It's very difficult to put into practice Yeah, that is a challenge. All right. Here's a question from Mike who's saying, you know, we hear the ROI question a lot with instructional programs and they know it's more than the number of seats occupied or the number of hours training. Any suggestions for Mike on what to initiate looking into? Yeah, great question, Mike. And I'm not sure exactly what industry you're in. I couldn't see if you'd answer that question when somebody was asking where all of us were hailing from. But I think it would just highly depend on your business. One of my favorite examples of this is Boston Whaler, our customer. They build boats and what they need to do is they need to reduce manufacturing errors.

And so what they did is they deeply integrated their training data from Administrate into their manufacturing resource planning software that ran their assembly line. And so what would happen is when they would go to schedule an employee for a shift out on the assembly line, they would make sure that they scheduled the employees that had done the best or progressed the furthest on their training. When the employee then went out to the assembly line to build the boat, they would not be allowed to accept jobs for which they hadn't been certified for. And then when they actually started to do the job, maybe install a propeller they would have the training documents from within. surfaced right in front of them out on the assembly line floor through their manufacturing resource planning software, and they could refer to that, remember which way the propeller turned and all of that stuff.

Those are descriptive analytics from a couple of different systems, but all that meant that the training team can now look into the data and say, Wow, what happened? This propeller came off the assembly line and it was broken. Who was working there? Not in a blaming sort of way, but it's kind of, well, who was working there? Where did they go to the training? When were they last trained? Maybe their instructor kind of showed them wrong. There's a whole bunch of questions you can start asking using data to then say, You know what? We need to invest more in propeller installation training. And that meant that they can move the needle very dramatically on manufacturing errors and a whole bunch of different errors. Areas within that factory, while they doubled the size, both employees and an output over an eight month period. So that's really the ROI that they were looking for.

How do we double output? How do we grow this operation? And the training team was a really critical component of that. And it's just such a great story. It's one of my favorites Thanks, John. We're running out of time, but I do see a couple more good questions here in the chat. So just wanted to ask you this next one from Simon first. How do you approach more nebulous or softer organizational measures such as culture change when the KPIs are not as clearly defined? That's a great question. You know, not everybody is building boats, installing propellers. Right. And administrate. We're a software company We are a knowledge based industry. Right. And we have challenges like this. Right.

We want to implement some sort of culture change. Maybe it's a culture of increased quality, or maybe it's a culture of, you know, we've got offices all over the world in the Middle East, Ireland, here in Edinburgh, out in the U.S. How do we make sure that our employees are communicating efficiently and well together despite their different cultural backgrounds? What we found is that some of the things that are softer, like how well do you think you're communicating with your colleagues you need to do that through kind of traditional assessments, but you need to ask the question, what tends to happen is training teams get frustrated because they want to be able to ask these questions. They want to dove deeper into the experience or the behavior of employees or maybe even ask managers, Hey, we took your team. They went on this training for cultural sensitivity We sent out these questions.

Have you noticed a change? Right. But they don't have any spot to put these answers into that can then be related back to the training data and so forth. And so that's where it falls down or it gets put into a spreadsheet. The spreadsheet doesn't get updated or it gets lost or you have to redo it all again next quarter. And so just being able to have a spot where you can ask questions, get the answers back, and then start doing measurement is a real key to solving that problem. And that's what we do at Administrate. And it's it's pretty interesting. It's not perfect. You know, people answer differently based on how they're feeling that day or what they eat for breakfast. But it does get you a far, you know, far, far more forward than you would if you're not asking the questions and not storing that data. Thanks, John.

Let's do one more question, and this will be our last one. This one came from Shruti. To what extent have you noticed organizations become successful and measuring the impact of training? What would your top tips be for larger organizations who want to measure every impact? So my number one tip and I'm biased because I get very passionate about this, but my number one thought for training leaders, whether they're at a large, huge multinational intergalactic company, right. Or a smaller startup of three or four people is you need to have infrastructure that permits the training team to not focus the majority of their time on delivery. We see this happen all the time, and that is training teams are really really busy.

There's not a minute to lose. There's every single minute of their day is booked and they're working on administrative tasks. They're working on making sure that people get the right meals or that they show up to the right place or the log in to the right webinar. They consume the right e-learning content, or they get logged into one of five LMS's that might be in play. That is really important work, but that is not high value work. And so for us, my top tip is to get that infrastructural stuff taken care of, however you do it, and there might need one system, it usually needs like ten, and they all need to be integrated. But if you can get that taken off your plate, then you have all that time available to then think deeply about where is it that our organization wants to go? How can training help them get there, and how can the training team and L&D team be on the forefront of that change?

And when we see that, that's when it's really sweet. And you wouldn't believe the stories about training teams running really key initiatives that are company changing that are not normal for teams to be running or leading. And that's when you kind of know that you got it right. So that's that's that's what we really like to see

Great advice. And you're getting tons of kudos here in the chat. So, John, thanks again for being with us and and sharing this great content. All right. Thank you so much, everyone. And don't feel don't hesitate to reach out awesome. All right, everybody. Up next, we have Manoj Kulkarni joining us from Realize It. Just want to check in with our next speaker Manoj. Are you with us? I am. All right. Well, we will be getting started with your session in just about 10 minutes. So, everybody, you've got a quick break here. If you need to grab anything or take a quick walk. We will meet you back here at the top of the hour.