It’s easy to rely on metrics like “completed trainings” to chart the effectiveness of your compliance program. Compliance expert Courtney Sander reality checks the underlying stats you’re (probably) using to prove the value of your program.
Every compliance program should achieve these three goals: align the business with rules, regulations and laws; prevent misconduct; and detect misconduct. And as well-intentioned as we are (and even though those objectives seem straightforward), it’s easy to stray from these three things. Quite frankly, keeping a program on track and only focusing on these three things is hard work.
But good compliance program management doesn’t happen behind the scenes. It happens when we’re out from behind our desks and partnering with our functional leaders. And yet, we often fail to use our most powerful, underused tool in the proverbial compliance shed.
That’s right. Data.
Data helps you run a lean, strong program because it:
- Leaves out the technicalities and simplifies the complexities of compliance. It helps you communicate about compliance because data is a language everyone can understand. Leaders and employees alike can “get” the value-add compliance offers.
- Helps you make decisions about what you’re planning and doing in your compliance program. You can measure the scope of compliance issues and assess how your initiatives are mitigating those issues, and make the most of your budget.
- Empowers you to work smarter, not harder. Meaningful metrics shifts your focus away from efforts that are ineffective or redundant, and allows you to pour more resources into better opportunities.
Yet talking about data evokes confusion in a lot of compliance practitioners. We’ve all been taught from the start of our compliance careers that things like “number of calls to the hotline each quarter” and “training completion rates” are what we’re supposed to be reporting to our senior leadership. Hearing new recommendations about the kinds of compliance metrics we use launches us out of our comfort zones and into a whole new world — it’s hard to even know where to start!
But what if we used this new world of meaningful metrics as a growth opportunity? Growth for our programs and our ever-growing professional skill sets. Getting how to use data right helps you take action, make decisions, tell a complete and accurate story and demonstrate how your program is effective.
After all, effective compliance isn’t about big teams or big budgets — or even big data. It’s about being effective with what you have and delivering results.
A Gallup survey last year found that among employees who had received training on ethics and compliance, fewer than one in four rated the session as excellent. If you’ve seen the worst of the worst when it comes to compliance training, it’s not hard to see why.
Let’s talk about fake compliance data
In a compliance program, I refer to “fake compliance data” as anything that’s presented as a meaningful or helpful statistic but fails to address the three goals of compliance. And I get why this information is used: It’s easier to gather because we own it. I’ve used it in the past because I thought I was doing the right thing. The release of the DOJ’s evaluation of corporate compliance programs helped me see there was a better way to do things.
Training completion rates are fake compliance data
Disclaimer: If you have some sort of regulatory or legal requirement to meet an arbitrary training goal, I am in no way discouraging you from complying with that requirement. You obviously have to do this. However, there’s more you can — and should — do with those training outcomes.
I recently asked a group of 100-plus compliance practitioners how many of them have turned on a training course and then gone back to their day jobs. There were no more than five honest souls who initially confessed to this training transgression. (I gave everyone a second chance to answer and the results were much closer to accurate.)
If you’re currently reporting on training completion rates and have no true obligation to do so, you’re not alone. A study by Deloitte and Compliance Week shows that this is the most common way to measure effectiveness. And since this doesn’t actually do anything to measure effectiveness, consider reporting on how compliance training is influencing real business outcomes. (More on that in a bit.)
Sure, you can mandate all employees complete compliance training, but that’s all they’ll do — they’ll complete the training and see the hour they spent helping Example Bob navigate his workplace challenges as a check-the-box exercise that they’ll have to do next year.
Reporting on completion rates fails to address whether the training was:
- Appropriate for the person taking the training
- Applicable to the person’s roles and responsibilities
- Incorporated into what that person does and, thereby, preventing misconduct
- Measuring completion rates alone is legal accountability, not compliance effectiveness.
Policy acknowledgements are fake compliance data
Requiring people to sign off that they read, understand and know how to apply your policies is classic legal accountability that does nothing to prevent or detect misconduct. Reporting on policy acknowledgments doesn’t demonstrate compliance effectiveness. In fact, I don’t think this is anything the compliance team should report on.
Just as you update your browser without reading the terms and conditions, or “allow all cookies” without verifying what those cookies do, you go ahead with these activities because you consider them a must to have the best online experience. Signing a policy acknowledgment is often a condition (or at least an implied requirement) of employment, so, of course, people sign them and then go about doing their jobs in ways that align with the organizational culture.
Training test scores are fake compliance data
Training test scores alone do nothing to prevent or detect misconduct. People (myself included!) often pay enough attention to pass the quiz at the end of the course. And if they don’t pass? No worries! They study the right answers, retake the test and magically receive 100%. Did their knowledge about the test subject increase between tries? Likely not — just their knowledge about how to pass the test.
When you ask people what they will do if they encounter a situation, they’re more than likely going to tell you exactly what they think you want to hear — and in a multiple-choice scenario, you’re further limiting those responses. Hearing that 98% of people would do the right thing isn’t the same as real-life outcomes, especially when real business output doesn’t align with those training test scores.
Using training test scores to demonstrate compliance effectiveness is made worse when you give the same training year after year. Not only do employees know what Model Employee Bob should tell Sketchy Government Official Brenda because they remember last year’s training, but you’re also alienating those who don’t work with government officials — sketchy or otherwise — because they remember how Bob and Brenda didn’t help them do their jobs better over the course of the past year. Any employee who’s been around for longer than a year is single-handedly sabotaging your data.
Training course evaluations are fake compliance data
Course evaluations can be as skewed as online product reviews — and there are three types of people who submit product reviews: those who love the product, those who hate the product and those who ordered the product knowing there was a months-long backorder and gave the product a one-star review because they haven’t received their order after two weeks. These aren’t exactly the folks you want to stake your job and program effectiveness on, are they?
What’s worse, training course evaluations neither demonstrate the organization’s alignment with rules and regulations, nor prevent or detect misconduct.
Training course evaluations do offer helpful data points — just not for compliance effectiveness.
Understanding our employees’ experiences with the course itself can help you iterate and improve your training. Between supporting employees who work from home and making your training more inclusive, technical things like page load time, use of sounds and spoken words, text size and other comments about how learners interact with the course should be taken into consideration.
Number of communications sent or trainings provided are fake compliance data
Counting the number of emails compliance sends on behalf of the senior leadership team is a close cousin to reporting on completion rates. You’ve likely deleted at least a dozen unread emails this morning from senders within and outside your organization. Quite frankly, I don’t blame you. And yet, someone somewhere in the comms team is counting that send towards their annual goals.
You can use these types of metrics if you use them alongside other data points and don’t count this as your primary source for measuring your tone at the top.
Feel-good survey responses are fake compliance data
Surveys are often seen as the true measure of compliance culture health. The data you get back from surveys is unreliable for many reasons, including:
- People don’t want to report their coworkers.
- Disengaged employees don’t respond.
- People who are intentionally non-compliant aren’t going to respond or they’ll say everything is fine as-is (because they don’t want you to be effective).
- People don’t trust that anonymous surveys are anonymous so they’ll never put anything negative about their colleagues or the organization as a whole.
You do get the occasional odd-ball response from people on a crusade to make the organization a better place, one departmental survey at a time. (After all, everyone who completes the survey is entered into the drawing — and who doesn’t like a free $5 Starbucks gift card!?)
What’s more, they disrespect our employees’ time and responsibilities. People aren’t paid to complete surveys (with some exceptions) in their jobs, so surveys are rightfully a low priority activity. Answering surveys in a thoughtful and accurate manner takes time — and often, people don’t see how the questions are directly tied to their jobs.
However, if you are invited to include questions on HR’s annual employee engagement survey, ask questions that help you triangulate and verify some other metric you’re already monitoring.
- Want to know if those emails your leadership is sending out about ethics and compliance are making a difference? Ask a pointed question like, “How would you rate how [CEO/BU president/your manager] demonstrates our values of [honesty, respect, and transparency]?” Analyze this alongside your senior leadership email send and open rates.
- Need to know if visible compliance activities are positively impacting the business? Try, “Do you feel that compliance [messaging/training/activities] help you be more effective in your role?” Assess this alongside recent compliance initiatives and real business outcomes (more on this soon).
- Want to know if employees actually know where to report compliance and ethics concerns? Ask, “Where can you go to report compliance and ethics concerns?” and include options like the compliance team, their manager, senior leadership, helpline, and a peer. Are you seeing reasonable, comparable reporting rates to bona fide channels?
In fact, I once offered multiple choice questions that asked employees to select the helpline number and where to find our policies. Why? Because if they knew the answer, they could choose it right away. Either calling those numbers (I included other internal phone numbers for things like benefits and weather emergencies) or visiting the Compliance intranet site could have helped them answer those questions easily! The results of those questions were eye-openers to the true awareness our employees had about the ethics and compliance program.
If you do use survey data, ensure it’s not the sole source indicator of your organization’s cultural health. Monitor trends in that data and triangulate responses with other metrics that help you understand the true picture of whatever you’re trying to mitigate.
Let’s shift the compliance discussion for our leaders and employees
Gathering the right information — before and after implementing your compliance initiatives — can help you shift the narrative and demonstrate that you’re doing the right things, addressing real business risks, and prove your organization made a worthwhile investment in compliance. (And nothing beats a good ROI presentation for the folks who are approving your budget, right?)
Spoiler alert: I’m not giving you specific metrics to monitor in this column. Why? This isn’t a one-size-fits-all exercise. You must define a clear objective that’s right for your program, identify the data points found within your processes and organization that help you answer that question and ensure your program has the budget and bandwidth to collect, analyze and remediate situations the data reveals.
Identifying the right metrics is so important, I encourage my clients to seek out data in every initiative we partner on. I want to ensure we’re focused on solving the right problems (and that I’m supporting them on problems that my services can solve) and make sure they’re prepared to demonstrate their effectiveness. Why? Good data is good business.
So as you embark on your own data-finding scavenger hunt, use these clues to identify what you’re seeking.
Training outcomes are good compliance data
I’ve nagged quite a bit on training data so far. But not all training data is bad data. Drill into initial test results and get good, actionable information!
Dig into training outcomes and ask things that help you get to the root cause:
- Which questions did people answer incorrectly?
- Where are those people located?
- What roles do those people hold?
- Which business outcomes can I cross-check to understand the true issues?
In a sea full of information, this data acts as surrogate hand-raising and helps people get the attention they need. Once you know who needs additional guidance, provide an appropriate solution (like additional training and resources) to help those employees do their jobs the right way.
When you take a deeper look into how compliance training and other parts of your compliance program are impacting the business — whether directly or indirectly — you help your employees get away from feeling like any compliance activity is an arbitrary, check-the-box exercise.
Cheering for your favorite team (or against the one you hate), makes it easy to forget that NFL players are employees. If they violate team or league rules and regulations, they can be fined or even suspended.
Real business outcomes are good compliance data
Understanding true compliance effectiveness — whether preventing or detecting misconduct — happens when you analyze the outcomes of what people do every day at your organization.
What’s more, the metrics you’ll seek from business outcomes are likely already things the business tracks. Functional leaders already monitor how well their teams are performing and ensure they’re doing their jobs as expected, so when measuring compliance effectiveness is done the right way, the conversation you have with functional leaders is more about “help me help you.”
Figuring out which metrics are available to you can be a challenge. My favorite way to understand how something works — and figure out how to measure compliance success — is to talk to folks who are experts in those processes and observe what they do. I tell them that I want to simply see what they’re doing — not to catch them in the act and get them in trouble!
Depending on what problem you’re trying to address, good compliance data looks like these examples:
- % red flag invoices paid
- % non-compliant expenses reported for reimbursement
- % gifts and entertainment overspend
- # unreported lobbying contacts
- Policy and procedures click-through rates
- % accurate approvals
It’s important to note that, with few exceptions, the compliance team doesn’t own “compliance data” — your functional leaders do!
As a general rule of thumb, I like to use:
- Numbers or specific quantities when any instance would be a serious issue. “3 [competitive information gathering projects] did not follow our legal approval process.”
- Percentages because they quickly tell the full story. “98% of the 2,000 [invoices] we sampled were properly approved.”
You’ve got friends on the audit team that can help you parse the riskiest transactions, sample real business outcomes and understand how abstract risks are manifesting into real organizational issues. And if they’re not your friends yet, make a note to set up some time to learn what they do and how it connects with what you do. You may even learn that they audit certain areas (like third-party payments) on a scheduled cadence, giving you instant data trends!
Then, use your initial data set to set achievable goals and practical action plans to help you reach them. People care about what you’re improving and how you’re doing it—and using before and after metrics helps demonstrate that your program is taking action and moving the needle towards better compliance and business outcomes.
Complete data is good compliance data
You need to tell the full and accurate story with data. No picking and choosing here to spin the story to make things better (or worse) than they actually are! Using and presenting accurate data builds credibility for you and your compliance program.
Instead of gathering data to justify why you need to do anti-bribery training, approach your fact-finding exercise by asking a question like, “How are our anti-corruption policies being followed in high-risk transactions?”
Using data to answer specific questions about how the business is operating (you know, what its culture is) instead of seeking answers that rationalize your compliance program is a game changer. This often means using multiple data points.
For example, if you’re trying to address bribery and corruption in high-risk transactions, you might sample data related to high-risk transactions, such as gifts to those customers, travel paid for those customers to visit your sites, the items invoiced by your agent to get that business, and the expense reports for the sales team who managed that agent and customer. Instead of having to address all of the ways that bribery and corruption manifests in your high-risk sales transactions, you can focus your scope to tasks where non-compliance is high.
Here are five takeaways:
- Data isn’t an inconvenient add-on; it’s a must-have for effective compliance programs. Data-driven compliance enables practitioners to break through silos and run leaner, more effective programs that truly impact the business.
- You might not use the right data the first time. That’s OK! This process takes experimentation and innovation, both of which evolve your program.
- Data doesn’t need to be flashy. In a world of automation and machine learning, give yourself the opportunity to learn about what you truly need to collect along your compliance adventure.
- Compliance teams don’t own many compliance metrics. “Compliance” is something the business does or doesn’t do. You must demonstrate how your compliance program isn’t an organizational feature — it’s a benefit to functions to incorporate compliance into what they’re already doing.
Using data the right way is monitoring. Metrics fuel monitoring and help you decide whether compliance initiatives should ramp up in certain areas of the business, stop doing what’s ineffective or redundant and use your program’s resources to focus on truly preventing and detecting misconduct.