RSS

Tag Archives: diversity training roi

Three Critical Dimensions and Seven Levels that Turn Diversity & Inclusion Evaluation into Results

“Three Critical Dimensions and Seven Levels that Turn Diversity & Inclusion Evaluation into Results”

Evaluation is a task that every Diversity Practitioner will face at one time or another. No matter what your role such as Trainer, Consultant, Chief Diversity Officer (CDO), Council Member, ERG/BRG Leader, etc., conducting an evaluation to assess key aspects of your Diversity and Inclusion initiatives is inevitable.

Two Definitions of Evaluation

 

People do not always agree on one definition of evaluation. Following are statements that reflect two different definitions:

  • “Evaluation is the systematic process of collecting and analyzing data in order to determine whether and to what degree objectives have been or are being achieved.”
  • “Evaluation is the systematic process of collecting and analyzing data in order to make a decision.”

Notice that the first ten words in each of the definitions are the same. However, the reasons-the “Why!”-for collecting and analyzing the data reflect a notable difference in the philosophies behind each definition. The first reflects a philosophy that as an evaluator, you are interested in knowing only if something worked, if it was effective in doing what it was supposed to do. The second statement reflects the philosophy that evaluation makes claims on the value of something in relation to the overall operation of a Diversity program, project, or event. Many experts agree that an evaluation should not only assess program results but also identify ways to improve the program being evaluated. A Diversity program or initiative may be effective but of limited value to the client or sponsor. You can imagine, however, using an evaluation to make a decision (the second definition) even if a program has reached its objectives (the first definition). For example, for Non-Profits, Federal grants are based on the first statement, that is, whether the program has achieved its objectives, but the harder decision to downsize or change may be a consequence of the second definition of evaluation.

For some, endorsing Diversity Evaluation is a lot like endorsing regular visits to the dentist. People are quick to endorse both activities, but when it comes to doing either one, many Diversity Practitioners are very uncomfortable. In this blog, I want to reduce your discomfort by demystifying some important aspects of designing and conducting a Diversity ROI evaluation by helping you get to know a few Diversity metrics processes that matter in evaluation design.

In both for-profit and nonprofit organizations, organizations possess data (and information) that could help to evaluate a Diversity program or project. These data are the one thing that all evaluations have in common regardless of the particular definition of evaluation you embrace: “evaluation is the systematic process of collecting data that help identify the strengths and weaknesses of a program or project. The data may be as simple as records of attendance at training sessions” or, as complex as “showing test scores showing the impact of a new educational program on increasing students’ knowledge across an entire school system”.

Evaluating Efficiency, Effectiveness, and Impact

We can define Diversity evaluation even more closely as a process. The process is guided by the reason for doing the evaluation in the first place. An evaluation might be a process of examining a Diversity training program, in light of values or standards, for the purpose of making certain decisions about the efficiency, effectiveness, or impact of the program. To carry out this task, you need to understand the concepts of efficiency, effectiveness, and impact. Think of these three terms as the levels of a program evaluation.

“Efficiency” relates to an analysis of the costs (dollars, people, time, facilities, materials, and so forth) that are expended as part of a program in comparison to either their benefits or effectiveness. How is efficiency, or the competence with which a program is carried out, measured in a program? The term itself gives clues to what this is about. Diversity Practitioners would look at the efficiency with which details are carried out in a program. Diversity programs and initiatives often begin with recruiting, gathering materials, providing for space, setting up fiscal procedures, and so forth. In other words, the relationship between the costs and end products becomes the focus of an efficiency evaluation. Although very important, these aspects of efficiency have no bearing on the program’s effectiveness. If the investment in the program or project exceeds the returns, there may be little or no efficiency.

For example, let’s consider an assembly line facility that houses a rather substantial training and staff development department. As part of this department, ten instructors are responsible for ensuring that five hundred employees are cycled through some type of Diversity training every six months, for a minimum of twenty hours of training each cycle. The training revisits the employees’ basic knowledge of their job and introduces new concepts of Diversity that build additional competencies since the last training. The staff development department might work very efficiently by making sure that all employees cycle through in a timely fashion, in small enough groups to utilize the best of what we know about how adults learn. The students’ time on task is often not enough, however, and many of them do not retain much of what was covered in the training. Thus the program is not effective.

The department may be efficient in that it fully utilized the time of each of the available trainers, it stayed within the parameters of the staff development budget, it kept employee “down time” to a minimum, it used materials and equipment that were available, and it completed the Diversity and Inclusion training agenda for the company. Yet there may be an increase in cultural miscommunication incidents and generational conflict across levels because employees make simple, basic mistakes and assumptions about others who are different than themselves. The department’s training was efficient but not necessarily effective.

When you examine the “effectiveness” of your Diversity and Inclusion initiatives, you are asking this question: “Did the activities do what they were supposed to do?” Simply put, an initiative’s effectiveness is measured in terms of substantive changes in knowledge, attitudes, or skills on the part of a program’s or initiative’s participants. Although the right number of participants may have been recruited and the best possible site may have been secured, the effectiveness test is this: Did the activities provide the skills to effectively handle Diversity and Inclusion-related situations! Did the participants gain the knowledge they need to work across generational differences? In another example, the same Diversity and Inclusion department staff may conduct a training session on a new approach to deescalate cultural conflict situations. The trainer may pretest all the employees as they begin their training session. Upon completion, the employees are post-tested and the results compared to determine whether their knowledge increased, decreased, or stayed the same. An increase in their knowledge would be an indication that the training was effective-it did what it was supposed to do. Yet two weeks after the training, when one of the employees was back at her job location, a situation arose in which she engaged in a serious altercation with another employee and failed to use the skills taught. She used her older, more comfortable procedure for addressing communication differences across cultures and caused a problem that put her and her coworkers at the risk of suspension. Here is an example of training that was effective-the worker passed all the posttests-but had little impact on changing the behavior of the employee.

Thus the impact that the program has had on the people or organization for which it was planned becomes an important evaluation consideration. “Impact” evaluation examines whether and to what extent there are long-term and sustained changes in a target population. Has the program or initiative brought about the desired changes: Are employees using the new procedures? Do your employees have more job satisfaction?

Evaluators frequently pay too little attention to assessing impact. One reason is that “impacts” often manifest themselves over time, and Diversity Practitioners have already turned their attention elsewhere before computing this aspect of the evaluation. The actual impact that training in new procedures might have in people’s everyday life often needs time to percolate and evolve. An attempt to collect impact data after allowing for this delay may run into a number of roadblocks such as learner turnover (you cannot find them), job or circumstance change (they no longer are in situations that demand heavy use of the skill taught), or lack of time or resources for the evaluator to conduct these follow-up activities.

Still, program and project sponsors are most interested in impacts. Whether a learner feels satisfied with the training or the training results in knowledge gain means little to a sponsor or employer if the learning doesn’t help the organization.

Evaluating Alternatives

The second philosophical statement that defines evaluation presents it as the process of delineating, obtaining, and providing useful information for the purpose of selecting among alternatives. Thus, it may not matter whether the program was efficiently conducted, effective, or had an impact on behavior or functions, Instead, the value of the evaluation is in its being able to compare one activity to another, one initiative or program to another, or one employee to another so that decisions can be made in the presence of empirically collected data. Diversity Search Committees perform this kind of evaluation. In the course of their work, they describe job candidates’ strengths, outline previous experiences, and acquire other useful information that makes it possible to choose among a number of candidates. A company planning to adopt and purchase a computer system will perform this kind of evaluation on all the systems it is considering. It will select the one that performs the best given the needs and resources of the company.

 

Identifying Areas to Improve

Finally, there is a third way of defining evaluation: Evaluation is the identification of discrepancies between where a program or initiative is currently and where it would like to be. For example, an organization’s multicultural marketing department may have as one of its goals at least one face-to-face focus group with emerging market customers per year. Currently, its Sales force sees fewer than half of the multicultural customers in a year. Records of face-to-face calls indicate the discrepancy between where XYZ Corporation is currently as opposed to where the organization wants to be.

Personnel evaluations often take on this definition as well. A new employee’s first evaluation may be an example of the first definition, that is, an evaluation against some minimal standard of performance. After this initial evaluation, certain performance goals are set for the employee (either mutually or by the supervisor or team. The next and all subsequent evaluations of the employee are compared with those performance goals or standards. The discrepancies are identified and remediation strategies are developed.

The Critical 7 Levels — Don’t Perform Your Diversity Initiative or Diversity Program Evaluation without Them!

Other levels of evaluation as defined by the Hubbard Diversity ROI Methodology refer to the eventual us of the evaluation data and who might make use of the results

Diversity Return on Expectations (DROEx®), for example, must be based on evidence and impact results. I have found it useful to first to distinguish the “evidence-based, outcome-focused” measures from other types of “activity only” measures. Anyone responsible for implementing Diversity initiatives is also responsible for evaluation. Whether you calculate the impact or not, from management’s and/or the stakeholder’s point of view, “you will always own the ROI of the initiatives you implement”. So, the amount of evaluation that you provide to meet expectations depends on the types of decisions that your organization must make and the information needed to make those decisions. There are 7 levels you can use in the Hubbard Diversity Return-on-Investment DROI® evaluation methodology to effectively demonstrate your ROI impact and show a “chain of impact” to meet customer and stakeholder expectations that is evidence-based and credible:

  • Level 0: Business and Performer Needs Analysis
  • Level 1: Reaction, Satisfaction, and Planned Actions
  • Level 2: Learning
  • Level 3: Application and Behavioral Transfer
  • Level 4: Business Impact
  • Level 5: Diversity Return-on-Investment (DROI®), Benefit-to-Cost Ratio (BCR)
  • Level 6: Intangibles

For example, if your only requirement is to ensure that participants have positive attitudes toward the initiative, then a Level 1 evaluation is sufficient. But, if your goal is to determine whether your diversity initiative is having a positive effect on job performance, then you will have to perform a Level 3 evaluation. This means you will also have to conduct Level 1 and 2 evaluations in order to assess the learning performance applications and Job impact at Level 3 (an example of the “DROI® Chain of Impact”). They provide the basis for determining whether participants demonstrated that they learned by putting these skills and attitudes to use (verified by a Level 3 evaluation).

There’s no doubt that we must communicate effectively and demonstrate our value to the bottom-line. Diversity ROI metrics and performance improvement processes help us focus first on tangible outcomes, then on interventions to meet expectations. When Diversity Practitioners focus primarily on the intervention, such as the Diversity content, the method, or the technology, it’s easy to be led astray by current fads, thus wasting valuable time and money. Instead, focus first on the desired outcomes and DROI® analytics to determine what kind of measurable diversity intervention, if any, is necessary to meet customer and key stakeholder expectations.

Conducting glitzy Diversity training or other Diversity activities and implementing fad-based interventions can distract decision makers from what truly counts. The glitz may make things fun, louder and interactive, not necessarily better. Without a clear, data-based front-end analysis of organizational performance gaps, any intervention, including Diversity training and the like, is a guess. Add in sophisticated Diversity intervention technologies without an adequate front-end analysis, including metrics, and it becomes an expensive and often complex guess. Systematic Diversity training design procedures, for example, must include DROI® analytics and metrics, needs assessments, objectives, targeted competency-based design, and multi-level evaluation processes. That framework provides a method to get the coveted seat at the C-Suite table. Why? Because when used properly, that knowledge base can help companies increase revenue and decrease costs using Diversity and Inclusion practices that impact organizational performance outcomes. In other words, you can earn your seat at the executive table by applying what you already know as a Diversity ROI-focused professional. It’s successful because the DROI® metrics and processes you use are solidly based on the behavioral science research results which provide strategies detailing how diverse people interact and what drives their behavior to produce successful organizational outcomes.

If we examine any other professional discipline or field of study, like Medicine, Engineering, Accountancy, Science, etc., we expect that they are able to prove the efficiency, effectiveness, and impact of the solutions, programs and initiatives they what us to support. Why should Diversity and Inclusion be any different if we want our processes to have credibility and support?

Conducting a comprehensive Diversity Evaluation is the only true way to know if your Diversity and inclusion programs or initiatives are delivering the outcome results expected by key stakeholders to meet the needs of the organization. It is essential that Diversity Practitioners master critical Diversity and Inclusion evaluation methods using technologies that are rooted in Diversity ROI science. Why? Because the perceived value and credibility of what we do to be seen as a true Business Partner and Professional depends on it! Are you evaluating your Diversity and Inclusion initiatives and programs at this level?? What’s your department’s Brand and Credibility Image in your organization??

 

Tags: , , , , , , ,

Using Diversity ROI Measurement Skills to Get a Seat at the Table

Introduction

As the Diversity and Inclusion industry climbs the ladder of recognition, relevance, and importance, more and more Diversity and Inclusion professionals want to learn how they can gain C-Suite access and earn a seat at the proverbial table of power. After all, it’s no secret that if you sit with the executive team, you become instrumental in strategic decision making-one of the best ways to influence change. In order to get the coveted seat, you must speak the language of business and focus on the critical business objectives and metrics, such as increasing revenue, decreasing costs, and reducing cycle time. There’s no doubt that we must communicate effectively and demonstrate our value to the bottom line. Diversity ROI metrics and performance improvement processes help us focus first on tangible outcomes, then on interventions. When people focus primarily on the intervention, such as the diversity content, the method, or the technology, it’s easy to be led astray by current fads, thus wasting valuable time and money. Instead, focus first on the desired outcomes and DROI® analytics to determine what kind of measurable diversity intervention, if any, is necessary.

Conducting glitzy Diversity training or other Diversity activities and implementing fad-based interventions can distract decision makers from what truly counts. The glitz may make things fun, louder and interactive, not necessarily better. Without a clear, data-based front-end analysis of organizational performance gaps, any intervention, including diversity training and the like, is a guess. Add in sophisticated Diversity intervention technologies without an adequate front-end analysis, including metrics, and it becomes an expensive and often complex guess. Systematic Diversity training design procedures, for example, includes DROI® analytics and metrics, needs assessments, objectives, targeted competency-based design, and multi-level evaluation processes. That framework provides a method to get the coveted seat at the table. Why? Because when used properly, that knowledge base can help companies increase revenue and decrease costs using Diversity and Inclusion practices that impact organizational performance outcomes. In other words, you can earn your seat at the executive table by applying what you already know as a Diversity ROI-focused professional. It’s successful because the DROI® metrics and processes you use are solidly based on the behavioral science research results about how diverse people interact and what drives their behavior to produce successful organizational outcomes.

Using Diversity ROI Measurement Sciences and Analytics

Measuring the ROI of Diversity initiatives requires Diversity professionals and practitioners understand and utilize the full range of measurement sciences and approaches Diversity and Inclusion has to offer. If our work is to be seen as credible and providing value to the organization, we must hold ourselves to a high standard whether or not the C-suite and others are asking for it. It is disheartening to know that although Donald Kirkpatrick’s “Four Levels” of training measurement has been around since the 1950’s, and a primary intervention in the Diversity and Inclusion field is diversity training, only a select few Diversity Professionals and Practitioners consistently use a comprehensive, Instructional Systems Design (ISD) research-based approach to diversity training evaluation. My books, “Measuring Diversity Results (1997)”, “How to Calculate Diversity Return on Investment (1999)”, “The Diversity Scorecard (2004)”, “Implementing Diversity Measurement and Management (2004)”, “The Diversity Discipline (2008)”, and “Diversity Training ROI (2010)” and others outline numerous approaches to measure the ROI impact of Diversity interventions. They are designed to give Diversity Practitioners business acumen and “how-to” measurement tools, templates, strategies and techniques to demonstrate C-Suite level analysis capability.

It is regrettable, yet understandable, that Diversity interventions and initiatives are attacked as “having little or no value for the money invested.” Many of the diversity measurement processes used by some practitioners can barely pass a true Level 1: Reaction, Satisfaction, and Planned Actions evaluation instrument test. A majority of the instrumentation found qualifies only as a basic “smile” sheet at best. This doesn’t mean that credible, research-based Diversity training and initiative evaluation does not take place in our industry, only that many Practitioners have not consistently utilized solid, DROI®-based measurement sciences to document their impact on the organization as evidenced from the feedback in a wide-array of surveys conducted over the years.

The People-Profit Link

The link between people and profits is clear. Utilizing human capital assets is the primary vehicle to drive organizational performance and results. Thus, in a global economy and diverse world, developing a diversity competent employee base using a learning culture is vital for long-term, national and global success. Performance and profits are closely linked and require a comprehensive diversity analytics and measurement strategy to utilize human capital assets effectively. Skill development is one of several activities that must be undertaken to achieve an organization’s strategic business objectives. The only way to determine that skill development and diversity training are having the desired effect is to use formal training evaluation and measurement processes as well as ROI-based cost-benefit analysis methods. The results of these activities can confirm the positive effects of diversity training and development and identify improvements to make it better. Diversity training measurement and evaluation strategies can contribute to maximizing an organization’s overall return on investment (ROI or return on mission (ROM™), a crucial skill for anyone sitting in the C-Suite.

How You can Earn Credibility with the C-Suite

So, to gain a seat at the table, you need to do more than speak the language of business. You should play to your unique strength-an in-depth understanding of Diversity ROI and how it can successfully drive organizational performance. With continued efforts to learn, refine, and apply what’s already known about enhancing Diversity and Inclusion practices with ROI metrics, you can bring a unique and powerful perspective to the top management team that can make a measurable difference in the organization’s performance!

Dr. Ed Hubbard is the President & CEO of Hubbard & Hubbard, Inc., and recognized as the Founder of the Diversity Measurement and Diversity ROI Analytics fields. For more information about the Hubbard Diversity ROI Institute, log onto http://www.hubbardnhubbardinc.com/certification-workshps.html

 

Tags: , , , ,

Diversity Training ROI

Introduction

There seems to be a myth operating within some communities of practice, which suggest that the results created by diversity initiatives defy measurement. It is often presented with a belief that creating a measurable diversity process that drives business is something of a complex and mysterious art form. Organizations are looking for strategies to deal with increased competition, options for reducing cost, and increasing productivity to affect the bottom-line. Many diversity training evaluations use poorly designed “smile sheets” and focus primarily on attendance records that have very little to do with the business needs and specific performance requirements. I believe this must change.

Accountability is a key issue for diversity training just like any other business unit. Diversity training is only one of several initiatives that are undertaken to achieve an organization’s diversity objectives. Left without the effective practice of utilizing consistent, time-tested methods of Diversity training ROI evaluation practices, diversity organizations can become an easy elimination target for those who do not believe in its value. Consequently, the idea of being able to calculate the diversity return on investment (DROI®) of diversity training is mandatory if diversity is to take its place at the strategic partnership table.

Setting A Standard of Excellence for the Diversity Field

The lack of measurement practices for diversity sets diversity apart from the rest of the organization. Therefore if we, as diversity professionals, want to be effective communicators of value added to the bottom-line, we must build rapport with our audience using DROI® methods. The business case and rationale for diversity must be linked to strategic business objectives and initiative results must be displayed and communicated in organizational impact-related terms.

I am convinced that the only way to determine that diversity training and skills development are having the desired effect is to use formal training evaluation processes and cost-benefit analysis methods. The results of these activities can confirm the positive effects of training and development and identify improvements to make it better.

Benefits of Diversity ROI Evaluation

Evaluation can contribute to maximizing the organization’s return on training investment. Evaluation helps us to use measurement to make decisions (example: to stop, modify, or expand a process’ benefits). This implies:

  • Knowing what decision the evaluation data will help you make.
  • Measuring scientifically, using data collection methods and research designs that separate the effects your program is having from all other influences on your outcome variables.
  • Choosing the right measures for what your program is really trying to accomplish.

A Seven Level Causal Chain-of-Impact

I have found it useful to first to distinguish the “evidence-based, outcome-focused” measures from other types of “activity only” measures. Diversity ROI must be based on evidence and impact results. Anyone responsible for diversity training is also responsible for evaluation. The amount of evaluation that you provide depends on the types of decisions that your organization must make and the information needed to make those decisions. There are 7 levels you can use in the Hubbard Diversity Return-on-Investment DROI® evaluation methodology to demonstrate impact:

  • Level 0: Diversity Needs Analysis
  • Level 1: Reaction, Satisfaction, and Planned Actions
  • Level 2: Learning
  • Level 3: Application and Behavioral Transfer
  • Level 4: Business Impact
  • Level 5: Diversity Return-on-Investment (DROI®), Benefit-to-Cost Ratio (BCR)
  • Level 6: Intangibles

For example, if your only requirement is to ensure that participants have positive attitudes toward the course, then Level 1 evaluation is sufficient. But, if your goal is to determine whether your diversity course is having a positive effect on job performance, then you will have to perform a Level 3 evaluation. This also means you will also have to conduct a Level 1 and 2 evaluation. They provide the basis for determining whether participants want to use what they have learned and have indeed learned the appropriate attitudes and skills.

How to Get Started?

Where Do You Begin? Your first step in evaluating diversity training is to determine your major evaluation questions or objectives based upon a needs assessment or cultural audit conducted at the outset. The second step is to plan and conduct the appropriate level of diversity training evaluation. The following decisions are examples:

  • How satisfied are the participants with the course? What are their Planned Actions (intention to use)?
  • Do the participants believe they learned the values and skills that the course was intended to teach?
  • Can the participants demonstrate the values, knowledge, and skills that are taught in the course?
  • Do the participants report that they are using their diversity values, knowledge, and skills on the job?

What Standard Will You Set?

If full utilization of a diverse workforce and diversity is to be a reality in our lifetime, we must use every tool or resource available to fully monitor and communicate the effectiveness of this effort. When we start to show management exactly how much value diversity training programs can contribute to the process of building an inclusive work environment, then diversity-training initiatives will become a strategic requirement. A true commitment to diversity takes effort, time and money, none of which can afford to be wasted in a competitive marketplace. Our internal standards as professionals must compel us to measure our results without having to be asked.

So, how does your diversity training efforts measure up? What are you doing to show that the diversity training you conduct adds “evidence-based” value to the organization and it’s bottom-line in real measurable terms? I know that some practitioners are showing their impact. Let us hear about your success.

Dr. Ed Hubbard is the President & CEO of Hubbard & Hubbard, Inc., and recognized as the Founder of the Diversity Measurement and Diversity ROI Analytics fields. For more information about the Hubbard Diversity ROI Institute, log onto http://www.hubbardnhubbardinc.com/certification-workshps.html

 

Tags: , , , ,