Background

Organization

NCOPDS

The Senior Leader Course (SLC) is part of the Noncommissioned Officer Professional Development System (NCOPDS) for the U.S. Army. SLC is a branch-specific professional military education (PME) course which provides NCOs training in the leadership, technical and tactical skills and knowledge needed to lead a platoon-size element. Attendees are NCOs who have been selected for promotion to SFC. These NCOs must complete SLC successfully before they can pin on the rank of SFC.

Challenge

The Mission SLC is a 35-day, full-time, resident course taught at the Mission Center (MC) NCO Academy (NCOA). Mission SLC trains approximately 700 soldiers a year, in five cohorts. Students learn the Common Core Competencies expected of all NCOs selected for promotion to SFC as well as special team-specific tasks.

When the Department of Defense stopped travel in response to the COVID-19 pandemic, it stopped transportation for all PME courses, resulting in course reservation cancellations at the MC NCOA. In an effort to try and address the backlog of students needing these PME courses for career progression, the MC NCOA moved SLC, which had the highest number of cancelled reservations, to online distance learning.

The products developed for SLC distance learning were asynchronous and intended to be delivered via BlackBoard.mil, which was the only LMS available to MC NCOA. A team of 24 SLC instructors and 6 MC NCOA Instructional Designers spent March-May 2020 adapting the resident SLC instruction for asynchronous delivery. After MC NCOA Leadership review, the delivery model was temporarily moved to MS Teams, in hopes of presenting a more interactive and synchronous course. A pilot cohort of two classes composed of local students, started in June, 2020. During the pilot class, data was gathered and the NCOA training management team continued to refine products and develop procedures to prepare for an August, 2020 cohort of 12 classes composed of students across the operational force.

The pilot cohort of dL SLC was successful enough that this method of delivery will be used through FY21 to train NCOs requiring MC SLC.

The MC NCOA Training Management team has continued to gather data to support this initiative. Data collected shows that the SLC instructors struggled, initially, with online instruction. A bank of resources was created for these instructors, which addressed effective online delivery and how to facilitate instruction using a remote platform.

Feedback from the August cohort showed improvement from the Level 1, reaction standpoint; however, the MC Command Team agreed that a formative evaluation of the SLC dL course would help identify areas of improvement in course delivery for both students in the operational force and assigned instructors. The evaluation team included three master's degree students enrolled in the Evaluation course offered by Organizational Performance and Workplace Learning (OPWL) department at Boise State University. The project was conducted in the Fall of 2020 (Deinhardt., Baker, & Johnson, 2020).

Program and Stakeholders

The MC SLC is a combination of synchronous and asynchronous online learning. The course is 35 days long. In dL format, the students spend about 4-5 hours a day in synchronous online meetings with instructors and fellow students three days a week. The rest is self-paced but instructors are available at all times for student questions and the students frequently meet on the off days for group work.

In addition to the main goal of training NCOs to transition to positions as senior leaders at the platoon-level, another intended outcome is to reduce the risk of exposure to, or the spread of, COVID-19 to soldiers, civilians or family members.

Through communication with the client, the evaluation team helped develop a training impact model (TIM) for the SLC dL Course. The TIM outlines the means and end results of the program:

• Resources: Facilities, tools, materials, data, personnel, etc. used to run the program

• Activities: Processes executed to run the program

• Program Capabilities: Knowledge, skills, and attitudes that participants will acquire

• Critical Actions: Job-specific behaviors that participants will exhibit

• Key Results: Job-specific outcomes that participants will leave behind as a result of their job behaviors

• Business Goals: Organizational outcomes to which the program will contribute

Upstream Stakeholders

Several stakeholders played a role in deciding to develop courseware which could be delivered online and advocating for dL delivery of SLC as well as the actual course development and delivery.

Direct Impactees

The past and future students who attend(ed) MC SLC are the direct impactees of the program. They are all NCOs who have been selected for promotion to SFC and require the course for career progression.

Indirect Impactees

Success or failure of MI SLC impacts on not only the direct impactees, but also the soldiers and NCOs the direct impactees supervise, leadership at the direct impactees’ unit of assignment, and the operational force

The primary indirect impactees of the program are the soldiers and NCOs course attendees supervise. The secondary indirect impactees of the program are the supervisors, managers and leaders who make up the command teams at units of assignment.

Evaluation Request

Evaluation Feasibility and Risk Factors

Upon conservation with the upstream stakeholders and analyzing available information, the evaluation team assessed the evaluation feasibility and risk factors of the evaluation project. The team’s assessment of evaluation feasibility revealed that it was a feasible evaluation project to implement and complete within the available time frame. The evaluation team also identified and monitored the project risk factors throughout the project.

Evaluation Purpose and Type

Through discussions with the MC NCOA Commandant, Deputy Commandant, and Education Specialist, the evaluation team learned that the overall goal of this evaluation is to identify areas for improvement in course delivery of the recently implemented SLC dL course for students and assigned instructors. Thus, this is a formative evaluation for which suggestions for improvement will be advised. Because the evaluation aims to assess the efficacy of the course material in the distance learning format as well as uncover unintended outcomes, this is a goal-based evaluation with a goal-free approach.

Results

As the overall approach to this evaluation, the evaluation team followed Chyung's (2019) 10-step evaluation process. The 10-step process guided the evaluation team to conduct a thorough, systematic evaluation that focuses on the upstream stakeholders' and direct impactees' needs and supports the upstream stakeholders' intended use of the evaluation findings.

In conjunction, the evaluation team used several levels of Kirkpatrick's (1996) four-level training evaluation framework to evaluate the three dimensions:

• Course Implementation (Level 1 Reaction, based on participants’ inputs about how well the course was implemented as distance learning and allows

them to be successful)

• Instructor Preparation (Level 1 Reaction, based on participants’ inputs about how well instructors support student learning)

• Training Management (Level 1 Reaction, based on participants’ input about how well the instruction supports/supported their learning)

The evaluation team also used three sources of data to investigate these dimensions:

• Alumni

• Current Students

• Instructors

Three methods of data collection were used as well:

• Self-administered surveys

• Extant data

• Teleconference interviews

Conclusions

Recommendations

The evaluation team used extant data from end-of-course surveys and assessment results of alumni, an anonymous web-based survey of current students, alumni and supervisors of alumni and interviews with assigned instructors to gather and analyze data for this evaluation report.

Dimension 1 Recommendations

Addressed:

• Course Welcome Information

• Student familiarity with MS Teams

• Ability to complete coursework on time

• Organization of Course

Dimension 2 Recommendations

Addressed:

• Course administration

• Lack of Test Control

• Instructor performance

Dimension 3 Recommendations

The evaluation team did not provide recommendations for Dimension Three, Training Management, due to the low response rate from supervisors of course alumni.

By maintaining what is working well and incorporating some changes to existing practices, the SLC dL program can improve course delivery for students in the operational force and for assigned instructors.

References

Brinkerhoff, R. O. (2006). Telling training’s story: Evaluation made simple, credible, and effective. Berrett-Koehler.

Clark, R.C. (2008). Building Expertise: Cognitive Methods for Training and Performance Improvement (3rd ed.). Pfeiffer.

Clark, R. C., & Mayer, R. E. (2016). e-Learning and the science of instruction: Proven guidelines for consumers and designers of multimedia learning (4th ed.). Wiley.

Chyung, S. Y. (2019). 10-step evaluation for training and performance improvement. Thousand Sage.

Deinhardt., C., Baker, N., Johnson. T. (2020, Dec 10). Evaluation: MC senior leader course - dL: Formative [MS Word document]. OPWL 530 course site. https://Blackboard.boisestate.edu

Kirkpatrick, D. (1996). Evaluating training programs: The four levels. Berrett-Koehler.

Sparr, J. L., Knipfer, K., & Willems, F. (2016). How leaders can get the most out of formal training: The significance of feedback-seeking and reflection as informal learning behaviors. Human Resource Development Quarterly, 28(1), 29–54. https://doi.org/10.1002/hrdq.21263

Willmore, J. (2018). Job Aids Basics: Vol. 2nd edition. Association for Talent Development

Appendices

Appendix A. OPWL Portfolio Worksheets for the Evaluation Project