Improving Courses Across an Online Program: A ... - Semantic Scholar

28th Annual Conference on Distance Teaching & Learning

For more resources: http://www.uwex.edu/disted/conference

Improving Courses Across an Online Program: A Design-Based Approach Karen Swan Stukel Professor of Educational Leadership University of Illinois Springfield Daniel Matthews Associate Professor of Educational Leadership University of Illinois Springfield

Introduction Design-based research blends empirical research with the theory-based design of learning environments. It centers on the systematic investigation of innovations designed to improve educational practice through an iterative process of design, development, implementation and analysis in real-world settings (Wang & Hannafin, 2005). Design-based research helps us understand “how, when, and why educational innovations work in practice” (Design-based Research Collective, 2003, p. 5), because the innovations it explores are grounded in educational theory. This paper discusses how faculty in a fully online graduate program are using a collaborative, designbased approach grounded in both instructional design theory and the Community of Inquiry framework to incrementally improve their courses over time. Following an initial peer review and corresponding revisions based on the Quality Matters rubric to improve course design, these educators are making iterative improvements in course implementation based on student responses to the Community of Inquiry survey. Preliminary findings suggest that this approach can yield significant improvements in learning outcomes. Background The fully online Master of Arts in Teacher Leadership (MTL) program at the University of Illinois Springfield was the 2010 winner of the Sloan-C award for Outstanding Online Program. Faculty from the program are using a design-based approach to improve core courses in their program. The approach combines an initial Quality Matters (QM) peer review to address course design, with the iterative review of student responses to the Community of Inquiry (CoI) survey to incrementally improve course implementation. Quality Matters (QM) is a faculty-oriented, peer review process designed to assure quality in online and blended courses. The rubric is based on instructional design principles (Quality Matters, 2005) and is organized around eight categories – course overview, learner objectives, assessment and measurement, resources and materials, learner engagement, course technology, learner support, and accessibility. MTL faculty are using the QM review process to ensure that all their core courses meet course design standards in each of these areas. The QM framework, however, only addresses course design. The Community of Inquiry (CoI) framework (Garrison, Anderson & Archer, 2000), on the other hand, addresses learning processes. It represents online learning experiences as a function of relationships among three presences -- social, teaching, and cognitive -- and views all three as working together to support deep and meaningful learning. MTL faculty are using student responses to the 34 item (13 teaching presence, 9 social presence, and 12 cognitive presence items) CoI survey (Arbaugh, et al, 2009; Swan et al., 2008) to incrementally improve both the design and facilitation of their courses.

Copyright 2012 The Board of Regents of the University of Wisconsin System

1



Finally, to measure the effectiveness of this design-based approach, MTL researchers are collecting data on learning outcomes in all the courses involved. Grades on major assignments as well as overall grades in each course are standardized to percent correct scores and compared semester to semester to track the effects of the course redesign efforts. Our Design-Based, Iterative Approach In recent years, a number of researchers have begun to incorporate design-base approaches to improve online and blended courses and programs. The approach used in our work is generally consistent with Anderson and Shattuck’s (2012) summary of characteristics of design-based research. Anderson and Shattuck note that design-based research is: • • • • • •

Situated in real educational contexts; Focused on design and testing of significant interventions; Employs mixed methods; Involves multiple interventions; Involves collaborative partnerships between researchers and practitioners; and Evolves design principles.

Figure 1. Iterative, theory-driven, collaborative, course redesign model Our application of design-based research can be summarized as encompassing three broad stages. We began with theory-based frameworks, Quality Matters and Community of Inquiry, which were used as compasses to guide the overall direction of our work. Second, we generated data related to those frameworks, and that data provided empirical guidance to course revisions. Third, we established a collaborative, teacher-scholar approach to implementing improvements in our courses. Figure 1 summarizes our approach in a way that emphasizes the ongoing, iterative nature of the interaction between theory and data, and our focus on collaborative continuous improvement (Matthews, Bogle, Boles, Day, & Swan, in press).


2



Quality Matters (QM) revisions were guided by course reviews completed by an instructional designer and two faculty with educational leadership expertise using the QM Rubric. Across faculty and courses, QM revisions centered on the development of module level objectives and linking these to corresponding assessments. The QM review resolves to either meeting or not meeting QM standards. In our case, none of our core courses initially met the standards, but all met them after a single revision. Community of Inquiry (CoI) revisions followed a semester after the implementation of QM revisions and was based on student responses to the CoI survey. Unlike the QM review, these are based on student perceptions of learning processes in the courses relative to the 34 teaching, social and cognitive presence items. We thus reviewed students’ average ratings for each item to identify those with the lowest ratings, and then met to consider how these ratings might be improved. Because continuous improvement in CoI scores is possible, the CoI review and revision processes were iterative; that is, they were repeated semester to semester (See Figure 2).

Figure 2. Sequence of revisions Preliminary Results Preliminary results involving one course in Educational Research Methods showed statistically significant improvements over a period of four semesters (Swan, Matthews, Bogle, Boles & Day, 2012). Outcome measures for this course included final exam scores, grades on a research proposal, and final course grades, all standardized to percent correct. Analysis of variance found significant differences for the final exam scores at the p=.05 level and for overall course grades at the p=.001 level. Post hoc analyses showed these were only significant when comparing baseline and final semester scores. Differences in scores on the research proposal were not significant. Cohen’s (1992) analysis of eta squared results was used to calculate effect sizes for all result. Effect sizes of the cumulative QM/CoI revisions were small for the research proposal (.11) and the final exam (.16), but moderate for overall course grades (.29). These results suggest that the combination of an initial QM revision followed by iterative CoI revisions across multiple semesters can result in improved learning outcomes. Our ongoing research will explore whether or not these findings hold across core courses in our department. Conclusions This study demonstrates the value of an iterative approach to improving online courses using the Quality Matters design model and the Community of Inquiry framework as assessment guides. While the QM and CoI models view learning from different epistemological perspectives, they complement each other and


3



increased the average scores in all of the identified student outcomes over a four semester period of time. The Quality Matters design analysis provides a faculty-oriented, peer review process designed to assure quality in online course structure and, when combined with the CoI model provides the instructor with two separate but important views designed to enhance student learning. The Community of Inquiry survey provides a structure to determine student satisfaction with the course in the teaching, social. We believe that it is possible that other theoretical frameworks that address course design and online learning process could be used to drive the revision process, but the adjustments to the course designs and the analysis of student achievement must be ongoing. It also requires instructors to make their courses available for analysis, adjust their course designed based on these analyses and collaborate with their colleagues in a constructive manner. These analyses, therefore, must be presented in an environment that is safe for all and where improvement is the goal. References Anderson, T. & Shattuck, K. (2012). Design-based research: A decade of progress in education research? Educational Research, 41(1), 16-25. Arbaugh, J. B., Cleveland-Innes, M., Diaz, S. R., Garrison, D. R., Ice, P., Richardson, J. C., Shea, P. & Swan, K. (2008). Developing a community of inquiry instrument: Testing a measure of the Community of Inquiry framework using a multi-institutional sample. The Internet and Higher Education, 11 (3–4), 133–136. Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5–8. Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education 2, 87–105. Matthews, D., Bogle, L., Boles, E., Day, S., & Swan, K. (2012). Developing communities of inquiry in online courses: A design-based approach. In Z. Akyol & D. R. Garrison (Eds). Educational Communities of Inquiry: Theoretical Framework, Research and Practice. Hershey, PA: IGI Global. Swan, K., Matthews, D., Bogle, L., Boles, E. & Day, S. (2012). Linking online course design and implementation to learning outcomes: A design experiment, The Internet and Higher Education 15, 81-88. Swan, K. P., Richardson, J. C., Ice, P., Garrison, D. R., Cleveland-Innes, M. & Arbaugh, J. B. (2008). Validating a measurement tool of presence in online communities of inquiry. e-mentor, 2 (24) at http://www.e-mentor.edu.pl/artykul_v2.php?numer=24&id=543 Wang, F., & Hannafin, M. J. (2005). Design-based research and technology enhanced learning environments. Educational Technology Research and Development, 53(4), 5-23. About the Presenters Karen Swan is the Stukel Professor of Educational Leadership and a Research Director in the Center for Online Learning, Research and Service (COLRS) at the University of Illinois Springfield. Dr. Swan’s research is in the general area of technology and learning on which she has published and presented extensively. Her current interests center on online learning. Dr. Swan received the 2006 Sloan Consortium award for Outstanding Achievement in Online Learning by an Individual and in 2010 was inducted into the first class of Sloan-C Fellows, and the 2010 Distinguished Alumnus award from Teachers College, Columbia University. Email: Phone:

[email protected] 217-206-7424


4



Daniel Matthews is an associate professor in the Department of Educational Leadership and a Faculty Research Fellow with the UIS Center for Online Learning, Research and Service at the University of Illinois at Springfield. His responsibilities include teaching on-campus and online educational research methods courses. His current scholarship focuses on improving online education through the use of design-based research that encompasses the Quality Matters and Community of Inquiry Frameworks. Address:

Email: Phone: FAX:

University of Illinois Springfield One University Plaza, MS 377 Springfield, IL 62703 [email protected] 217-206-7521 217-206-6494


5