|
Notes for T4 Kurt Rowley
3. Evaluating WBT Effectiveness: Using systems theory to help see the big picture Evaluating WBT effectiveness is challenging given the characteristics of distributed learning environments. The application of systems theory is one approach to incorporating a 'big picture' view in evaluation efforts.
3.1 Introduction to systems theory Systems theory has emerged as a way of thinking about the interactions that occur between individual parts of a system, as well as between the parts and the system's environment. For example, different specialists study the biology, botany, geology, etc. of the rain forest, where the plants, animal life, soil composition, etc. have characteristics unique to that particular environment. The complex ecology of the rainforest is governed by a series of communications and control mechanisms that monitor environmental inputs (such as rain and sunlight) and outputs (such as plant growth and food production) as well as the impact each factor has on the relationships between the component parts. When considered as a whole, the system presents 'emergent' properties, or properties that exist only as a result of interactions between the parts. A good example of interactions among system components lies in the role insects and birds play in plant reproduction. In other words, the whole is greater than the sum of its parts. Systems theory then deals with "an abstract idea of a whole having emergent properties, a layered structure, and processes of communication and control which in principle enable it [the system] to survive in a changing environment" (Checkland & Scholes, p. 22, 1990). Systems theory emerged as a field of study during the 1940s and 1950s with the goal of creating a taxonomy of characteristics that could be consistently identified across a variety of system types. Initial work concentrated in the field of biology, but rapidly spread through the natural sciences and into the social sciences. Although little progress has been made toward developing a unified system's taxonomy, a diverse body of literature has evolved within the past 50 years, replete with ideas, theories, and the products of thinking about systems. Many systems theory concepts guided the organization of the complex logistical problems encountered when NASA engineered the early space program. Instructional design was also an early product of systems thinking. 3.2 Systems view and WBT evaluation Why is a systems view an important part of evaluation, what is different about this new approach? Our academic culture and technology are to a large part based on "mathematical thinking." The methodologies of the physical and even social sciences have traditionally been based on the belief that phenomenon around us can be described through mathematical reasoning. This approach leads to a view of the world being composed of variables interacting in somewhat simplified cause and effect formulas. This view allows us to predict phenomenon in many arenas, such as in the mechanical world, or even the financial world, but it clearly ha limited powers to explain real-world complexities such as movements in the stock market, or the fact that grades in school can not predict career success. Mathematical reasoning often leads to management of organizations using "linear" reasoning. Examples of this type of view might include the use of overly-simplified measures of student performance such as grades rather than portfolio assessment, attempting to improve education and training through manipulating simple variables such as class size, or length of time on task in order to improve results, etc. Systems thinking is a broadened view of this mathematical, scientific paradigm. With systems thinking we view the world as being composed of interacting "systems." The phenomenon of interaction are complex and it is difficult to reduce them into mathematical formulas. Under systems thinking, the systems of the world interact based on logical but ill-defined rules and guiding principles which may or may not be intuitive to us. A systems view is a perspective of trying to understand the various patterns which occur and recur in systems and their subsystems. With a systems view we endeavor to learn how subsystems interact together towards common purposes. This leads to a view that we must develop a "profound understanding" of our systems during any design evaluation. For example, we must first learn what motivates students to achieve, then use that information to continuously improve results. We first learn how the students tend to interact in the classrooms, on the web, at home and with peers, and then use that information to improve the instructional process. Clearly this systems view requires careful and deliberate thinking about cause and effect, means and ends. It is a view of complex systems in a delicate balance. With mathematical thinking we believed we could improve the world by manipulating a few "variables." What was missing was the recognition that the system response to that manipulation was a complex and somewhat chaotic interaction of various sub-systems, not just a predictable, formula-driven phenomenon. Also, we were hard-pressed to design better systems, or to improve the functioning of systems by adjusting their structure and rules. By using a systems view we acknowledge that we must change systems, subsystems, and processes. We must be prepared to revise anything and everything in the system. This could mean evaluating the patterns of interaction, the expectations of performance, the available technologies, perhaps even the characteristics of the roles of all of the participants in education if we wish to understand how effective new systems are, or how to improve the designs of new systems.
3.3 Challenges of WBT/Distance Learning (DL): Learner often more disciplined / self-directed
Example (experience with BSU, wonderful experiences but also the problems were harder to solve...)
3.4 Review of systems theories, with application to DL identified General Systems Theory
EXAMPLE for WBT: An example of this process might be to use GST terms to help diagram the design of an asynchronous learning environment using the world-wide web. System components would include the role of the teacher as a decision-making and information-producing sub-system, who interacts with the sub-system of the software and computer delivery environment, and the student as a decision-making, learning, and skill-performing sub-system, who also interacts with the software and computer sub-system. System boundaries include constraints of the natural limits of the interaction of the teacher and student sub-systems through the intermediary sub-system of the web environment. By considering the boundaries of this scenario, the designer would realize that the exchange of information between teacher and student is a critical factor due to the complexity of problems and questions that might arise from this information exchange and the narrow bandwidth of web-based systems. To address these challenges to the systems design, the designer might decide to increase the bandwidth of the communication by adding audio links to the web-based environment, or by requiring regular office hours when the instructor will be available for written or spoken chats.
Cybernetics
Example for WBT. Not every system contains cybernetic components, but systems that mimic the control functions of natural organisms, such as WBT distance learning environments must contain some cybernetic qualities. In order to identify the requirements for cybernetic functions, it is useful to inquire into the nature of the dynamic environment of distance learning. One aspect of a typical distance learning environment that could require cybernetic capability is the operation of the delivery system (WBT). Given the diverse needs of students and the single electronic presence of the teacher in most distance learning environments, there may be a need for regular feedback from the students with regard to their individual needs. In a cybernetic system, this feedback would serve as a controlling mechanism. Thus, a controlling mechanism for adjustment to the processes comprising the delivery system must also be available. This might be a regularly-scheduled stopping point in the course where the students complete feedback forms and submit them to the instructor. Finally, the instructor must be able to adjust the system by means of the controlling mechanism, by making modifications to the course assignments, or other aspects of the course curriculum.
Chaotic systems Chaos theory and chaotic system behavior has only recently begun to transfer fro and m the physical to the social sciences. Chaos theory describes the behavior of highly complex systems under conditions of turbulence and change. The theory defines highly complex systems, and displays with stunning realism how many natural systems appear to follow these simple principles (Gleick, 1987). In Chaotic systems small differences in interactions can produce large effects. An example of a chaotic system is the dripping of water from a leaky faucet. The surface tensions and fluid dynamics are such that it is virtually impossible to predict the timing of the next drip. Still, when graphing the time intervals of the drips, certain patterns and boundaries emerge. This is a simple but fundamental condition of chaos. Other frequently cited examples include the weather and the turbulence of a stream of water. While predicting behavior patterns for these systems might be possible if their characteristics were perfectly known, the tiniest deviations in conditions from the least expected values will have exponential impact as the effect proceeds, or the system operates, until the entire system flow has deviated from the prediction. A few brief generalizations which may follow from the concepts of chaotic systems:
Example, the initial conditions in a distance learning environment often set the
atmosphere for the remainder of the course. Initial technical problems with the
use of e-mail or other computer-related problems could potentially alter the
outcomes of the entire course 3.5 Other systems theory-based discipline Complexity (fractals, complex procedural geometry). Complexity is an area of math and geometry in which simple formulas are repeated in procedures to produce unique patterns, such as fractals. These patterns replicate across scale and seem to be found regularly in the geometry of nature. Systems thinking. This is an engineering application of systems theory, and includes systematic processes for managing large-scale, complex operations. The early years of the NASA space program were the first large-scale application of systems thinking. There are many formalizations of systems thinking, and it has been applied to many domains. Systems Design (Instructional Design, Soft Systems Design, etc.). Systems design disciplines generally involve systematic approaches to design that includes analysis of conditions, problem-solving strategies for developing system performance objectives and possible design approaches, then implementation and evaluation stages. The use of systems theory occurs at many levels in these disciplines, including a fundamental focus on creating systems in which every element contributes to the accomplishment of the defined objectives. Quality Improvement started as a systems discipline, including a high degree of cybernetic-type feedback from all customers, and formal tracking mechanisms for the measurement of improvements in performance of the system. Learning Organizations started as Cybernetics, with a focus on simultaneous improvement of all elements of the organization. In this view, social organizations are systems, and processes used within organizations are documented and evaluated over intervals of time. An interesting focus on the identification of critical counter-intuitive processes in learning organization theory is borrowed directly from the original cybernetics theory.
3.6 Using systems theory guidelines to help evaluate WBT A synthesis of the systems theory literature reveals many possible applications of systems theory to WBT evaluation. Due to the complexity of distance learning environments, these approaches to evaluation are particularly important. Systems theory can help the designer take into consideration complexities that ordinary evaluation can not easily include. By identifying a few recurring concepts which seem to emerge from some of the major applications of systems theory, it is possible to construct a few evaluation principles for using systems theory to assist in the improvement and validation of WBT systems. Some guidelines for applying systems theory, have been listed below. These guidelines can be used to form questions for evaluation processes, or they can be used to help give focus to evaluation processes. 1. Identify the components and characteristics of your sub-, co-, and supra-systems. Systems thinking reflects the reality of the complex, multi-dimensional organization of actual systems. 2. Think about the impact of your actions on components and on the whole system over time. It is impossible to understand parts without seeing the moving whole. Recognize the interrelationships between events in the changes to the system over time. Don't be blinded by parochial thinking. 3. Learn the languages of other departments, other components of the system. It is impossible to have common goals without common understanding and communication. Find ways to communicate at the level of concept understanding - pure thought if possible - and that may mean learning alternate uses of words, etc. Still, avoid coining new terms or new usages of words if possible. New terms or usages rarely become part of the common vernacular - most cultures only generate a few new words each year. If you can not use common terms to express new concepts, you are not likely to communicate effectively except with great effort. 4. Seek to understand the processes of your systems. Learn how they interact with other systems. Systems must be understood to be optimized. 5. Identify the control functions of your systems, and how they maintain and adapt the system to its environment. These may be motivators, incentives, or even "strange attractors" such as recognition or altruism. Systems will naturally change and adapt to feedback based on control. 6. Track all vital flows in your system. Keep track of the flows of resources, information, authority, customer support materials, finances, and so on. Know how the flows work, and recognize when they have problems, they are the life of the system! 7. Stay customer-centered no matter what. If it's not good for the customer, it will eventually not be good for you either. This is a systemic feedback function. Make sure that feedback from the customer feeds directly to the "top," and is not filtered through the organization. 8. Listen to non-traditional sources for system improvement input. Create an improvement culture and be sure there are no disincentives to improvement. 9. Validate the design of your systems. Even if the system has no planned design, there is an implicit design. Verify that the design makes sense by using systems inquiry and design tools. 10 Do not forget the unpredictability of complex systems. Realize that complex problems often have simple solutions, but be careful to protect and nurture start-up situations and system changes. They can be highly sensitive to problems. 11 Learn the culture and adoption characteristics of systems targeted for change. Expect only small successes at first, but prepare for more if the innovation filters successfully through the communication networks of the early adopters. 12 Expect resistance to a systemic change when the innovation crosses cultural mores and rules, or when it encounters poor attitudes among the adopters. Plan to defeat that resistance ahead of time. Develop a system diffusion or communication plan for the innovation. In this plan study the culture, and design a campaign to promote the necessary attitudes for change. Execute and monitor that plan. 13 Involve all of the players - otherwise it is not true systems thinking. 14 Take a standard analysis approach, but look for perturbations and exceptions as keys to understanding the working issues of the system, and finding the problem areas. The reality of complex, chaotic systems dictates that only by understanding the long-term patterns of chaos will we be able to identify the core principles, or "attractors" at work. 15 Learn from rational metaphors - see what others have done, know that improvements have worked well in the past and have also failed in the past. After all, most of systems thinking utilizes operational metaphors as fundamental tools. For that matter, all of our communication involves metaphors. Still, don't forget that metaphor is not the same as the authentic system. Recognize that true understanding of a system lies in comprehension of the thoughts behind the design and operation of the system, not in its metaphors. 16 Don't quit too soon, push out the envelope of your scope of the system. Notice how many other systems depend on you, both sub- and supra-systems of yours. They are all interested in your welfare, you need only bring systems issues to their attention in a rational fashion, and solicit their help. Many times seemingly isolated problems are really linked to the larger supra-systems that they are a part of. 17 Be interested in the issues and problems of the systems processes you depend on. Without becoming involved in some of the details of the processes, it is impossible to understand the cause and effect chains of the results of the organization. For example, in educational systems, if student family problems are ruining your chances at instructional interventions working, get involved with the families to the extent that you can, and help them. You will only be helping yourself succeed if you help your client's sub- and supra-systems. 18 Assess the agendas of each of the components of the systems. Ask each member of the organizational system what their goals are, and what they are working towards. Assess authentically what their goals are by their results of their labors. If the operational agendas are not aligned with the common visions, you will never be a completely functional, nor predictable system. Remember that the best way to change, or align agendas is to just get people working towards the common vision. When success results, the agendas should begin to shift automatically. As the people work, the organization can also learn. (get into the trenches, leaders and designers solve more real problems if they are in the middle of them) 19 Expect design and evaluation to be somewhat ambiguous - just work toward coalescence - the reality of complexity and the principles of chaotic systems guarantee that good design will be difficult to create by deliberate efforts, but may be found during applicaiton of valid core principles. Collaboration during design is important with any complex, open system. 20 Do not assume that a complex problem requires a complex solution, or that a simple problem will give in to a simple solution. Based on chaos research, the simplest systemic problems often require complex and involved solutions, and complex systemic problems often have very simple solutions. When problems are found during evaluation, don't jump to conclusions about the best solution.
3.7 Discussion questions What are some typical systems challenges in DL? (Think of systems theories) How can these systems approaches (and their various methodologies) be applied effectively within the constraints of real-world WBT development? What is the possible ROI of using a specific systems principle in a DL design situation?
|
|
© Copyright 2006, Kurt Rowley, Ph.D. |