A Generic Instructional Vocabulary for an Interoperable Distributed Learning Environment: Adapting MAESTRO the Writing Process Tutor
Kurt Rowley
Advanced Training Technologies Branch U.S. Air Force Research Laboratory (AL/HRCT) 7909 Lindbergh Dr., Brooks AFB, TX 78235-5352, USA E-mail: RowleyK@cobra.brooks.af.mil
Abstract
The MAESTRO research program is part of an ongoing USAF effort to design intelligent tutors that will meet the needs of teachers and students in a classroom environment. MAESTRO teaches the procedural skills of an expert writer, based on cognitive research into the writing process. MAESTRO was written in Asymetrix Toolbook and implemented successfully at 12 High Schools as part of the 9th grade writing curriculum. A simple generic instructional vocabulary based on an instructional system ontology is being developed for MAESTRO in order to meet expected future needs for tutors that function in an interoperable distributed learning environment (DLE). The vocabulary will eventually be implemented and improved as part of an interoperable DLE architecture to be tested with multiple tutors, including a new version of MAESTRO. Ongoing research into the implementation of interoperable DLEs will help address future USAF needs for interoperable tutoring components for advanced technology training/tutoring systems, distributed team tutoring environments, as well as traditional automated training.
Keywords interoperability; reusability; distributed learning environment (DLE); intelligent tutoring system (ITS); domain-independent instructional models
1. Introduction
The challenges of interoperability, reusability, and standardization are not unique to the ITS community, as many domains of instructional and educational computing are currently struggling with similar issues (Rowley, 1995; 1996; Lynch, 1993). Competitive forces in the computer world often lead to a divergence of standards, with a lack of cooperation leading to closed proprietary standards, or de facto standards that resist improvement (Molka, 1992). However, both developers and users benefit from the development of open, forward-thinking standards of interoperability. The ITS community is in an interesting position with regard to the challenge of interoperability and reusability standards. The goal of highly effective individualized instruction through intelligent tutoring has remained somewhat elusive due at least in part to the challenges associated with implementation of ITS technology. Some of the reasons given for failure of wide-spread application of ITS in training and education include difficulties in the generalizeability of specific tutor designs, and problems transitioning tutor models into field implementation (Jona & Korcuska, 1996; Yum & Crawford, 1996). Interoperability is one potential approach to improving the generalizeability of ITS implementations, and improving the ability of tutors to be more easily designed and developed to meet specialized field needs.
In order for standards of interoperability to be useful in the long run, several theoretical and technical challenges must be addressed. This includes the challenge of how to represent instructional strategies to allow for the sharing of those strategies among tutors using both common and divergent knowledge and instructional resources. To be successful, interoperability standards must supply sufficient technical generalizeability to meet the needs of researchers and developers, as well as the application needs of the end-users. This requires that interoperability standards have both a sound footing in learning and computational theory, as well as the capability of meeting the needs and requirements of users. In order to study how to construct an interoperable tutoring environment that could be readily implemented and meet real-world end-user computing and training needs, we are developing a new generic instructional vocabulary for use with the instructional knowledge bases and instructional resources of an intelligent tutor. We selected a successful, previously implemented tutoring system as the test-bed for evaluation of the vocabulary. The tutor selected is MAESTRO, a writing process tutor developed and implemented as part of a U.S. Air Force technology transfer project, and tested in a large-scale year-long study with over 3,000 high school students (Rowley, in press; Rowley & Crevoisier, 1997). The MAESTRO Writing Process Tutor research was conducted through the U.S. Air Force's Fundamental Skills Training (FST) Program. FST is a multi-year research effort to transfer advanced, adaptive training technology capabilities developed under Air Force technical training research to public education. As part of an interoperability study, the FST MAESTRO tutor will be re-designed to work in an interoperable Internet-based distributed learning environment (DLE). The new version of MAESTRO is being designed to met the needs of a distributed user audience, and with an interoperable architecture that will eventually allow for the sharing some of its of resources among multiple tutors.
In order to present the case for converting an existing ITS to an interoperable system and testing a generic instructional vocabulary, this paper will start with a description of the current MAESTRO tutor design, briefly describe the field-based testing and evaluation of the tutor, propose a generic instructional vocabulary for an interoperable DLE version of MAESTRO, discuss an ideal interoperable DLE architecture for multiple tutors, and finally consider future Air Force needs for interoperable ITS DLEs.
2. A Procedural Skills Tutor
MAESTRO teaches the procedural skills of an expert writer, based on cognitive research into the writing process, and several years of evaluative teacher feedback. The expert writing process has been identified in some detail through a now classic protocol analysis of an expert writer (Flower & Hayes, 1980). The major components of these writing tasks include goal-setting, generating ideas, developing a writing plan, translating ideas into text, and revising the text with regard to the original writing goals. Based on years of cognitive research into the writing process, Bereiter and Scardemalia recommend a supportive environment for instruction in the writing process, noting that "The use of procedural facilitation--simplified routines and external supports--can help students through the initial stages of acquiring more complex executive processes..."; (Bereiter & Scardemalia, 1989, p. 363).
MAESTRO is designed to provide procedural facilitation through the use of a tutored writing environment. Using MAESTRO, the student can navigate through, and perform work in 22 workspaces that simulate procedural tasks performed by the expert writer (Fig. 1). The MAESTRO student interface helps the student develop an expert-level mental model of the writing process. MAESTRO monitors the student's use of the workspaces in order to determine the level of mastery of the writing process achieved by the student, control the student's access to the writing environment according to the student's level of mastery of the expert writing process, and provide relevant coaching and advice.
MAESTRO was designed to be generic in the sense that the controlling module does not reference specific workspaces by name, but accesses all instructional and tutorial resources according to rules. The rules and knowledge bases have been formalized and are coded directly into MAESTRO using the 'Open Script' language. The rules and knowledge bases allow the tutor to produce adaptive individualized instruction, by comparing student operation of the workspaces with a model of the expert writing process.
3. Field Implementation
MAESTRO was written in Asymetrix Toolbook and implemented successfully
over LANs in computer labs at 12 High Schools in the U.S. during the 1996-97 school year
in a traditional controlled experiment. The tutor was implemented as a regular part of the
writing curriculum. The tutor was designed to be integrated with the classroom. MAESTRO
supports writing assignments selected or supplied by the English teacher. Ongoing
research is addressing the effectiveness of MAESTRO.
Figure 1: MAESTRO Knowledge/Skills Hierarchy
Initial responses to MAESTRO in the classroom have been very positive, with both teachers and students providing extensive feedback related to use of the tutor. Field input related to difficulties with the tutor have been carefully considered, and MAESTRO has undergone minor revisions in order to improve usability. In general, the teachers are satisfied with the performance of MAESTRO and many have indicated that they can trace improvements in the students' writing coherence to use of the tutor. The students have for the most part quickly learned the interface, and learn the writing process skills taught in the workspaces. Traditional motivation problems of students appear to be reduced in most cases, although some of the teachers and students have commented that they favor some of the writing assignments and workspaces. In general, the tutor's classroom effectiveness appears to be consistent with studies of related writing tutors (Carlson & Miller, 1996; Rowley & Miller, in press; Zellermayer, Salomon, Globerson & Givon, 1991). Ongoing experimental measures of changes in student writing quality will provide additional insight into the use of MAESTRO. Due to the success of the MAESTRO tutor, and related writing tutors, this tutor will be adapted to work with an interoperable DLE. The adaptation of MAESTRO will begin with development of a generic representation of its rules and knowledge bases.
4. Generic Instructional Vocabulary
In order to meet expected future needs for a writing tutor that can function in an interoperable distributed learning environment (DLE) a generic, domain-independent instructional vocabulary based on an instructional system ontology is being developed. The vocabulary is designed to provide a standardized, domain-independent representation of an instructional strategy that will allow an instructional approach to be easily reused across multiple DLE systems. The vocabulary will be tested with a new interoperable DLE version of MAESTRO's instructional knowledge base. Mizoguchi and associates (1996) argue that a well-designed common vocabulary can be developed through the engineering of a task ontology, much like the use of knowledge engineering in expert systems. Their view of a task ontology is different from a traditional domain ontology, which describes pure domain knowledge. They suggest that a task ontology be focused on representing a problem-solving process, with the representation broken down into four generic components: nouns representing objects, verbs representing activities, adjectives modifying noun objects, and other concepts specific to the task (see 3.1 in their paper, 1996). This type of task ontology is an appropriate construct for instructional knowledge base primitives, as it allows for the representation of the components of an instructional system and their interrelationships in an objective-seeking problem solving context.
In order to develop a generic instructional vocabulary, our taxonomy
uses terminology that allows a broad range of instructional strategies to be represented
in a sentence-like form. The vocabulary is composed of the terms that can be objects,
activities, attributes, goals, or criteria. These terms can be
combined using a simple grammar in order to represent instructional strategies in a
sentence-like form. The sentences follow the form of: "objects with various attributes
perform activities that accomplish instructional objectives as measured
by criteria." The proposed vocabulary is designed as a generic tool that can
be used to represent MAESTRO's instructional strategies, and ultimately will allow
instructional strategy information to be shared between diverse tutoring systems. The
following list includes a sample of the terms used for the vocabulary. Note that the list
of terms is not complete at this time, as the vocabulary is still under development.
Proposed Generic Instructional Terminology
Structural Objects (showing only top level of the task ontology)
- learner (lower level by traits, attributes, prerequisite knowledge and skills, history, etc.)
- instructor (human)
- cohort/peer learner(s) (in the learning environment, or as team members)
- instructional resources (interactive software systems, etc.)
- reference information (print or electronic indexing)
- tools (physical or electronic)
- facilities (physical)
- environment (human, physical, or electronic)
Activities of Objects (functions of structural objects and their attributes over time)
- observing or otherwise sensing (human, all inputs)
- learning declarative information
- learning procedural skill
- remembering (human, contextual)
- deciding (human choice)
- reasoning (human cognition)
- exploring
- requesting feedback or information
- responding to inquiries
- practicing or performing (authentic activity)
- interacting with teammates or peers
- playing (relaxing or recreation)
- presenting information
- reminding of information or process steps
- showing
- assessing performance
- controlling learning process
- managing learning environment
- monitoring work or performance and providing feedback
- interacting cooperatively
- interacting competitively
- interacting socially
- presenting (print, electronic)
- connecting (remote, networking)
- simulating (kinetic or cognitive performance environment)
- reminding (print, electronic)
- showing (print, electronic)
- assessing (performance)
- controlling (print, electronic)
- monitoring (electronic)
learner
instructor
cohort / peer learner
instructional resource (interactive software system)
Modifying Attributes for activities of objects (modifiers for any
level of ontology)
- communication between and among
- time required for XX (learning, practice, performing, etc.)
- storage space (physical and electronic)
- capability or capacity (human intelligence and abilities)
- interest in (human)
- level of mastery of (learning, performance)
- use of cognitive tool XX to (software object)
- use instructional resource XX to (software object)
- looking-up reference information in order to (electronic)
Instructional Goals of activities (skill representation, the purpose or objective of an activity)
- a fact or concept is known
- a skill or task has been (or can be) performed independently or interdependently
- a choice or decision has been (or can be) reached
- personal or group needs have been (or can be) met
- goal achieved or progress made toward goals (or rewards)
- student successfully supported or guided during learning and practice
- directed or controlled use of technology by student
- assessed and certified student performance
- managed learning environment for multiple (potentially interacting) students
- designed curriculum for multiple students to accomplish educational objectives
- domain knowledge and skills improved (instructor's knowledge)
- interacted cooperatively/collaboratively with learner during teacher-directed work
- socialized learner
- provided peer product feedback or review
- provided peer tutoring or mentoring
- participated in collaborative group learning activities
- capabilities of a human instructor extended
- learner coached
- dangerous or expensive learning environments simulated
- individualized instruction provided (adaptively)
- timely instruction or feedback provided (just-in-time)
- individualized, self-paced instructional environment provided
- capabilities and developing skills of student and adapt instruction addressed
- instruction sequenced (adaptively)
- ability or proficiency of student assessed
learner
instructor
cohort / peer learner
instructional resources (interactive software system)
Performance Criteria for goal accomplishment
- targeted instructional and educational outcomes improve by XX amount
- accomplishment of XX objectives & purposes demonstrated by YY (specific deliverables)
- XX instructor time (freed for more productive and individualized interaction with students)
- students, instructors are more satisfied as measured by XX
- improved performance as evidenced by XX measures of outcomes, attitudes or opinions
- XX measures of process/product increase/decrease by YY amount
A simple basic format for this problem-solving vocabulary reflects the notion common in a systems approach to instructional design that any instructional strategy can be represented as an objective-seeking system (Gagne, Briggs & Wager, 1992). For purposes of this discussion, the concept of an objective-based instructional system will be reduced to a simple, single goal-seeking equation that is scaleable and can be represented as a statement. The general syntax of the statement is that one or more objects are modified by attributes, and combined with an activity in order to eventually accomplish an instructional goal according to a skill improvement criteria for a desired outcome. An example of using this type of statement to represent one part of an instructional strategy is:
LEARNER (object) uses COGNITIVE TOOL XX (attribute) to perform REMEMBERING (activity) until FACT IS KNOWN (goal) as measured by OUTCOMES IMPROVE BY XX (criteria)
The statement can also be represented as an equation in which the object interacts with an attribute, and the goal interacts with the criteria:
LEARNER [uses COGNITIVE TOOL XX] + REMEMBERING = FACT KNOWN [measured by OUTCOMES IMPROVE BY XX]
The equality condition of the equation represents the final state of the instructional strategy. The equality goal is not reached until the objective is accomplished according to the completion criteria. In other words, each instructional strategy statement represents an independent problem-solving process in which an activity is specified that should accomplish an objective according to a completion criteria. If the process were formed as a generic equation, one possible representation could be the following:
OBJECTS[use ATTRIBUTE] + ACTIVITY = GOAL[measured by CRITERIA]
Or, for a more readable general statement:
OBJECTS use ATTRIBUTE to perform ACTIVITY until the GOAL is achieved as measured by the CRITERIA
This is a simple example, but it should illustrate the general concept of using this instructional vocabulary to represent the parts of an instructional strategy. In order to meet the needs for a workable representation of a complex strategy using this ontology in a real application, the variables will be able to embed or link to other objects, or statements. These additional objects will be additional representations of activities, and will continue to use the vocabulary. Through this type of multi-layered approach, the vocabulary will be used to implement the instructional knowledge base.
5. Interoperable Distributed Learning Environment Architecture
The generic instructional vocabulary for MAESTRO is a
work-in-progress that will be implemented, tested, and adapted as part of an interoperable
DLE architecture to be used with multiple future tutors, starting with a new version of MAESTRO.
The new architecture includes a controlling inference engine that will control tutoring
events, utilizing inputs from both the user interfaces, multiple tutoring knowledge bases,
and various instructional resources. The inference engine will also update the knowledge
bases as needed, particularly a student knowledge base, or student model. This
architecture will include a high speed network for the interface between the knowledge
bases and the inference engine. The new architecture will also include an Internet
connection and will use JAVA applets to coordinate communication and control connected
devices, including remote COTS software, embedded systems, and browser interfaces for
users. The interoperability definitions will work within standardized frameworks, using
HTML, KQML (for standardized access to the knowledge bases) and possibly KIF (knowledge
interchange format) for communication and interaction between the components of the system
(see Fig. 2). The knowledge bases will be used for all instructional decision-making in
the DLE environment, including regulation of field-based training devices (VR headsets,
Global Positioning Systems/GPS, JAVA-capable appliances, data display devices, video
conferencing systems, etc.), COTS tools (word processors, decision support systems, etc.),
and instructional resources (simulations, dialog environments, interactive JAVA
workspaces, WWW-based HTML resources, etc.). The instructional resources will be
integrated into the system and controlled by the tutoring engine, using a common API and
providing JAVA applets to the user for interaction where necessary. The tutor's
'personality' will be the only component that is tutor-specific, residing in a stand-alone
database and communicating with the tutoring engine through a standardized protocol. All
other components of the tutor will be reusable, interoperable, and hopefully
generalizeable across multiple tutors and device components, allowing for the support of a
wide range of distributed tutoring environments, from stand-alone desktop systems to
remote, field-based collaborative team training using personal data displays.
Figure 2: Ideal Interoperable DLE Architecture
The goal of research into vocabularies for interoperable DLEs is to develop theory,
engineer, construct, and evaluate the entire architecture. However, the current study is
focused only on the development and testing of only the subset of the architecture
required to adapt the MAESTRO tutor into an interoperable DLE format and test a
generic instructional vocabulary. Further research will address the use of the vocabulary
to construct domain-independent rules for use to construct instructional knowledge bases
in a more complete version of the interoperable DLE architecture.
6. Future Interoperable Tutor Components
Ongoing research into the implementation of interoperable DLEs will help address future USAF needs for interoperable tutoring components for use in advanced technology training environments such as embedded training/tutoring systems, distributed team tutoring environments, and other automated training systems. For example, training for military team tasks often requires realistic, field-based conditions. Adaptive support for these types of training needs requires a system in which the intelligent tutor or coaching system would reside on location, and be embedded in field equipment, or in remote field-based based data devices. Recent developments in several new technologies suggest new capabilities for the use of advanced training technologies to support these types of military training needs. This includes the rapid decline in the cost of 'wearable' VR-type display devices, remote positioning systems, and wireless modem technologies. New 'wearable' VR-type hardware is becoming affordable, with users being free to roam while they use the equipment. Satellite-based global (GPS) and ground-based local area positioning systems are becoming affordable, and in the case of local positioning systems, able to record position to within less than a centimeter. These devices will soon be able to be worn like watches or used as measuring devices. The advent of the 'wireless' age includes the expansion of digital cellular modems that provide Internet-based links to any cellular location. In combination, these new technologies can provide a platform for adaptive software that supports a new range of field-based virtual training environments in position-dependent open locations such as large buildings or facilities, on simulated maneuvers, during simulations of dangerous field-based activities, as well as in many traditional training and performance environments.
As part of the need to support field-based team training, and other possible applications for interoperable DLEs in future Air Force training, rapid repurposing and reuse of the components of intelligent tutoring systems, such as the MAESTRO tutor, will play an increasingly important role. In order to successfully support these types of training environments, it is important to develop an environment in which full component-level interoperability is available, allowing multiple components to be combined in a 'plug-n-play' fashion. Testing generic vocabularies with heterogeneous knowledge bases and inference engines, and developing an architecture for interoperable approaches to the integration of instructional resources will be critical steps in inventing the new technologies that can successfully support Air Force advanced training technology requirements in the future.
REFERENCES
Bereiter, C., and Scardemalia, M. (1987). The psychology of written composition. Hillsdale, NJ: Lawrence Erlbaum Associates. Carlson, P. and Miller, T. (1996). Beyond word processing: Using an interactive learning environment to teach writing. Technical Paper of the Human Resources Directorate, Technical Training Division, Brooks AFB, Texas.
Flower, L., and Hayes, J. R. (1980). The dynamics of composing: Making plans and juggling constraints. In L. Gregg, & E. Steinberg (Eds.), Cognitive processes in writing (31-50). Hillsdale, NJ: Lawrence Erlbaum Associates.
Gagne, R., Briggs, L., and Wager, W. (1992). Principles of Instructional Design. Fort Worth: Harcourt Brace Jovanovich College Publishers.
Jona, Kemi and Korcuska, Michael. (1996). Same architecture new domain (SAND): An alternative methodology for building high-quality reusable tutoring systems. ITS'96 Workshop on Architectures and Methods for Designing Cost-Effective and Reusable ITSs. http://advlearn.lrdc.pitt.edu/its-arch/papers/
Kwok-Keung, Yum, and Crawford, John. (1996). On the feasibility of an interoperable tutorial machine to support the development and delivery of teaching. ITS'96 Workshop on Architectures and Methods for Designing Cost-Effective and Reusable ITSs. http://advlearn.lrdc.pitt.edu/its-arch/papers/
Lynch, Clifford. (1993). Interoperability: The standards challenge for the 1990s. Wilson Library Bulletin, March 1993, 38-42.
Mizoguchi, R., Sinitsa, K, and Ikeda, Mitsura. (1996). Task ontology design for intelligent educational/training systems. ITS'96 Workshop on Architectures and Methods for Designing Cost-Effective and Reusable ITSs. http://advlearn.lrdc.pitt.edu/its-arch/papers/
Molka, Judith. (1992). Surrounded by standards, therre is a simpler view. Journal of the American Society for Information Science, 43(8), 526-530.
Rowley, K. (in review). An ITS for the classroom: The field-informed design of MAESTRO the writing process tutor. Technical Paper of the USAF Armstrong Laboratory Human Resources Directorate, Technical Training Division, Brooks AFB, Texas.
Rowley, Kurt. (1996). Increasing the effectiveness of ITS research and development. ITS'96 Workshop on Architectures and Methods for Designing Cost-Effective and Reusable ITSs. http://advlearn.lrdc.pitt.edu/its-arch/papers/
Rowley, Kurt. (1995). Understanding software interoperability in a technology-supported system of education. Cause/Effect, 18(3), 20-26.
Rowley, K. and Crevoisier, M. (1997). MAESTRO: Guiding students to skillful performance of the writing process. Proceedings of ED-Media 97, Calgary, June, 1997.
Rowley, K., and Miller, T. (in press). Informing the design of a supportive writing environment: Assessing student performance by instructional styles, and by modes of student control. Technical Paper of the USAF Armstrong Laboratory Human Resources Directorate, Technical Training Division, Brooks AFB, Texas.
Zellermayer, M., Salomon, G., Globerson, T., & Givon, H. (1991). Enhancing writing-related metacognitions through a computerized writing partner. American educational research journal, 28(2), pp. 373-391.