Graph and Property Set Analysis

The purpose of this dissertation study was to conduct the next stage of research in the development of a new methodology (Smith, 2005) based on an analysis of Graphs and Property Sets (GAPS). The objective of the methodology is to measure the degree of similarity in structure and content of mental model representations. Such measures are useful in determining if and to what extent instructional interventions promote understanding and the acquisition of expertise with regard to complex phenomena and problem solving situations. This methodology builds on earlier research (e.g., Spector & Koszlka, 2004) and was tested in a prototype study (Smith, 2006). The research was developmental in nature and consisted of a formative evaluation of the methodology aimed at answering the following questions: 1. Does the methodology provide useful comparisons of student-constructed models based on relevant attributes of structure and content that are embedded in the model elicitation methodology? 2. What improvements in the methodology are needed prior to further research and development and eventual implementation in the form of a mental model assessment tool? a. What improvements are needed regarding the mental model elicitation methodology? b. What improvements are needed in the mental model representation analysis methodology? The study revealed that the methodology can provide useful comparisons of student-constructed models. The determination of usefulness was based on the feedback received from two professors who are instructors for beginning students in the instructional design program which provided the subjects for this research. The study also identified specific improvements that are needed prior to further research and development of the methodology. For this study, a mental model is an internal cognitive structure created by an individual to explain external phenomena, to solve problems, and/or to predict outcomes of actions and decisions. Such internal structures cannot be observed directly, and methods for representing an individual's mental model vary according to the latitude of expression given the individual and the extent of assumptions that must be made concerning the degree of similarity between the internal model and the external representation. The methodology evaluated in this study represents a systematic attempt to combine freedom of expression on one hand with structured detail elicitation on the other. The intention is to reduce the number of inferences and assumptions an investigator must make in interpreting a mental model representation and address finer levels of comparisons between and among models. The methodology uses an application of graph theory (Chartrand, 1977; Diestel, 2000) and can be distinguished from other graph-based methodologies by one or more of the following characteristics. Subjects create their own graphs to represent their mental models. Subjects provide detailed property sets for each graphic element. Property sets define both the concepts in a subject's mental model and the subject's understanding of how concepts are related. Finally, comparisons between models are based on analyses of properties of graphic elements rather than linked pairs of concept labels. Property set analysis may determine whether or not similar labels in different mental model representations refer to the same concepts. It also may determine whether or not similar concepts are identified with different labels. Assumptions that subjects understand and use concept labels the same way can lead to inaccurate conclusions about the degree of correspondence of one model to another. The research context was a graduate program in instructional design at a large, Southern university. Individuals may enter the program as masters students or doctoral students. The focus of this study was limited to comparisons of mental model representations between novices and experts in the field of instructional design. The methodology was used to examine gaps between the knowledge and conceptions of beginning students and the knowledge and conceptions of their professor who is an experienced practitioner in the field of instructional design. The initial state of student knowledge and conceptions can have significant implications for the design and delivery of instruction. First, understanding students' prior knowledge provides a starting point in bridging the gap between their beginning state and the learning objectives of the instruction. Second, learning of new material takes place with regard to a larger world view students may have. Integration of new knowledge within this larger context requires some awareness of the context's relevant attributes. Next, examination of students' initial conceptions and mental models may reveal misconceptions that must be overcome in order for the learning objectives to be achieved. Misconceptions can be firmly entrenched, and may require design and/or delivery approaches beyond those sufficient to instruct students without such handicaps. It is assumed that a comparison of mental model representations of beginning students with the mental model representation of an experienced practitioner will reveal both initial states of the learners and misconceptions they may have. Participants included three professors who are experienced instructional designers and 19 graduate students in an introductory design course in the Instructional Systems Program. Participants were trained to use the methodology to represent their mental models in responding to an instructional design problem. Mental model representations of students were compared with that of the professor teaching the introductory instructional design course. The comparisons addressed: (a) the degree of similarity in content and structure; and (b) specific areas in student models which might indicate misconceptions or knowledge gaps. The mental model representations of the other two experienced instructional designers were compared to that of the professor teaching the course. This analysis determined that the methodology has utility in comparing the models of persons with similar expertise (experienced designers) as well as those with different levels of expertise (professor/student). It also confirmed that the methodology identifies more similarities between persons with similar expertise than between persons of different levels of expertise. Answer to research question 1. A comparison of student-constructed models based on relevant attributes of structure and content is considered useful if it reveals misconceptions or gaps in knowledge that, if present, will affect the design and/or delivery of instruction for the purpose of improving the potential for learners to achieve the targeted learning goals. The comparison analysis results were shared first with the professor teaching the class of student participants to determine the usefulness of the methodology in identifying misconceptions or knowledge gaps that can affect instructor decisions concerning the design and/or delivery of instruction. Next, the results were shared with the other two professors, one of whom also was an instructor for beginning students in instructional design. The third professor, who did not teach instructional design students, did not comment on the specific application of results; however, the two professors teaching in the instructional design program responded that the information would aid them in making course design and delivery decisions. They indicated surprise regarding (a) the amount of information that the methodology could produce and (b) the extent of the gaps in knowledge that were revealed between entry level students and their professor. Answer to research question 2. Required improvements in the methodology were addressed using qualitative data obtained from analysis of mental model representations and participant responses to questionnaires and interviews. Questionnaires and interviews were used to obtain participant feedback on the representation process. Comparative analysis data and data from the questionnaires and interviews were examined to determine what improvements are needed prior to further study and implementation of the methodology. The initial analysis results and recommended list of changes were shared with the experienced practitioner group (i.e., the professors) to obtain their reactions to the proposed improvements. The list of recommendations include: (a) an improved training plan with more examples and additional practice; (b) assessment of understanding of both the representation process and the problem statement prior to model elicitation; (c) better design of the model to be used for comparison with student models; and (d) a set of guidelines for constructing the database and performing qualitative analyses. Because this study was limited to a single application and participant group, results cannot be generalized. However, the mental model assessment methodology design is not limited to this specific application. The results of this study can set the stage for future research using other subject areas, different educational levels, and additional populations. The intended features of this methodology are that it will: (a) be generalizable across domains and populations; (b) be applicable for a variety of purposes in education and training including educational research and instructional design; (c) be scalable for practical use in secondary, tertiary and work settings; (d) be appropriate for complex problem solving domains; (e); produce metrics that identify the degree and basis of correspondence between mental models; and (f) provide greater insight into the structure and content of a person's mental model than what is now provided by current mental model assessment approaches. Further research may produce a validation for broader applications and eventual implementation in the form of a mental model elicitation and assessment tool.