Concept Mapping, a Tool for Assessment of First Year Students' Understanding of the Design Process
This card describes a paper presented at the 2019 ASEE Conference.
Are you teaching a first year engineering course, or the engineering design process? Are you looking for an assessment tool for the design process? Do you want to implement 'connections' out of the 3C's in your course?
This card describes the use of concept maps as a tool to assess first year students' understanding of the engineering design process in a design-based introduction to engineering course at Arizona State University. It requires students to identify and make connections of various design process related topics and demonstrate their comprehensive understanding of the design process. Even though this card describes the use of the tool in a specific context, i.e., the assessment of engineering design process knowledge, its use can be extended to help students make connections of any other topics within a course, or across curricula, either as an instructional tool or assessment tool.
The engineering design process has become one of the essential topics for first year engineering courses. Many such courses now incorporate design activities for students to practice applying the design process in either simulated or real world situations. The assessment of design process knowledge is usually done through the evaluation of design deliverables, or close or open-ended questions. Traditional assessment methods exhibit many flaws, for example, some of them may not be individual or process based, and others are only connected to lower levels of Bloom's Taxonomy.
Concept maps, i.e., graphical node-arc representations that depict relationships among concepts, on the other hand, used as an assessment tool, is open-ended and it requires students to internalize the knowledge, identify key concepts that are relevant, and document relationships between the concepts, demonstrating knowledge of the engineering design process at multiple levels of Bloom’s Taxonomy.
This tool was utilized in the freshman-level 2-credit 15-week introduction to engineering course taught during the Fall 2019 semester at Arizona State University, three times throughout the semester to study changes in students' understanding of the design process. Each time, students were instructed to create a concept map that demonstrates their understanding of the design process. More specifically, students must identify key phases (steps) of the design process, as well as any relevant information and details related to each phase, including goals, key tasks, strategies and tools involved, and possible outcomes. And they must use a concept map to show connections of different (specific) concepts identified.
The first assessment was done at the beginning of the first class period and it was not graded. The goal of the first assessment was to learn about their understanding of the design process before engaging in any course activities. The other two were each done during a 50-min lecture as in-class assessments and both were included in their final course grades. The second one was done during the middle of the semester after the introduction of the design process and completion of a two-week team-based design challenge where students applied the first few phases of the design process to create a conceptual design. While the last assessment was done at the end of the semester after a 10-week large hands-on team-based multidisciplinary systems-design project.
In the first folder included below, a description of the course, including its format, outcomes, and a list of weekly topics can be found. In that folder, a link to another card describing the two-week design challenge, and a link to another ASEE paper describing the 10-week design project can also be found.
In the second folder below. detailed instructions provided to students for the assessment, along with the grading rubrics can also be found.
Through the analyses of the concept maps generated by students, it was learned that there is no difference in students’ understanding at the start of this course regardless of whether they had prior knowledge and experiences about engineering design or not; and through this course, the two-week design challenge in particular, students’ understanding of the design process in all aspects has greatly improved; and students’ understanding was further improved after the ten-week design challenge in areas of ‘customer involvement throughout the design process’, ‘research/information gathering’, ‘model/analysis’, and the ‘iterative characteristic’ of the design process.
The concept maps generated by students were also categorized based on their complexity and the inter-connectedness of concepts included. It was learned that at the end of the course, 83% students were able to successfully make connections of various engineering design process concepts introduced during the semester.
In the third folder included below, four examples of concept maps generated by students can be found.
Link to EM
Throughout the semester, many specific concepts and tools are introduced that are related to the engineering design process. Students should not leave the course with many small pieces of information, without having a complete picture of how they relate to each other. This tool provides an excellent opportunity for students to think deeply about how the specific concepts are related to each other, through the construction of a complete picture around a central topic.
For example, 'design criteria' is one of the concepts introduced in the course about the design process. Instead of just knowing that 'design criteria need to be identified' in the design process, this tool requires students to think about what specific goals, tasks, and outcomes in the design process are related to 'design criteria', and when does the 'design criteria' matter. For example, it can be connected to the 'define the problem', 'brainstorm', 'evaluate design options', 'test design' phases in the design process as well as other specific topics including 'customer wants', 'AHP (Analytic Hierarchy Process)', 'decision matrix', 'test procedure', etc. There is also an important connection between 'design criteria' and 'customer'.
Through this example, it can been seen that this tool helps facilitate the development of connected thinking.
The paper that is the focus of this card was presented in the Best of First Year Programs: Best Paper Session at the 2019 ASEE Annual Conference. The paper can be viewed at: https://www.asee.org/public/conferences/140/papers/27035/view. It is also included as a link under the folders section of this card.
In the folder, the following documents are included:
- course information including weekly topics
- slide that shows the engineering design process introduced in the course
- link to the card describing the team-based design challenge
- link to the 2018 ASEE paper that describes the team-based multidisciplinary systems design project
- assessment instructions
- assessment rubrics
- example student work
- Identify specific design process related concepts, such as phases included in the design process, as well as goals, specific tasks, tools & strategies, and possible outcomes related to each phase
- Correlate all the specific concepts listed above
- Organize the concepts by constructing a diagram that represents their correlations
- In the first assessment, where baseline knowledge is gathered, students may feel overwhelmed by the concept of concept mapping. Because of this, I would recommend explaining what specifically students need to do without mentioning concept mapping.
- If the purpose is not to understand changes in students' understanding that occur throughout the course, this tool can be used only once, at the end of the semester.
- Instead of using class time for this assessment, it can also be done outside of class as a homework assignment or take-home test. Though the disadvantage of using the take-home test format is that it might discourage students to get clarifications of what is expected - please see the next bullet point.
- Some students may be confused by what specifically is expected. So some clarification may be needed. For example, two out of ~80 students created a concept map about a specific design, and their concept maps included specific design considerations for that specific design.
- It is challenging to balance making the instructions somewhat vague so that answers are not given out to students, and also making it clear to students of what is expected. Through the implementation, it was found that the best way to handle this is to make the written instructions somewhat vague and provide clarifications during the assessment if needed.
- If this assessment is done in class and there is not enough time to allocate to this assessment, students could be given less time for it. The majority of the students were able to complete it within 30 minutes.
- This tool can also be used as an excellent in-class active learning tool. Students can be asked to work either individually or as groups to generate concept maps showing connections of topics introduced previously periodically throughout any course. This will not only help make lectures more engaging, but also facilitate a deeper understanding of the topics.