This card describes a paper presented at the 2019 ASEE Conference.

Are you teaching a first year engineering course, or the engineering design process? Are you looking for an assessment tool for the design process? Do you want to implement 'connections' out of the 3C's in your course?

This card describes the use of concept maps as a tool to assess first year students' understanding of the engineering design process in a design-based introduction to engineering course at Arizona State University. It requires students to identify and make connections of various design process related topics and demonstrate their comprehensive understanding of the design process. Even though this card describes the use of the tool in a specific context, i.e., the assessment of engineering design process knowledge, its use can be extended to help students make connections of any other topics within a course, or across curricula, either as an instructional tool or assessment tool. 

Background Information 

The engineering design process has become one of the essential topics for first year engineering courses. Many such courses now incorporate design activities for students to practice applying the design process in either simulated or real world situations. The assessment of design process knowledge is usually done through the evaluation of design deliverables, or close or open-ended questions. Traditional assessment methods exhibit many flaws, for example, some of them may not be individual or process based, and others are only connected to lower levels of Bloom's Taxonomy. 

Concept maps, i.e., graphical node-arc representations that depict relationships among concepts, on the other hand, used as an assessment tool, is open-ended and it requires students to internalize the knowledge, identify key concepts that are relevant, and document relationships between the concepts, demonstrating knowledge of the engineering design process at multiple levels of Bloom’s Taxonomy. 

Context 

This tool was utilized in the freshman-level 2-credit 15-week introduction to engineering course taught during the Fall 2019 semester at Arizona State University, three times throughout the semester to study changes in students' understanding of the design process. Each time, students were instructed to create a concept map that demonstrates their understanding of the design process. More specifically, students must identify key phases (steps) of the design process, as well as any relevant information and details related to each phase, including goals, key tasks, strategies and tools involved, and possible outcomes. And they must use a concept map to show connections of different (specific) concepts identified.

The first assessment was done at the beginning of the first class period and it was not graded. The goal of the first assessment was to learn about their understanding of the design process before engaging in any course activities. The other two were each done during a 50-min lecture as in-class assessments and both were included in their final course grades. The second one was done during the middle of the semester after the introduction of the design process and completion of a two-week team-based design challenge where students applied the first few phases of the design process to create a conceptual design. While the last assessment was done at the end of the semester after a 10-week large hands-on team-based multidisciplinary systems-design project.  

In the first folder included below, a description of the course, including its format, outcomes, and a list of weekly topics can be found. In that folder, a link to another card describing the two-week design challenge, and a link to another ASEE paper describing the 10-week design project can also be found.
In the second folder below. detailed instructions provided to students for the assessment, along with the grading rubrics can also be found. 

Outcome  

Through the analyses of the concept maps generated by students, it was learned that there is no difference in students’ understanding at the start of this course regardless of whether they had prior knowledge and experiences about engineering design or not; and through this course, the two-week design challenge in particular, students’ understanding of the design process in all aspects has greatly improved; and students’ understanding was further improved after the ten-week design challenge in areas of ‘customer involvement throughout the design process’, ‘research/information gathering’, ‘model/analysis’, and the ‘iterative characteristic’ of the design process.

The concept maps generated by students were also categorized based on their complexity and the inter-connectedness of concepts included. It was learned that at the end of the course, 83% students were able to successfully make connections of various engineering design process concepts introduced during the semester.  

In the third folder included below, four examples of concept maps generated by students can be found. 

Link to EM

Connections 

Throughout the semester, many specific concepts and tools are introduced that are related to the engineering design process. Students should not leave the course with many small pieces of information, without having a complete picture of how they relate to each other. This tool provides an excellent opportunity for students to think deeply about how the specific concepts are related to each other, through the construction of a complete picture around a central topic.

For example, 'design criteria' is one of the concepts introduced in the course about the design process. Instead of just knowing that 'design criteria need to be identified' in the design process, this tool requires students to think about what specific goals, tasks, and outcomes in the design process are related to 'design criteria', and when does the 'design criteria' matter. For example, it can be connected to the 'define the problem', 'brainstorm', 'evaluate design options', 'test design' phases in the design process as well as other specific topics including 'customer wants', 'AHP (Analytic Hierarchy Process)', 'decision matrix', 'test procedure', etc. There is also an important connection between 'design criteria' and 'customer'.

Through this example, it can been seen that this tool helps facilitate the development of connected thinking. 

ASEE Paper 

The paper that is the focus of this card was presented in the Best of First Year Programs: Best Paper Session at the 2019 ASEE Annual Conference. The paper can be viewed at: https://www.asee.org/public/conferences/140/papers/27035/view. It is also included as a link under the folders section of this card.

Materials Included 

In the folder, the following documents are included:
- course information including weekly topics 
- slide that shows the engineering design process introduced in the course 
- link to the card describing the team-based design challenge
- link to the 2018 ASEE paper that describes the team-based multidisciplinary systems design project
- assessment instructions
- assessment rubrics  
- example student work 

Student Outcomes: 

• Identify specific design process related concepts, such as phases included in the design process, as well as goals, specific tasks, tools & strategies, and possible outcomes related to each phase 
• Correlate all the specific concepts listed above  
• Organize the concepts by constructing a diagram that represents their correlations