How can we promote learning by changing students' behaviors through design?
CRIr. a mobile app designed in a one-semester class at Carnegie Mellon University meant to help middle students better to aggregate and showcase all digital evidence of what they have created, achieved and mastered. It is a group project (3 people).
This project is in response to Maker Ed’s Open Portfolio Challenge, which aims to develop a common set of practices for portfolio creation, reflection, sharing, assessment, and technology solutions to create an open, decentralized, and distributed lifetime portfolio system for makers. A digital portfolio implies a portable, modular form that can be shared. The target age the platform serves is youth between 13-18 years of age.
You can view/download the final presentation deck here.
What is the ecosystem of portfolio making?
Because this challenge is in a very complicated space with a lot at stake which is also known as a “wicked problem”, we started with some background research to narrow down our focus. We consolidated our research findings into a stakeholder map and a sequence model.
1. 1 Stakeholder Map
We created a stakeholder map based on the information we had collected. It helps us to decide on who to interview and what to ask.
1.2 Sequence Model
Our team decided to create a sequence model to visualize the motivations and tools that fuel the portfolio making process. It helps us to narrow down on which area to focus on.
Middle school students look for fun elements to share with friends
We created four personas as composite archetypes from our user-derived data. We thought that personas should give designers different points of view from motivations to goals to consider in the design process, so we mapped out these personas in a Venn diagram format.
Middle school students, who are our primary user group, require an introduction to documenting their progress and the motivation to document. Self-motivation is rare amongst these users, so motivation is an important aspect to consider for them. So we looked at “fun” and interactive elements that can be brought into our product to engage students. They are also greatly influenced by peers, so we wanted to explore the social aspect of the solution.
The lack of documentation habit or unguided documentation prevent students from realizing the full value of portfolio making.
From the initial user research, we discovered that digital portfolios can help students concretely show growth, help them receive feedback from teachers and peers, and also help them reflect on their learnings. However, the role of portfolios for students now are currently a little more than a collection of projects throughout the years. Projects are not gotten organized so that students are not able to use portfolio to promote learning.
Moreover, we gravitated toward exploring how peers and teachers can support students' learning in the process of students documentating their works. We found that self-reflection and critiques were the two key interactions that had the design potential.
“Critiques help novices to understand key principles in a domain, to compare alternatives, to articulate the goals behind their work, and to recognize how others perceive their work. "
What does the preferred future look like?
We built some visioning gameboards to look at how our product might be able to help students in different scenarios. From our gameboard sessions, we were able to create a consolidated user journey that we wanted students using our product to take.
We envisioned the solution to be an informal place for students to regularly document their progress, share with friends and learn from each other.
Iterate fast and release often
For our initial prototype, we interviewed middle school students from Pittsburgh Center for the Arts who were in a digital media class and showed them paper models of what we would like to happen throughout their interaction with the product. We also demonstrated the paper prototype with the teachers of the children, to get more feedback from a teacher’s perspective.
We found that the youngest set of middle schoolers (5th-6th grade) had a hard time concentrating on our explanations and the demonstration of the paper prototype. With the level of feedback provided, we had to assume that the attention was simply to be respectful since there was very little constructive feedback given. “That was good,” seemed to be the overall response they would give us.
We then developed a digital prototype with guiding questions for the interviews that give the user more choice in the matter.
We found that teachers wanted to add specific questions for the class and would like to be more active in the process. So we decided to allow teachers to be able to submit “recommended questions” for their students to answer in their beginning and ending the interview.
CRITr. is a mobile solution for middle students to reflect on their projects, provide/receive feedbacks, and ultimately foster a documentation habit.
We arrived at the solution after conducting research on the role of portfolio. Rather than focusing on creating the final portfolio piece, we aimed to designing for the process of portfolio-making. We want to transform the role of project documentation to a tool for developing important learning processes and achieving growth mindset.
Additionally, students in this age range are often less experienced in creating portfolios or other documentation artifacts. Although project-based learning is becoming more prevalent in certain specific educational contexts (Barron & Darling-Hammond 2008), much of the classwork that students engage in prior to this point in their educational careers is based more on evaluation than creation. As such, our system needs to not only facilitate documentation practices, but also scaffold them in a way that helps students learn how to assemble meaningful portfolios.
CRITr. helps users self-reflect on their learning, as well as give/receive critical feedbacks to/from peers. These actions are all under appropriate scaffolding and guidance, so the app carries out the educational meaning.
A Two Part Solution
The system we designed consists of two components that work in concert to deliver the learning experience we imagined. One part is for students to document their works and provide/receive feedbacks. To scaffold the domain learning, we want to encourage teachers’ involvement in students’ documentation by allowing them to customize reflection questions and critique guidance.
1. Improve Your Self-Reflection Skills
At the begining of the project, students are asked to set their learning goals.
2. Document Your Project In Minutes
Students can easily to take photo of their works, and upload as "moments" to their personal page.
3. Promote Learning Through Peer Critiques
4. Track Your Learning and Growth
CRIr. curates your accomplishments in a portfolio makes your growth more visible. Past challenges and successes become clear, leading to faster learning and mastery.
CRITr. provides values to both teaches and students
1 ) CRITr. provides a holistic view of students’ projects. Students have one place to go back to reflect on previous projects by viewing their past interviews and critiques. It also provides a safe space for students to shape their understanding of domain project subjects.
2 ) CRITr. positions portfolio making and documentation as a tool for developing important learning processes. This includes developing articulation in meaningful reflection and critiques.
What Went Well?
1 ) View portfolio making through a unique perspective
Through our research, we gathered the insight that many evaluators agreed that portfolio making helps students grow in their learning. We decided to look at portfolio making through this perspective of growth. We wanted focus on how students could develop a growth mindset through building a documentation habit which builds student reflection and feedback skills. Looking at portfolio making through this lens helped us create a unique tool for students.
2 ) Each study we conducted helped us narrow our focus
From our expert user session to our final prototyping session, all of our sessions gave us more information to make changes for our final product. We listened to many of the student and teacher user perspectives to create a product that addresses their needs.
3 ) Received well by audience
At our final presentation, audience members liked and wanted us to expand on our ideas. Many of the audience members reaffirmed the findings from our research studies. For example, during one of our prototyping sessions, a teacher brought up the value of critiquing which led us to our informal critique feature in our app. Audience members loved this idea and thought that it was a great addition to the portfolio making experience.
What Needs More Work?
Due to time constraints and resources, there are a couple of areas that we would have liked to explore more but couldn’t. We’re going to talk through the specific parts of these areas that are interesting to us.
1 ) Competitive analysis of current portfolio or documentation tools
This is something that would have been useful early on in our research. We did gather some information about current tools like Schoology and Blackboard through our user interviews, but it would have also been valuable to go more in depth. We could evaluated some lesser used features that could have been related to our research focus points.
2 ) Delve deeper into teacher’s role and involvement
We realized how vital teacher’s roles could be to scaffold the important domain-specific reflections a little too late in the process. It would help further our understandings if we could go back and do a user study on teachers to understand their motivations and goals through current tools and what they hope to be able to achieve in the future.
3 ) How we can effectively add incentives?
We did a little bit of user testing around goals but felt that the way we approached it wasn’t the right way. The feedback that we got was that students already make personal goals. We need to take some more time to reframe goal making in a way that motivates and reminds students to use our tool.
4 ) Connect back to portfolios
Currently, our tool focuses on the capturing and analysis aspect of portfolio making. Students can certainly use our tool to aid them in creating a cohesive portfolio piece, but it doesn’t directly lead students to make that extra step. With more time, we can figure out this last piece of the process to bring our tool into each part of the portfolio making loop.
5 ) Consider expanding to non-multimedia focused project subjects
Because our prototyping sessions were all done at the Pittsburgh Center of Arts, our prototypes ended up geared toward that group of students. Because of this we were able to take advantage of some of the aspects of their projects that could benefit from multimedia fueled critiques. However there are many other project subjects that can’t really use multimedia for critiques so we would like to explore how we can incorporate those subjects into our tool.