How can we promote learning by changing students' behaviors through design? 
CRIr. a mobile app designed in a one-semester class at Carnegie Mellon University meant to help middle students better to aggregate and showcase all digital evidence of what they have created, achieved and mastered.  It is a group project (3 people). 

This project is in response to Maker Ed’s Open Portfolio Challenge, which aims to develop a common set of practices for portfolio creation, reflection, sharing, assessment, and technology solutions to create an open, decentralized, and distributed lifetime portfolio system for makers.  A digital portfolio implies a portable, modular form that can be shared. The target age the platform serves is youth between 13-18 years of age. 

Design Focus 
+ A top-down approach to designing the product and the experience
+ Reframed the role of portfolio-making to building a growth mindset 


The lack of documentation habit or unguided documentation prevent students from realizing the full value of portfolio making.

Digital portfolios can help students concretely show growth, help them receive feedback from teachers and peers, and also help them reflect on their learnings. However, the role of portfolios for students now are currently a little more than a collection of projects throughout the years. Projects are not gotten organized so that students are not able to use portfolio to promote learning. At the same time, many educators see the great value of portfolio making and see the final product as an important way to reflect students’ growth. The lack of documentation habit or unguided documentation prevent students from realizing the full value of portfolio making.

Introducing CRITr.
CRITr. is a mobile solution for middle students to reflect on their projects, provide/receive feedbacks, and ultimately foster a documentation habit.

We arrived at the solution after conducting research on the role of portfolio. Rather than focusing on creating the final portfolio piece, we aimed to designing for the process of portfolio-making. We want to transform the role of project documentation to a tool for developing important learning processes and achieving growth mindset.
Additionally, students in this age range are often less experienced in creating portfolios or other documentation artifacts. Although project-based learning is becoming more prevalent in certain specific educational contexts (Barron & Darling-Hammond 2008), much of the classwork that students engage in prior to this point in their educational careers is based more on evaluation than creation. As such, our system needs to not only facilitate documentation practices, but also scaffold them in a way that helps students learn how to assemble meaningful portfolios.
CRITr. helps users self-reflect on their learning, as well as give/receive critical feedbacks to/from peers. These actions are all under appropriate scaffolding and guidance, so the app carries out the educational meaning. 

A Two Part Solution 
The system we designed consists of two components that work in concert to deliver the learning experience we imagined. One part is for students to document their works and provide/receive feedbacks. To scaffold the domain learning, we want to encourage teachers’ involvement in students’ documentation by allowing them to customize reflection questions and critique guidance.

Students' Side 

1. Improve Your Self-Reflection Skills
 At the begining of the project, students are asked to set their learning goals.

2. Document Your Project In Minutes
Students can easily to take photo of their works, and upload as "moments" to their personal page.

3. Promote Learning Through Peer Critiques 

4. Track Your Learning and Growth
CRIr. curates your accomplishments in a portfolio makes your growth more visible. Past challenges and successes become clear, leading to faster learning and mastery.

Teachers' Side 

CRITr. provides values to both teaches and students 
1 ) CRITr. provides a holistic view of students’ projects. Students have one place to go back to reflect on previous projects by viewing their past interviews and critiques. It also provides a safe space for students to shape their understanding of domain project subjects. 
2 ) CRITr. positions portfolio making and documentation as a tool for developing important learning processes. This includes developing articulation in meaningful reflection and critiques. 


background research - discovery - product envisioning - prototyping

What's the role of portfolio?

Because this challenge is in a very complicated space with a lot at stake which is also known as a “wicked problem”, we started with some background research (literature review and expert consultation) to narrow down our focus. 

1. 1 Stakeholder Map 
We created a stakeholder map based on the information we had collected. Because we believe that the ultimate goal of this product is to help students succeed, we took a student-centered approach in visualizing this concept map. The table in the lower left details the key interactions between students and each stakeholder group. In the main section, the stakeholders are mapped out based on the proximity to students in different stages of students' learning journey

1.2 Sequence Model  
Our team decided to create a sequence model to express motivations and tools that fuel the portfolio making process.  The model describes the anatomy of current student portfolios from brought categories down to the gritty structural parts of each digital portfolio. 

From the initial background research, we discovered that one attribute of portfolio is documentation, which opens possibilities for detailing students' work process that final pieces can't show. We also found that peers and teachers have the greatest influence on shaping middle school students' learning behaviors. 
Thus, we gravitated toward exploring how peers and teachers can support students' learning in the process of students documentating their works. We found that self-reflection and critiques were the two key interactions that had the design potential. 

“Critiques help novices to understand key principles in a domain, to compare alternatives, to articulate the goals behind their work, and to recognize how others perceive their work. "    
 --- "Structuring, Aggregating, and Evaluating Crowdsourced Design Critique" 

To understand users' behaviors and needs

The discovery phase was a quick, high‐intensity effort that allowed us to  begin research into user needs, behaviours and pain‐points, and define a desirable role for the app. 
We conducted two kinds of interviews. At first, we interviewed portfolio-making experts to understand their interpretation and process of portfolio-making. In this way, we would know what are the final effects we would like to achieve with our product. Then, we conducted guerrilla interviews with target users to understand their needs and behaviors. 

Analyzing interview data by using affinity diagram

In About Face: The Essentials of Interaction Design, the reading mentions that personas should give designers different points of view from motivations to goals to consider in the design process. We created four personas as composite archetypes from our user-derived data. Because we felt like our personas were too complex to be mapped out on a 2D axis, we decided to represent our personas in a Venn diagram format.

Middle school students, who are our primary user group, require an introduction to documenting their progress and the motivation to document. They are not necessarily going to see documentation as a way to get into college and have rarely had a reason to document their process outside of some school assignments. 
Self-motivation is rare amongst these users, so motivation is an important aspect to consider for them. Taking Stephanie Chang’s article “Chapter 6: Platforms for Recording, Storing, and Sharing”, into consideration, we looked at “fun” and interactive elements that can be brought into our product to engage students. 
We thought this platform could be an informal place for students to regularly document their progress, share with friends and show student progress which in turn fosters a growth mindset.
This platform can be an informal place for students to regularly document their progress, share with friends and show student progress which in turn fosters a growth mindset.

How did we get here? 

We built some visioning gameboards to look at how our product might be able to help students in different scenarios. 
Scenario One: 
Kim gets a notification in her Facebook news feed about one of her friends raising a virtual pet by doing the equivalent of updating her status, by updating her progress on a school project. She is curious and decides to try out the system(our project).
Scenario Two: 
Jane sees that Amanda likes her project on her phone. She checks the newsfeed for others’ projects and comments on other’s work.
Scenario Three: 
Mr. O’Neil teaches an art course and uses the (project) to let his students have a gallery. The gallery treats each student’s personal gallery like a storybook with each project as a chapter.

Journey Map  
From our gameboard sessions, we were able to create a consolidated user journey that we wanted students using our product to take.

How did we test our ideas? 

Round 1
For our initial prototype, we interviewed middle school students from Pittsburgh Center for the Arts who were in a digital media class and showed them paper models of what we would like to happen throughout their interaction with the product. We also demonstrated the paper prototype with the teachers of the children, to get more feedback from a teacher’s perspective.
We found that the youngest set of middle schoolers (5th-6th grade) had a hard time concentrating on our explanations and the demonstration of the paper prototype, but the older middle schoolers (7th-8th grade) displayed more interest in the prototype. With the level of feedback provided, we had to assume that the attention was simply to be respectful since there was very little constructive feedback given. “That was good,” seemed to be the overall response they would give us.
"Umm…. I think they are all good."
When asked to set a goal, students were either not sure how to respond, claimed they already had goals set but did/could not articulate them, or did not seem very interested in the idea. We also developed to prototype with the intent of letting students import and export photos to sites like Instagram, but we found that students either did not have a smartphone, did not use social media, or both. They did not stay very organized and rarely ever save the process of their work. They don’t feel the need to look back on their previous work, but would still like to improve their skills.

Round 2
We then developed a digital prototype with guiding questions for the interviews that give the user more choice in the matter.
We also developed a more robust peer critiquing model that allows students to comment simply as well as receive guiding questions.
Students reported their comfort and discomfort with the decisions we made in the design. Students are not always comfortable when they have to videotape their answers, and students prefer comments without guiding questions.
On the other hand, teachers want to add specific questions for the class and would like to be more active in the process.
For the interviews, we decided to allow teachers to be able to submit “recommended questions” for their students to answer in their beginning and ending the interview. It was suggested that we remove the mandatory guiding questions for the critique since the students felt they were capable. But there will always be someone who needs the support, so scaffolding may be necessary for some students.

We prepared a set of slides and video demo that we walked through in our class presentation.


What Went Well?
1 ) View portfolio making through a unique perspective
Through our research, we gathered the insight that many evaluators agreed that portfolio making helps students grow in their learning. We decided to look at portfolio making through this perspective of growth. We wanted focus on how students could develop a growth mindset through building a documentation habit which builds student reflection and feedback skills. Looking at portfolio making through this lens helped us create a unique tool for students.
2 ) Each study we conducted helped us narrow our focus
From our expert user session to our final prototyping session, all of our sessions gave us more information to make changes for our final product. We listened to many of the student and teacher user perspectives to create a product that addresses their needs.
3 ) Received well by audience
At our final presentation, audience members liked and wanted us to expand on our ideas. Many of the audience members reaffirmed the findings from our research studies. For example, during one of our prototyping sessions, a teacher brought up the value of critiquing which led us to our informal critique feature in our app. Audience members loved this idea and thought that it was a great addition to the portfolio making experience.

What Needs More Work?
Due to time constraints and resources, there are a couple of areas that we would have liked to explore more but couldn’t. We’re going to talk through the specific parts of these areas that are interesting to us.
1 ) Competitive analysis of current portfolio or documentation tools
This is something that would have been useful early on in our research. We did gather some information about current tools like Schoology and Blackboard through our user interviews, but it would have also been valuable to go more in depth. We could evaluated some lesser used features that could have been related to our research focus points.
2 ) Delve deeper into teacher’s role and involvement
We realized how vital teacher’s roles could be to scaffold the important domain-specific reflections a little too late in the process. It would help further our understandings if we could go back and do a user study on teachers to understand their motivations and goals through current tools and what they hope to be able to achieve in the future.
3 ) How we can effectively add incentives?
We did a little bit of user testing around goals but felt that the way we approached it wasn’t the right way. The feedback that we got was that students already make personal goals. We need to take some more time to reframe goal making in a way that motivates and reminds students to use our tool.
4 ) Connect back to portfolios
Currently, our tool focuses on the capturing and analysis aspect of portfolio making. Students can certainly use our tool to aid them in creating a cohesive portfolio piece, but it doesn’t directly lead students to make that extra step. With more time, we can figure out this last piece of the process to bring our tool into each part of the portfolio making loop.
5 ) Consider expanding to non-multimedia focused project subjects
Because our prototyping sessions were all done at the Pittsburgh Center of Arts, our prototypes ended up geared toward that group of students. Because of this we were able to take advantage of some of the aspects of their projects that could benefit from multimedia fueled critiques. However there are many other project subjects that can’t really use multimedia for critiques so we would like to explore how we can incorporate those subjects into our tool.
Back to Top