Anca Hamuraru, bootcamp participant: Design Intervention as a Means of Removing Blockages and Allowing the Water to Flow Again: An Alternative to Perfect Problem Solving in Anthropological Contexts
A subjective introduction.
Dear reader,
I am an engineer who has discovered anthropology 5 years ago and fell in love with it for the new perspective, understanding and sometimes acceptance towards the world around us that it brings (thanks to all my wonderful teachers for that). I next became curious to learn how this knowledge can be applied for solving concrete problems. That’s what brought me to the “Ethnography and Design Bootcamp” organised by UvA together with Corina Enache, Rosalie Post and Nathan Sokoloff.
We had an intense 2 week adventure where we went from defining our research question up to prototyping and pitching a solution. It has been a creative, but also very coherent process. We had the freedom to explore, to research, to be creative, but all of it under a very clear and well structured guidance from our lecturers. We had clear goals every two days and continuous feedback from Corina, Rosalie and Nathan. When you are a beginner, it is amazing if you have experienced people like our lecturers sharing from their concrete work experience, teaching you the do’s and don’ts, paying attention to what you do and giving guidance, helping you to better make sense of your work. And so, we learned about how to efficiently define a problem, how to do fast ethnography, how to analyse our data, how to choose a good solution for the starting problem and what to focus on when pitching to a potential stakeholder. Bonus: all of this while working in a team of people you will meet there, during the bootcamp.
As part of our formal assignments, we each had to write an individual report at the end of the bootcamp. For some of us, the lecturers suggested making it into a blog post. So, here is mine. I hope you’ll find it useful and wish you goodluck in your journey!
The report.
The institutionalised educational environment is different from other big organisations in that two of its main actors are relatively large homogeneous groups: the teaching staff and the students. And while a successful educational process is what motivates and brings them together, the collaboration is not always as efficient as the two actors desire. One of the tools aiming to monitor and help in improving this collaboration is the feedback system. For the environment we studied, a technical faculty from Bucharest, Romania, a formal process for collecting feedback from students is imposed by the university. The Quality Committee of the faculty is responsible to implement the process and analyse the results.
The feedback process was introduced less than 20 years ago in this university, so it is relatively new. Nevertheless, the fill in rate has been below 30% in the last 3 academic years. During our interviews, TAs complained that students either provide negative and non-constructive feedback or they don’t provide feedback at all. Students, on the other hand, complain on the faculty’s Facebook page that their feedback is ignored and that’s why they gave up on providing it. The faculty, in turn, says that it cannot take action because the input rate is too low. As for the teachers, they are “the power holders”, as one TA mentioned. Lacking a formal process for handling the feedback, some teachers share conclusions and improvement plans with the students, while others do nothing. According to the same TA, the faculty is aware of these contrasts, but doesn’t act out of fear to not lose teachers. All the actors are trapped in a vicious circle. Instead of triggering a more action oriented dialogue and an improved quality of the educational process, the feedback procedures amplify the tensions between students and teachers and intensify the already existing contrasts inside each of the two groups. So how can the Quality Committee improve the feedback process to help in building a high performing and qualitative educational environment, where there is mutual trust and constructive communication between teachers and students?
To answer this question we tried to understand the formal and informal characteristics of the feedback process as a communication tool between teachers and students. We interviewed students to understand when and why they use or not use the tool. We also interviewed teaching assistants and teachers to understand what role the feedback plays in their teaching activities. We analysed official quality reports from the faculty and Facebook discussions to see what is the narrative built around the feedback process. And since our final goal was to design an intervention, we looked at how the two main actors, the teachers and the students, use the feedback form, from an interface perspective.
The interviews with the TAs pointed us to a strong tension: they were very unhappy that either students don’t provide any feedback or it’s just the under-performing ones that do it, but only to criticise in a non-constructive manner. On the other hand, on the faculty’s Facebook page, students were saying that they’ve always tried to provide complete and constructive feedback, but feel demotivated by lack of reaction from the faculty. That’s why we wanted to follow this tension and interview students with opposing perspectives.
Unfortunately, the students we interviewed were in the category of those providing constructive feedback. Given the limited time, we weren’t able to reach out to those who are strongly complaining or not filling out the feedback form. So, we decided to create a survey and get some input about what motivates or demotivates them to participate in the feedback process. The answers we received were very detailed and of high quality. But there were only two people who filled out the survey. It became obvious to us that the student group is strongly polarised between those that are vocal and unhappy and those who remain silent. We would have wanted to interview students from the silent group too, but then we realised that’s not needed. We already have two issues to address. One is the lack of transparency from the teachers and the faculty, which is demotivating for all students anyway. The other one is the power imbalance that comes from the institution leaving the teachers to decide if and how they want to react to the feedback they receive. Which, again, is a demotivating factor for both teachers’ and students’ involvement in the feedback process.
From our interviews with the TAs and the teachers we discovered that lots of the tensions are of a political nature and they would not only require a lot of time to research, but they would also be challenging to address. It became clear that we cannot fully solve the tensions around the feedback process. So, we aimed at bringing an improvement to the current situation and we focused on how could the interface be improved to increase students’ engagement while also addressing some of the problems identified: the lack of transparency and the power imbalance.
I experienced the transition from data gathering to data analysis and, further on, to designing an intervention as very challenging. It was difficult to separate them and I had a tendency to analyse and think of a solution while collecting the data.
What helped though in separating and transitioning from one another was my connection with the faculty, which was also, in fact, an ethical dilemma for me. Is there any conflict of interests between my role as a researcher in this project and my role as a former student and TA of the faculty our team is analysing? I thought that since I am not working, interacting with or representing any group in that academic environment, there shouldn’t be any conflict. But then could I be biased because the faculty environment is pretty familiar to me? I decided to address this question in three ways. Firstly, by not having interviews with people I already knew. Secondly, by sharing with my colleagues insights from my experience in that environment. After all, I was a source of data too. And thirdly, by realising that I can use my knowledge from that environment to explore what was in fact a new field for me: the feedback process, which didn’t exist 15 years ago, when I left the university. On top of it, I was very curious and excited to see how I can use my anthropology and ethnography skills to better understand my old faculty’s environment.
With this curiosity, during the analysis phase, I tried detaching from my experience as an insider and checking from an outsider perspective how does all the data fit together. And then is when I had two main insights. One was about how the entire narrative and use of the feedback system makes it a criticism collection tool. The other was about how the existing polarisation inside each of the teachers and students groups is basically amplified by the feedback process.
The transition from the analysis phase to designing a solution was also challenging. The time pressure made me dismiss solutions we didn’t have time to design. But the design thinking activities during the class helped a lot. Because we were encouraged to put all our ideas on paper, no matter the time needed to implement them, and then do an overall evaluation. By keeping all spectrum of ideas, from high to low impact and from high to low needed effort, we managed to design a solution that combines what’s best from each. A solution that requires a low to medium effort and which starts addressing some of the big problems around the topic we researched, thus having a meaningful impact. That’s E-Val, our feedbacker ranking system that proposes a ranking for both teachers and students based on the quantity and quality of their contribution to the feedback process.
I will not give a detailed description of E-Val, but I would like to highlight some of its key features. Students ranking is private. Teachers ranking is private too, but the institution has access to all teachers rankings. The quality metrics are based on evaluating non standard feedbacks such as: students answering open questions, teachers sharing summary and plans after processing received feedback or teachers answering students’ followup requests. Students and teachers rate each other’s input! With one exception: in students’ answers to open questions, 80% of the quality is assessed by an AI tool and 20% by the teacher. The tool checks for the length, coherence and completeness of the feedback.
E-Val doesn’t solve all the problems around the feedback process in the faculty we studied. But it creates a safe environment for constructive dialogues and debates between teachers and students. It unblocks a previously fragmented conversation that was often just a monologue of either the teachers or the students. A small change in the user interface design tells its users that their involvement in the feedback process matters and it rewards those that take it seriously. It also allows the two main actors to have a dialogue when mutually rating their feedbacks or when a wizard level student requests a response from the teacher on the feedback provided.
But what does it mean for the institution, for the Quality Committee? It means enabling a feedback process where the quantity and quality of engagement from both students and teachers increases. This way, the institution can more easily step in to address problems raised by students. It improves the lack of transparency and increases students’ trust in the institution, in teachers. It also means delegating to the digital feedback platform part of the responsibility of being an intermediate between teachers and students. And finally, it also means having a simple metric for evaluating teachers’ involvement in the feedback process, a metric that’s decoupled from their academic performance, thus making it easier for the institution to take action where needed. Teachers know that the institution has visibility over their gradings, which can act as a form of peer pressure determining more and more teachers to get involved in the feedback process. Which in turn has a positive effect for the students and also for the teachers that might have previously felt demotivated that their effort is not appreciated and possibly considered leaving the university.
In the context of design and anthropology, my main take away from this bootcamp is that we have more options than leaving a problem unsolved or solving it perfectly. There is also the option of starting to solve it. A problem is basically a situation where certain actors are blocked. Everyone is repeating in a loop their perspective and that just amplifies the tension. Which I visualise as a pipe that is blocked and through which water cannot flow anymore. A design intervention is just that: an intervention and not necessarily the big answer solving all problems. But it does remove part of the blockage and water starts flowing again. And very often it’s possible that the water that has just restarted flowing will slowly remove any remaining part of the blockage.
