How might we conduct slow longitudinal research over 12 months in order to benchmark findings and measure impact ?
The Sanitation Technology Platform (STeP) at RTI International, with support from The Bill & Melinda Gates Foundation (BMGF), embarked on a multi-year program bringing together commercial partners, scientists and academic labs developing sanitation technologies for systematic piloting. As the design researcher, my role was to assess the user experiences - changes in perception and behavior - of different stakeholders interacting with the system over a period of time for the technology, currently being piloted in three locations in Coimbatore, India. Together with a team of researchers, we conducted a study over 12 months leading the planning, design, and deployment of qualitative research probes that would help in a longitudinal documentation of the pilot technology as it was installed and used by the research participants.
Key research objectives included studying interactions, behaviours, and perceptions throughout the pilot study to understand user feedback/reactions to the technology and to track the nature of change in these over time. The probes were administered by a field team at four critical project phases namely pre-baseline, baseline, midline, and endline.
A combination of ethnographic research, qualitative research methodologies and related design tools were administered at the following critical evaluation stages. Data and Insights were captured at key stages to understand changes in stakeholder attitudes and behaviours.
Setting a foundation for research and evaluation
An initial pre-baseline study was conducted to develop a foundational understanding of the ecosystem (i.e., users, service providers, existing systems, other stakeholders). The study aimed to establish a frame of reference for key user perceptions and behaviours around water and sanitation, and to aid in subsequent framing and deployment qualitative research probes.
A detailed research plan was outlined to include possible tools and methods to be deployed for collection of user data. A questionnaire containing key information heads of basic utilities, cleanliness, sanitation and hygiene, waste management, and new system perceptions was developed and vetted following detailed feedback from the RTI team. We developed a cue card activity as part of the instrument to engage participants in conversations about the utilities they use in order to map various attributes, consequences, and values associated with different utilities to analyse variations across participants.
We conducted open-ended ethnographic immersions in 3 households and a bachelors’ dormitory room at Serene involving members of the household to get additional perspectives wherever possible. The interviews with the bachelors were conducted in dyads (i.e., simultaneously with the two participants).
WHO WE TALKED TO: 3 Families, 2 Bachelors, and 2 Facility Managers.
The immersions included a walk-through of the user’s household documented through notes and photos. Participants were asked to walk us through their cleaning rituals for various spaces paying special attention to the purpose and utility of each of these rituals and their perceived efficacy. Additional analogous research was conducted to support the immersions. We interviewed a facility manager and a facility walk-through with him observing and documenting the waste management system and its interactions with the stakeholders.
Participants were asked to walk us through their cleaning rituals for various spaces paying special attention to the purpose and utility of each of these rituals and their perceived efficacy
Data captured through the methods mentioned above was visualised and developed in the form of detailed maps of the household, as well as a site map for the larger site illustrating key points of interaction between the existing waste management system and its stakeholders. Insights gleaned from the pre-baseline were critical to the framing and alignment of key research inquiries for long range monitoring and evaluation over the subsequent project phases.
Benchmarking current perceptions and behaviours.
The primary objective for this phase was to frame an understanding of key perceptions and behaviours pertaining to water and sanitation across all users. Emergent themes from the baseline research were longitudinally tracked in subsequent pilot phases. Insights and information from the field research during the baseline was used to sharpen the survey tools for the next phase.
Similar to the pre-baseline, data was collected on the usage of and attitudes towards water and sanitation systems, eliciting feedback from users on the functionality of the system. The field team interviewed a total of 15 users living in a housing format of family quarters (i.e., married staff) and dorm rooms (i.e., bachelors). The team gathered responses across the entire user base through short in-person sessions with all users at their residences and the facility manager for the pilot site
The interviews were accompanied by a household walkthrough activity and a mapping exercise on water sources and their associated uses within the house. The sessions weredocumented to complement user data and insights with visual evidence of the user’s contexts.
Learnings from the previous research phase help frame and deploy the survey instruments. A household walkthrough activity was included in the questionnaire to map various sources of water and the associated rituals within the household.
Documenting change in perception and behaviour
Recognizing the critical link between technologies and human behavior, the research in this phase examined the technology’s impact on stakeholder perceptions, knowledge, and behaviors, and the implications of these for technology adoption and sustained use. The research objectives for Phase 1 aimed to understand the change in attitudes and behaviors of the users in the initial adoption period with the system having been turned on in closed loop for 2-3 weeks when user feedback was elicited.
A qualitative research partner was commissioned to conduct the interviews at the site. In order to aid efficient longitudinal documentation as the system was installed and loops closed, we deployed two types of qualitative probes. Insights from the previous rounds of data collection were used to frame a trimmed down version of the questionnaire with guiding questions including the positive and negative experiences of usage, implications for future adoption, and interpreting the value of the system.
Acknowledging that memory recall may suffer from delayed data collection, we developed a calendar for users to hang near the bath/toilet space and record their experience of using the new system.
A total of 12 interviews were conducted with users at their residences with a focus on gathering feedback on the system and resultant change in behaviours. Members of the household used the calendar to share their experience of using the toilet and record their feedback in the calendar. Calendars helped the users to detail perceptions that may have affected early-stage behaviours and attitudes.
Measuring impact over sustained user-technology interaction.
The focus for the final phase of research was to understand reactions to the system after continued use over a few months and to gain insight into how users understand and interpret the value of the system over time.
User responses from phase 1 were evaluated to frame a concise survey instrument in order to make the users reflect on their experience of interacting with the system and changes in perception over a period of time. We then identified and developed additional tools to complement the survey. For instance, a calendar check-in was included to review the participant's daily feedback routines and for it to serve as a probe for specific questions according to the nature of the feedback. An additional role playing activity was designed to witness how users respond to certain scenarios as realistically as possible in order to capture feedback in alternative ways without fatigue.
We interviewed 6 users at the location who had been involved in the pilot since the beginning (dropouts were observed in the number of users present from the beginning of the study with at least 5 new users replacing the previous ones.) Users were encouraged to think more about how things changed and the impact of the sanitation system within and beyond their household. We elicited feedback on their understanding of the system, perceived impacts, toilet use, change in behaviours, and reactions to the system.
(caption top) Role plays were an effective method towards understanding any additional system related behaviours and perceptions apart from the interviews. Users who had interacted with the system from the beginning were invited to participate in a 30-minute long session. After introducing the activity through some examples, and assigning into teams of two, a total of 4 scenarios were given to the participants to act out.
(caption left) During the interviews, calendars deployed during phase 1 were reviewed to check for the frequency of updates, as well as commonly occurring issues and challenges in recording feedback. Concerns recorded by the users were probed further during the interviews sessions.
Data obtained from the processes mentioned above was compiled into a summary of findings for the endline. Findings from this phase will be fed into the field team’s efforts to assist with the on-going technical evaluation of the system.
Research through the different project phases helped us gain a better understanding of user's context, their sanitation practices and behaviours, and changes in these practices over a period of time. While the resulting insights at the end of each phase were manifold, these findings and our understanding of the context translated into a set of overarching insights listed below.
A necessary mandate of the project was to be able to measure changes in behaviours associated with toilet and water usage with respect to the technology being tested. By employing design and design thinking, we developed tools, measures, and approaches to better gauge and promote the impact that the technology had on people, the environment around them and their interactions with it. These measures were critical toward adjusting and transforming the built environment for our users social contexts and values.
Through the course of the pilot study, extensive user data and insights were captured on the experience and behaviours of stakeholders with respect to technology and sanitation before, during and after installation of the new technology. Various design tools and methodologies administered at critical stages proved an effective way to bring in stakeholders and invite feedback. The tools themselves provided interesting alternative formats for research work which made the process all the more fun and rewarding.
Insights from this research were fed into the field teams efforts at each phase of the project to inform ongoing technical evaluation of the system. The findingsstand to guide future product development that would enhance the end user experience of similar technology systems.
For the users, the overall experience of using the treated water was largely positive. Early on, many users viewed the new system as an important intervention in addressing problems of water shortage. Initial concerns regarding physical contact with treated water and uses of the water were addressed through town-hall sessions with the users and through interactions with the field team.