BY: MOTIVATE DESIGN
User experience research is never easy. While there are a variety of methods for testing products and services with potential end users, in most cases these tests don’t allow researchers to understand how people might engage with them on an everyday basis.
Diary studies are one approach to revealing this information. In these studies, which usually run for a fixed period of time (days, weeks, or even months), people are encouraged to self-report on their experiences with a product, service, or task. This reporting is usually done online in the form of diary entries, blog posts, video clips, or audio recordings. In most instances, the reporting is facilitated through a smartphone, which can also track important usage data throughout the course of the study. Once the reporting period is completed, researchers analyze the data to identify qualitative and quantitative insights that can help them better refine the user experience to cater to the customer’s actual needs.
Like any research method, diary studies have a number of benefits and potential drawbacks that researchers should be aware of from the beginning. Here are some of the key points to keep in mind:
Pros
- Record experiences in a natural environment: The principle advantage of diary studies is that they give researchers the opportunity to discover how people utilize products and services in a natural environment. Many research studies intentionally strip away outside factors to create an environment free of variables so the user can focus solely on the product or service. While this can provide some useful information for designers, it’s also usually not reflective of the average user experience. Take, for instance, a social media app. A user might find it easy and intuitive to use sitting in at a table in an empty room. But how do they feel about using it while filing into a crowded subway or while waiting in line at a coffee shop? Diary studies offer researchers a window into that everyday experience, which can provide them with tremendous insights that might not have been apparent in normal usage testing.
- More likely to capture influential external factors: In addition to helping researchers understand how people use products and services, diary studies can also reveal external factors that affect the user experience. There’s simply no way to know when or where someone will use a product most often, for instance, until they actually have that product in their hands. Researchers can also learn what triggers might cause people to use that product, or what social situations might affect usage patterns. Rather than simply asking questions about it during the design phase or waiting to gather data after a rollout, diary studies provide a sort of “soft” prototype launch that can generate invaluable information that can greatly improve the final user experience (to the extent that the user experience is ever finalized).
- Collect observations in longer durations: With any product or service, there’s always a concern about novelty. Will people use something more regularly when it’s new and then abandon it after that initial enthusiasm wears off? Initial testing during the iterative design phase simply doesn’t last long enough to answer this question. Longer periods of use also allow time for more problems or concerns to emerge. They might even reveal a potential that goes beyond the scope of the original design, but could be incorporated in a new iteration (such as a new feature on a device). Only by actually allowing people to engage in the user experience over a period of time can researchers begin to answer this question in ways that produce valuable information.
- More time for in-depth consideration and opportunities for creativity: One of the challenges of testing products and services is that the people using them often don’t have the time to put serious thought about how they feel about the user experience. Initial impressions are certainly important, but sometimes users don’t have a good sense about how using something makes them feel until days after the initial use. They may need time to put their finger on what they feel or find a way to put those feelings into words. Diary studies provide multiple opportunities to give feedback in a variety of formats, which can often encourage people to not only think deeply about their experience, but also find unique ways to express how they felt. These insights can be extremely valuable to designers.
The principle advantage of diary studies is that they give researchers the opportunity to discover how people utilize products and services in a natural environment
Cons
- Researchers don’t get to observe participants: One of the recurring themes of user experience research is that people don’t often know what they want. Similarly, they also tend to not know what they’re doing. Some of the most valuable insights derived from design-thinking approaches to research that come from simple observation. Watching how people engage in a user experience can reveal patterns and trends that no one would think to ask about or report. By relying on the observations of study participants, researchers don’t have the opportunity to observe the user experience over time.
- Participants may not be totally accurate: Just as people don’t always know what they want or what they’re doing, they also have blind spots that can inject bias into their observations and recollections. In some instances, they may even behave differently because they know they’re recording their thoughts and feelings for someone else to read or hear. The inability to accurately recall events can also present a problem. While researchers can safely throw out obviously biased or untruthful responses, there are few safeguards in place to ensure that participants are recording their experiences accurately and honestly.
- Long selection process: Diary studies put a great deal of pressure on participants to be creative and detailed in their responses. Unfortunately, not all potential subjects will feel comfortable being so forthcoming or with finding unique ways to express themselves. In order to make sure that people don’t drop out of the study or provide less than adequate responses, researchers need to be highly selective in who they involve in such study. This, unfortunately, extends the preparation time as they try to find the best candidates, especially compared to more traditional research methods. There’s also a danger that this selection process will introduce an inherent bias into the test by filling the study with a certain type of user who may not be typical of the intended end user for the product or service.
While diary studies have a few notable shortcomings, the benefits of this approach to research make it a valuable option for organizations seeking to learn more about the genuine user experience. No approach to research is perfect, after all, and in any case, diary studies should form only one component of a broad array of research methodologies. By combining them with more traditional test methods and other innovative strategies like friendship groups, diary studies provide a glimpse into a specific aspect of the user experience that can help designers and engineers develop better products, services, and delivery methods.
