A
Use Case Scenarios Report for SMART Vaccines
Lori Ada Kilty
Microsoft Research
Introduction
The primary reason to carry out usability studies is to improve the design of products through observation of, and conversation with, real users. Real users can describe what they think or feel and demonstrate how they use the product or application. It is often best to start such usability studies early, before a product is fully designed or even when it is still in the idea stage, and then continue the studies throughout the development process. This is not always possible, however. In the case of SMART Vaccines, the contracts for Phases I and II did not commission the Institute of Medicine to conduct early-stage usability studies along with the product development effort.
Although many define usability and usefulness separately, I consider usefulness the first principle of usability. In my view, it does not matter how easy it is to use a product if it is not useful. Early site visits help to ensure the usefulness of the eventual design. For SMART Vaccines specifically, the usefulness was already established before the Phase III project began. Thus, the purpose of these site visits at this stage was to formally understand the usability of SMART Vaccines from real decision makers.
The typical metrics used in the software industry for usability studies—such as completion rate, errors, assists, task time, and mean time on task—
are primarily concerned with effectiveness and efficiency. The Phase III user group studies did not involve testing participants to this degree, but it is still useful to have these metrics in mind, even in the early stages and going forward—especially when thinking through what the expected target range should be.
Methodology
The usability study was conducted with three user groups: the Public Health Agency of Canada, New York State Department of Health (NYSDOH), and the Serum Institute of India. Two site visits (which involved interviews and direct observation) were undertaken at the Public Health Agency of Canada and NYSDOH. The Serum Institute of India provided feedback electronically.
The two visits took place in a conference room, not the users’ typical work environment. This made it easier to have in-depth discussions with multiple participants who were able to explain their intended use scenarios. However, these visits did not allow for direct observation of how the tool was employed in the users’ routine decision-making process.
User Scenarios and Key Interest Areas
The three user groups were interested in using SMART Vaccines in the following scenarios:
1. Identifying new vaccine candidates and influencing their development using an analytical system, which included the following features:
• Transparency
• The ability to facilitate discussions among provincial and other organizational leaders
• The potential to shorten the time to new product licensing
• A better understanding of the value of lifelong immunizations
• A lifelong reduction of disease burden
• Reduce the current “piecemeal” approaches among stakeholders and help improve coordination among them
2. Prioritizing existing vaccines for introduction, focusing on different disease variations, including
• Comparing the effects of a vaccine on the whole population of interest versus the effects on a targeted population
• Determining whether the packaging of vaccines (e.g., 1-dose syringe versus 10-dose vial) has an impact on desired public health goals
• Examining differences in cost, efficacy, and other factors between oral and injected vaccines
• Comparing the impact of vaccines requiring multiple doses with the impact of single-dose vaccines
• Determining whether certain ingredients will cause issues with uptake (thimerosal, for example)
Key Observations and Suggestions
The following observations concern only the user studies conducted with the Public Health Agency of Canada, NYSDOH, and the Serum Institute of India. Each group had different scenarios, and it should be made clear that many of the uses they were interested in were extensions of the original application of SMART Vaccines, which was to prioritize new preventive vaccines. The observations represent a composite of feedback provided by the three user groups, and some of the observations reflect feedback from one or two of the groups, rather than from all of them.
Although there were issues that came up during the usability studies that concern data presentation in SMART Vaccines, these issues are not included in this appendix because they are addressed by the committee elsewhere in the report as part of the data framework.
Table A-1 lists some user-reported bugs along with my suggestions to fix them. No critical bugs in SMART Vaccines were reported by the users, but the other, non-critical bugs are summarized in the table with regard to their major and minor impacts. Table A-2 contains additional use case observations with suggestions for enhancements. Table A-3 summarizes the positive attributes discussed by the users.
TABLE A-1
User-Reported Bugs and Suggestions to Fix Them
| Observation | Notes | Severity of the Bug | Suggestion |
|
1. Unable to save progress consistently. |
This bug was fixed prior to the site visit, but the results were inconsistent and did not consistently work for one user group member. The committee has not been able to reproduce this error, and it is likely that it is particular to the test subject’s individual computer configuration. |
Major |
The committee should continue to investigate this due to the high level of frustration this can create and should provide the ability to save progress consistently, throughout the software. |
|
2. Disease burden percentages appear to add up to 100 percent, but the tool still does not accept them consistently. |
Specifically, the problem occurred when one or more decimal places were in use, even if the percentages clearly added up to 100 percent. However, there were cases when one decimal place was accepted, and this behavior is inconsistent and confusing. |
Minor |
The tool should either allow one or more decimal places consistently or specify the decimal place limit for percentages or better inform the user about the data entry needs. |
|
3. The total attribute acceptance limit is not clear, and the software run does not complete if the limit is exceeded. |
|
Minor |
The tool should either inform the user on the limit of 10 attributes or increase the limit, or do both of the above. |
|
4. “Death” as an outcome is required even for diseases with no morbidities. |
|
Minor |
The tool should either inform the user that “Death” is a required outcome or allow the user to set criteria in advance so that the tool only requires data for criteria specified. |
TABLE A-2
Additional Use Case Observations and Suggestions for Enhancements
| Observation | Notes | Suggestion |
|
1. Age refinements are limited. |
Age groupings are too broad. Each morbidity entry should correspond with an appropriate age group, similar to the case with gender. Age groups available under disease burden today are not sufficiently granular for two of the user groups. The ability to compare different regions would be useful as well, according to one user group’s preference. |
Create more granular age groups and allow these groups to be combined as necessary. Suggest using the World Health Organization age group dataset. |
|
2. Subpopulation choices are confusing. |
It is not clear that selecting the subpopulation (female or male or special) applies only to disease and vaccine information and that the SMART Scores are calculated on the basis of the whole population. A whole population choice in the tool would be useful if subpopulation is not desired. |
The tool should clearly inform the user that the subpopulation data pertain only to the specific disease and vaccine candidates under consideration. The final results are based on the whole population. The committee should consider adding a full population option for analysis. |
|
3. Attributes not required for a scenario are required by the tool, which can adversely impact results. |
There are criteria that need to be specified even if they are not applicable to the scenario that the tool is being used for (e.g., time to adoption, research and development costs). If this tool is going to support the prioritization of already developed vaccines and not only new vaccine development, then there will be cases when some attributes do not need to be required. |
There should be an option of “zero” or “NA” for attributes that are not required for all scenarios. |
|
4. Data exist to calculate “total cost” offline and must be entered into the tool manually. |
“Total cost calculated” is currently calculated by the tool, but all the data are already in another file and need to be hand entered into the tool, which is tedious. |
Allow the import of spreadsheet files or requests for a “total cost” calculated offline in order to simplify data entry. To eliminate confusion, this value needs to be clearly defined. It would be better to ask for the “total cost” (assuming the user knows how to calculate this figure) and eliminate duplicate data entry or separate file upload. |
TABLE A-2
Continued
| Observation | Notes | Suggestion |
|
5. The field highlight and the cursor color are both blue, and this is confusing. |
|
Changing the color of either the cursor or the highlight would eliminate the confusion and should be an easy fix. |
|
6. Attributes, Weights, and Priorities need to be reentered whenever a change is made. |
Only weights can be adjusted without having to be reentered. If a user-defined attribute is added, then everything else will need to be reentered. |
Allow an option for Attributes, Weights, and Priorities to be adjusted without reentering all the choices. |
|
7. Must select “Continue” when on a previously completed screen instead of navigating from the top. |
It was unexpected behavior that when going back to an earlier screen, instead of selecting at the top of the screen to navigate, the user needed to select “Continue” and the data had already been entered and saved. |
Eliminate the need to select “Continue” when going back to a screen that is already complete. |
|
8. Change Attributes with Yes/No to Likert scale to allow for more granularity. |
The last three attribute groups (programmatic considerations, intangible values, and policy considerations) in the tool are not granular enough and offer only Yes/No as options. A Likert-scale gradation would be useful here. On the plus side, user-defined attributes can be added, and these do offer a Likert scale. |
Change Attributes with Yes/No inputs to a Likert scale to allow for more granularity. Provide guidance on adding user-defined attributes. |
|
9. Vaccine-related complications should be a quantitative entity instead of an attribute. |
Vaccine-related complications were originally a quantitative entity (in Phase I), but they were changed to an attribute (in Phase II) because they were not a priority for the developing-new-vaccines scenario. However, it would be much more beneficial to have this as a quantitative entity rather than an attribute for prioritizing existing vaccines. It could be useful for new vaccine development as well because users may want to set the tolerance level. |
Consider changing vaccine-related complications to be a quantitative entity again instead of an attribute. |
TABLE A-2
Continued
| Observation | Notes | Suggestion |
|
10. Multi-disease vaccine comparisons are difficult. |
When it was pointed out that the tool treats multi-disease vaccines as separate diseases and does not handle this scenario well, one of the user group members came up with a workaround by defining a second disease separately for analysis. |
The tool should eventually consider allowing multi-disease vaccines to be compared without treating composite diseases separately. |
|
11. Additional clarity required on data entry needs. |
It is not clear how the data on direct costs of vaccine use per year or incident cases prevented per year, for example, are being used in the software. |
The committee should consider offering additional information through notes or tool tips so the user understands how these data entries work. |
|
12. Results or output cannot be saved. |
After all the effort to set criteria and create a run, it would be hugely beneficial if there was an ability to save the results, especially if users are unable to save their progress consistently. |
Provide the ability to save the output results. |
TABLE A-3
User Reported Positive Features
| Observation | Notes |
|
1. The calculations are accurate. |
The numbers calculated were confirmed accurate by a user group member (subject-matter expert in this field) who independently validated the calculations on the side, noting that most users would not be able to easily do that. In summary, the fundamentals are solid. |
|
2. The screen layout and color scheme are pleasant. |
Another user group member emphasized that the software layout and colors were appealing. |
|
3. The final results are easy to comprehend. |
Because the tool provides comparative SMART Scores with color coding for vaccines, the final results are easy to comprehend. |
Conclusion
The studies with the three user groups were productive and contributed to a deeper understanding of core issues and usability. Some of the more challenging issues concern data quality and the ability to properly segment populations. The committee should review alternate scenarios, such as prioritizing existing vaccines, and study the user requirements to determine whether SMART Vaccines can be appropriately adjusted to effectively meet this task.
Most of the remaining usability issues can be addressed with tool tips and call-outs to help the users understand what they are doing. Many users of this tool will be casual users, so help within the interface would be invaluable for ensuring a more positive interaction with SMART Vaccines. The ability to save progress and upload data using spreadsheets would go a long way to easing some user frustrations.
