Guide to Ensuring Access to the Publications and Data of Federally Funded Transportation Research (2020)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

CHAPTER 12. Assessing and Managing Progress Once a system for preserving the products of research has been implemented, it will be critical to incorporate metrics that identify progress in implementation and success in meeting essential requirements and achieving reuse. These metrics can be used to provide feedback for changes to improve how the system functions. Initially, assessment will help the state DOT ensure that its researchers are complying with the U.S. DOT’s essential requirements. However, over time, regular assessment can help states in going beyond as they move toward the broader Open Science vision. Assessment of Essential Requirements Chapter 2 describes essential requirements in detail, but for assessment purposes, they break down as follows: 1. Research projects should be reported to TRB’s Research in Progress (RiP) database shortly after the award is received and should be updated during the project. 2. Every project should have an approved data management plan (DMP). 3. Every researcher involved in eligible research should have an Open Researcher and Contributor ID (ORCID) identifier. 4. RiP project status should be changed from “active” to “completed” within 2 months of completion. 5. ORCID identifiers and funding agreement number(s) should be included on the technical report documentation page and in any peer-reviewed publication submissions. 12. A S S E S S IN G & M A N A G IN G P R O G R E S S In This Section » Assessment of Essential Requirements » Assessment of Process and Support for Essential Requirements » Assessing Quality of Interaction » Assessing Policy and Process » Assessment for Going Beyond » Assessing Impact » Developing the Assessment Process » Chapter Checklist 103

104 12. Assessing And MAnAging Progress 6. Research data should be packaged with metadata and deposited in an approved repository. 7. Within 1 year of the end of the project, an email including the following information should be sent to the U.S. DOT Research Hub (RH), the National Transportation Library (NTL), and TRB: • Final report URL(s) or PDFs for any resulting publications, • URL(s) and associated descriptive metadata for any final data sets arising from the research project, • The funding agreement number of the project, • The RH Display ID for the project, What is Tracked? Assessment Metric Target Level Action Number and status of eligible funded projects (tracked at time of funding) Total count, total in progress, total completed Not applicable Shows how the Public Access Plan affects research and how the research organization contributes to public access. Number of projects entered in RiP database Number of entered/total projects Always near 100% because projects must be entered when funded If low, researcher training or communication with researchers may be needed. Number of projects with approved DMP Number of DMPs/total projects Always near 100% because DMPs are required near the start of each project If low, researcher training or communication with researchers may be needed. Number of projects labeled “completed” in RiP Number of marked completed/total completed projects Near 100% with 2-month window If low, compliance personnel may not be advising researchers of the need to mark completion. Number of data deposits Number of deposited/total completed projects Should increase over time Data packing and deposit is one of the most complex parts of the process. This metric will help the research organization track how well its support and training for data documentation and deposit are working. Number of correct technical documentation pages Number of correct/eligible reports Number of correct/total completed projects Should increase over time, but counting technical documentation can be more difficult than tracking reports or deposits. This metric requires the researcher either to submit documentation to his or her organization (e.g., the state DOT), in addition to submitting it to the sponsor, or to report that the task was completed. Support for documentation and communication with researchers about the documentation requirement will be critical if this number is low. Number of successfully deposited reports in NTL registry Number of deposited/total reports eligible Number of deposited/total completed projects Should increase over time. For percentage of completed projects, near 100% should be possible, since it should be feasible to deposit one report per project. However, percentage of eligible reports may be more challenging, since some eligible reports may be missed early. The ideal metric is the percentage of eligible reports that are deposited, but tracking eligible reports is more difficult than tracking completed projects. Low deposit rates indicate a need for more training and communication with researchers. Table 9. Tracking

105 12. Assessing And MAnAging Progress • ORCID identifiers (unique researcher IDs) for all publication author(s), and • Any documented project outputs or outcomes resulting from the research project. Assessing compliance with essential requirements for a state DOT or other research institution requires tracking from the beginning of projects. In doing so, all eligible projects (eligibility criteria are discussed in Chapter 2) should be identified and assessed on the seven items listed above throughout the course of the project. Table 9 shows each item to be tracked, one or more relevant metrics for assessment, and target levels. What is Tracked? Assessment Metric Target Level Action Number and status of eligible funded projects (tracked at time of funding) Total count, total in progress, total completed Not applicable Shows how the Public Access Plan affects research and how the research organization contributes to public access. Number of projects entered in RiP database Number of entered/total projects Always near 100% because projects must be entered when funded If low, researcher training or communication with researchers may be needed. Number of projects with approved DMP Number of DMPs/total projects Always near 100% because DMPs are required near the start of each project If low, researcher training or communication with researchers may be needed. Number of projects labeled “completed” in RiP Number of marked completed/total completed projects Near 100% with 2-month window If low, compliance personnel may not be advising researchers of the need to mark completion. Number of data deposits Number of deposited/total completed projects Should increase over time Data packing and deposit is one of the most complex parts of the process. This metric will help the research organization track how well its support and training for data documentation and deposit are working. Number of correct technical documentation pages Number of correct/eligible reports Number of correct/total completed projects Should increase over time, but counting technical documentation can be more difficult than tracking reports or deposits. This metric requires the researcher either to submit documentation to his or her organization (e.g., the state DOT), in addition to submitting it to the sponsor, or to report that the task was completed. Support for documentation and communication with researchers about the documentation requirement will be critical if this number is low. Number of successfully deposited reports in NTL registry Number of deposited/total reports eligible Number of deposited/total completed projects Should increase over time. For percentage of completed projects, near 100% should be possible, since it should be feasible to deposit one report per project. However, percentage of eligible reports may be more challenging, since some eligible reports may be missed early. The ideal metric is the percentage of eligible reports that are deposited, but tracking eligible reports is more difficult than tracking completed projects. Low deposit rates indicate a need for more training and communication with researchers.

106 12. Assessing And MAnAging Progress Assessment of Process and Support for Essential Requirements The metrics listed above directly address the essential requirements for compliance with the U.S. DOT Public Access Plan. However, the research organization’s structures including tracking, training, and oversight can have a big influence on the levels of compliance seen. Thus, it is also worth having regular assessments of the process and support structures themselves to identify areas for improvement. Assessing Quality of Interaction To assess the quality of interaction of stakeholders in the data preservation process, metrics will rely primarily on user surveys. An ideal preservation process should minimize the burden on researchers to make it more likely that they will comply. Regular surveys may uncover specific areas for improvement and additional support to researchers or personnel ensuring compliance. Specific topics for survey questions should include þ Quality and accessibility of training, þ Availability of and usefulness of DMP templates, and þ Time spent by researchers to complete the steps of the process (e.g., DMP completion, data upload, reporting compliance). In addition to surveys, some specific measurements will also be useful. These may include • Time spent by compliance personnel interacting with each researcher and • Time spent in training, attendance at training sessions, and number of training sessions. Assessing Policy and Process Like the quality of interaction, assessment of preservation policy and processes will rely primarily on surveys. Here, instead of questions about how smoothly the system works in practice, the questions are concerned with whether the high-level policies are appropriate and consistently applied. Specific topics for survey questions should include þ Clarity of stated policies and communication of those policies, þ Clarity and reasonableness of consequences of noncompliance, þ Relationship between policy and practice (i.e., Do they match?). Checklist: Interaction Survey: Question Topics? ? ? Checklist: Policy & Process Survey Questions? ? ?

107 12. Assessing And MAnAging Progress Assessment for Going Beyond Once a structure is in place to track compliance with essential requirements, additional measures can be considered for going beyond. Going beyond the minimum includes expanding the scope of publications that the state DOT considers eligible. The state DOT may also go beyond by establishing its own data and report repository internally or by contracting with an outside repository. The assessments previously described apply to going beyond as well. However, the scope of tracking will increase. Once a state DOT decides to go beyond, however, assessment of the impact of that choice will become particularly important. The motivation for essential compliance may be to avoid consequences of noncompliance, but the motivation for doing more will be to support the goals of Open Science and for the state DOT’s work to have a greater impact on transportation science and practice. Thus, assessment for going beyond will focus on assessment of impact. Assessing Impact A good starting point for assessment of impact is to ensure that the preservation process meets the FAIR guiding principles: Findable, Accessible, Interoperable, and Reusable (see Figure 11, p. 108). The one FAIR guiding principle implemented primarily by the researcher is describing the data with rich metadata. The rest of the time, most FAIR guiding principles are directly implemented by a trusted, long- lived repository. For instance, researchers do not need to procure persistent identifiers themselves, although they should still confirm that a repository conforms to the FAIR guiding principles. Assessment of these guiding principles (whether implemented by an external repository or the state DOT’s own internal repository) will rely on a user survey. The survey should ask about the user’s experience with reuse of other data sets, when problems with metadata will become more obvious. A repository that conforms to the FAIR guiding principles well will tend to maximize impact. However, it is also important to measure impact directly, through tracking of data set downloads and citations. The number of data sets reused and the number of papers or data sets cited is the gold standard metric for how much impact a data-sharing system has on the field. As researchers in all organizations doing transportation research become aware of the availability of data and reports, the rate of reuse should increase and impact should accelerate. ? ? ? Beyond minimum Expanding the eligible publications

108 12. Assessing And MAnAging Progress Developing the Assessment Process Good metrics require early establishment of a good tracking system with the metrics in mind. These metrics particularly call for attention at three stages of the system: (1) selection criteria for recommended or approved repositories, (2) a tracking system established when the researcher is first awarded a project that uses federal funds, and (3) embedding the survey into standard operating procedures at all levels. Survey questions should address the topics suggested above but should also focus on the elements of the process on which the organization is focused at each stage. Taking small steps to start and then building from there is suggested. The survey, in particular, should mirror these steps, so that, early on, questions focus on the elements that have been implemented most recently and therefore might be most in need of adjustment. Figure 11. FAIR Guiding Principles eusable 1. (Meta)data have a plurality of accurate and relevant attributes. • (Meta)data are released with a clear and accessible data usage license. • (Meta)data are associated with their provenance. • (Meta)data meet domain-relevant community standards. nteroperable 1. (Meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation. 2. (Meta)data use vocabularies that follow FAIR principles. 3. (Meta)data include qualified references to other (meta)data. ccessible 1. (Meta)data are retrievable by their identifier through the use of a standardized communications protocol. • The protocol is open, free, and universally implementable. • The protocol allows for an authentication and authorization procedure, where necessary. 2. Metadata are accessible, even when data are no longer available. indable 1. Data and metadata are assigned a globally unique, eternally persistent identifier. 2. Data are described with rich metadata. 3. (Meta)data are registered or indexed in a searchable resource. 4. Metadata specify the data identifier.

109 12. Assessing And MAnAging Progress Chapter Checklist From this chapter you should be able to þ Identify metrics to track each element of the essential requirements. þ Set up a system that tracks eligible projects from the start to support assessment. þ Develop questions for a user survey that will address policy and process from the user point of view. þ Identify metrics to assess impact.