The PhenX (consensus measures for Phenotypes and eXposures) Toolkit (https://www.phenxtoolkit.org/) is a publicly available, web-based catalog of recommended, well-established measurement protocols of phenotypes and exposures. The goal of PhenX is to facilitate the use of standard measures, enhance data interoperability, and promote collaborative and translational research. PhenX is driven by the scientific community and historically has depended on working groups of experts to recommend measures for release in the PhenX Toolkit. The urgent need for recommended, standard measures for COVID-19 research triggered the development of a “rapid release” process for releasing new content in the PhenX Toolkit. Initially, PhenX collaborated with the National Institutes of Health (NIH) Office of Behavioral and Social Sciences Research, the National Human Genome Research Institute, and the NIH Disaster Research Response (DR2) program to create a library of COVID-19 measurement protocols. With additional support from NIH, PhenX adapted crowdsourcing techniques to accelerate prioritization and recommendation of protocols for release in the PhenX Toolkit. Prioritized COVID-19-specific protocols were used to anchor and define specialty collections of protocols that were subject to review and approval by the PhenX Steering Committee. In addition to the COVID-19-specific protocols, the specialty collections include existing, well-established PhenX protocols, use of which will further enhance data interoperability and cross-study analysis. The COVID-19 specialty collections are Behaviors and Risks; Ethnicity, Race and Demographics; History, Treatment and Outcomes; Information Resources; Psychosocial and Mental Health; and Socioeconomic. The development and usage of PhenX COVID-19 specialty collections are described in this article. © 2021 The Authors.

Basic Protocol : Selecting COVID-19 protocols

The NIH-Wide Strategic Plan for COVID-19 Research states that the National Institutes of Health (NIH) “will facilitate the use of COVID-19 data to the greatest extent possible, both by those generating the data and by other researchers” (HHS/NIH, 2020, p. 29) and has responded with key behavioral- and social science−related efforts as part of the NIH coronavirus disease 2019 (COVID-19) research response (Riley et al., 2020). The NIH engaged PhenX (consensus measures for Phenotypes and eXposures) to help make COVID-19 data collection more consistent and results more readily comparable. Historically, the PhenX consensus process has relied on the contributions of working groups of experts, feedback from the scientific community, and overarching guidance from the PhenX Steering Committee (SC) to recommend well-established, standard measures for release in the PhenX Toolkit (https://www.phenxtoolkit.org/). The urgent need for recommended, standard measures for COVID-19 research triggered the development of a “rapid release” process for new Toolkit content (for terminology, see Table 1 and Table 2). Prioritized COVID-19-specific protocols were used to anchor and define specialty collections of protocols that were subject to review and approval by the PhenX Steering Committee. In addition to the COVID-19-specific protocols, the specialty collections include existing, well-established PhenX protocols, use of which will improve the consistency of data collection and enhance cross-study analysis. Initially, PhenX collaborated with the NIH Office of Behavioral and Social Sciences Research, the National Human Genome Research Institute, and the NIH Disaster Research Response (DR2) program to create a library of COVID-19 measurement protocols (referred to as protocols; see Table 1). With additional support from the NIH, PhenX adapted crowdsourcing techniques to accelerate prioritization and recommendation of COVID-19 protocols for release in the PhenX Toolkit, established COVID-19 collections of protocols, and made it easier to find COVID-19 protocols in the PhenX COVID-19 library.

Accordingly, the PhenX Toolkit provides two different resources for investigators interested in COVID-19 research: COVID-19 collections of protocols that are recommended as standard measures or common data elements, and a COVID-19 library of protocols currently in use. The COVID-19 collections of protocols are fully integrated into the PhenX Toolkit and include a combination of COVID-19-specific rapid release protocols and well-established PhenX protocols. Toolkit features to select and implement these collections of protocols are described. The COVID-19 library (of pdf files) is provided as a reference so that investigators can see what other COVID-19 protocols are in use. Search and sort features help visitors explore more than 100 measurement protocols and 400 modules (components of measurement protocols). The COVID-19 collections of protocols (https://www.phenxtoolkit.org/collections/view/8) can be reached by clicking the COVID-19 Research button on the homepage (Fig. 1). The PhenX SC also recommends use of the Social Determinants of Health (SDOH) Core collection of protocols for COVID-19 research. The SDOH collections (https://www.phenxtoolkit.org/collections/view/6) can be reached directly from the homepage or from the COVID-19 Research Collections page.

Strategic Planning

PhenX Toolkit COVID-19 library

In a collaboration with NIH DR2, PhenX created a COVID-19 library (Fig. 2). It is important for users to understand that this a resource separate from the typical PhenX infrastructure, for protocols that are processed through the PhenX consensus process and bioinformatics pipeline. The purpose of this resource is to provide current protocols focused on COVID-19 research; it was intended to help reduce the proliferation of new COVID-19 measurement protocols (related terminology found in Table 3). The PhenX COVID-19 library provides currently in use, publicly available COVID-19-related measurement protocols (case report forms, data clarification forms, surveys, questionnaires), and consists of 127 full instruments (or “measurement protocols”, Table 3) and 506 modules in PDF format. These protocols were submitted to NIH and then sent to PhenX for further processing. There are 14 main topics, which come from NIH DR2, and over 100 subtopics, which are PhenX-created. The full protocols are broken into modules, which are subject-specific. The protocols listed cover a range of topics, from COVID-19 symptoms and treatments to mental health and socioeconomic changes due to the pandemic. Protocols are categorized both by their source organization and by main topics and subtopics. Each module is assigned from 1 to 3 subtopics. The goal of this page is to provide a one-stop directory for investigators interested in incorporating existing protocols for COVID-19 research—rather than creating new ones—in their studies. This page is publicly available for use at no cost to the scientific community in a web-based resource.

Creating COVID-19 collections

Based on PhenX SC recommendations, crowdsourcing was implemented to rapidly filter and recommend a subset of the COVID-19 protocols that had been submitted to NIH. PhenX used crowdsourcing to help inform the subject focus of the collections and the content of the collections. The PhenX COVID crowdsourcing campaign took place in three distinct phases. The phases were distinguished based on their intended purpose rather than their touchpoint with participants. The crowdsourcing effort's full breadth is a novel approach, but is grounded in these preexisting methodologies to ensure functional outcomes.

A web-based platform called OptimalSort (https://www.optimalworkshop.com/optimalsort/) was employed to create and distribute all phases of crowdsourcing. Questionnaires appropriate for each phase were programmed in this platform and distributed over e-mail listservs. Phase 1's intention was to discover the protocol areas that researchers, and the general public, felt were important in COVID-19 research. This step used individual “cards” to sort; participants were presented these cards with a 5-min time limit to perform this task. Phase 2's intention was to group the results of Phase 1 into possible collections, using a Dendrogram Study methodology, and using the top 75% of topics from Phase 1 to form the basis of the next card-sort methodology. This step of the process used an “open card sort,” where individuals produced buckets of their design to create groupings without the bias introduced by the study hosts. Phase 2.5 verified the results from Phase 2 by having the participants pick a group for each topic area and place topics into the buckets in which they believed the topics belonged. These categories also should have received their highest-rated naming by the results of Phase 1 and Phase 2 and informed the COVID-19 collection names and study areas. Topics slotted into the categories in Phases 1-2.5 also provided a foundation used to inform which PhenX protocols and new COVID-19 protocols would be a part of these collections. Phase 3 was intended to pick the measurement protocols that would end up in the final collections. Since this step was very time intensive for the user, standard triage measures were used to trim the number of options for each task to a manageable level for participants. Individuals read through the possible protocol options and selected the one they believed was the best option. These were then ranked by the overall vote each one received.

PhenX then utilized the results to inform which existing PhenX protocols and which new COVID-19 protocols would be in these collections. The PhenX steering committee (SC) conducted a final review, and protocols were designated as “rapid release” as part of the evolved PhenX process. Overall, a typical start-to-finish process for a collection takes around 1 year; the process to release the COVID-19 collections took about 3 months. The collections were finalized and released in the PhenX Toolkit on October 30, 2020.

COVID-19 collections

The PhenX Toolkit created six COVID-19 Research specialty collections (Fig. 3, Table 4) to promote use of common data elements. The protocols were released with a “rapid release” status, indicating our effort to make these cutting-edge data collection protocols quickly available to the research community. Primary considerations for selection of the recommended COVID-19 protocols included their being from a reputable source, that they were broadly applicable to both research and clinical assessment, and that that they were based on input from crowdsourcing results. Other considerations included availability in other languages, applicability to people of various life stages, being available free of charge to the public, and willingness to address health disparities.

The creation of COVID-19 Research collections fully integrates the rapid release COVID-19 protocols into the PhenX Toolkit. That is, the Toolkit provides a brief description of each measurement protocol, the purpose and rationale for the measurement protocol's inclusion, standard protocols, references, and requirements (e.g., training, personnel, equipment, applicable licensing fees) for data or specimen collection (Figs. 4-8). Toolkit users can search or browse for these COVID-19-related protocols and save selected protocols in My Toolkit. Users can also download a custom Data Collection Worksheet (DCW) that details their selections and allows them to easily “cut and paste” protocols to incorporate them into their data collection study design documentation. Finally, users can also download a REDCap (Research Electronic Data Capture) instance for data collection.

The following provides a step-by-step process in selecting COVID-19 protocols to include in a new study. In this scenario, an investigator has designed an epidemiological study of the effects of the COVID-19 pandemic on minority populations. The investigator plans to assess a variety of phenotypes and exposures. The investigator is interested in how minority populations may be disproportionately affected by COVID-19 infections and by impacts due to shutdown mandates. The investigator wants to include high-quality measures, ensure consistent data collection, and be well positioned for data sharing and cross-study analysis when the study is complete.

The investigator has designed the study to include collection of basic demographic information and changes that may have occurred since the start of the COVID-19 pandemic. The investigator recognizes that using PhenX measurement protocols in the study will facilitate combining their data with other studies that have also used PhenX protocols and will allow them to more readily replicate findings from other studies. The investigator visits the PhenX Toolkit to identify recommended protocols of phenotypes and exposures that could be included in their COVID-19 study.

Materials

• Computer or laptop
• Internet browser

1.Navigate to https://www.phenxtoolkit.org and browse protocols in the COVID-19 Research Specialty Collections (see Fig. 3).

2.Add the entire Ethnicity, Race and Demographics collection to your toolkit. This will display the contents of the toolkit (Fig. 9).

3.Scroll to the bottom of the page and select “Add all Essential Protocols to your Toolkit” (Fig. 10).

4.Browse Social Determinants of Health: Core (Fig. 11).

5.Search the entire Toolkit for additional measurement protocols (Fig. 12)

6.To see additional COVID-19-related measurement protocols already in use, investigators can visit https://www.phenxtoolkit.org/covid19 to browse the COVID-19 Protocol Library.

7.Once all protocols have been added to “My Toolkit,” click “Select All” or individually select protocols to highlight the protocols of interest.

8.By selecting “Download,” investigators can choose to download a basic report, a detailed report, a data collection worksheet (multiple language options), or a data dictionary (multiple formats).

0. Basic Reports include a listing of all of the protocols in “My Toolkit” and the essential measures associated with them. For each protocol, it lists protocol ID, protocol description, specific instructions, and the protocol text.

1. Detailed reports include a listing of all of the protocols in “My Toolkit” and the essential measures associated with them. For each protocol, it lists protocol ID, protocol description, specific instructions, protocol text, selection rationale, language, participants, personnel and training required to perform the protocol, equipment needs, standards, references, protocol type, derived variables, and requirements.

2. Data Collection Worksheet(s) include questions to collect information from research participants.

3.         Data Dictionaries provides definitions of variables and other relevant parameters for a study's data management system. It also provides detailed information about each variable as well as skip patterns that are implemented throughout.

        Downloading in REDCap format allows investigators to upload directly to their pre-existing REDCap data collections

Background Information

The PhenX Toolkit is a web-based resource that makes it easy for researchers to add high-quality, well-established standard measures to their studies. The value of standard measures is widely recognized (Fortier et al., 2010, 2011; Hendershot et al., 2015; Windle, Bennett, & Noyes, 2011) because they facilitate cross-study analysis and are relevant for genome-wide association studies and other biomedical studies. Without standard measures, investigators must infer correspondence between data that are conceptually related but were collected using different methodologies, and attempt to harmonize disparate data across multiple studies. This can be a time-consuming and difficult process that often reduces the number of variables that can be used for data analysis.

This article highlights the features of the PhenX Toolkit that allow researchers to access and incorporate recommended COVID-19 protocols into their studies. These COVID-19 collections of protocols cover key subject areas related to the COVID-19 pandemic. In addition, researchers can utilize the COVID-19 library to access additional COVID-19-related protocols currently in use by a wide array of sources, such as the World Health Organization and the Environmental Influences on Child Health Outcomes Program. This overview is intended to introduce the PhenX Toolkit COVID-19 specialty collections and to promote the use of these COVID-19 measures and common data elements.

Critical Parameters and Troubleshooting

The PhenX Toolkit is intended to provide support for a variety of study designs. Because not all protocols are suitable for all study designs, decisions on which protocols to select are the responsibility of the investigator. For the protocols included in the COVID-19 Research collections and the entire PhenX Toolkit, the Toolkit provides the following:

• Related Protocols are other protocols in the Toolkit that may be of interest (related protocols are suggestions only);
• Essential Protocols that are required to correctly interpret selected protocols;
• DCW to improve consistency of data collection; and
• Data Dictionary (DD) to enhance data sharing and compatibility with REDCap.

For example, Current Age is considered essential to interpreting COVID-19 Related Employment Status. When a user selects COVID-19 Related Employment Status, they are encouraged to also select Current Age. The Toolkit also offers a DCW and REDCap DD. The DCW helps investigators integrate PhenX protocols into their existing research and also helps ensure that investigators collect the data needed for their measures. The DD describes the variables associated with the selected measures.

Understanding Results

PhenX provides a resource of recommended and standard measurement protocols for use by the research community. By incorporating standard protocols into their studies, investigators can amplify their results by comparing or combining their data with other studies using the same standard protocols. NIH is supporting and coordinating multiple data science and research efforts to better understand, treat, and mitigate COVID-19. Specifically, NIH supports data sharing and data interoperability via the PhenX Toolkit, the NIH Public Health Emergency and Disaster Research Response (DR2), and the Systemic Harmonization and Interoperability Enhancement for Laboratory Data (SHIELD). NIH also supports COVID-19 research initiatives and development of new technologies, as evidenced by the newly established Collaborating Network of Networks for Evaluating COVID-19 and Therapeutic Strategies (CONNECTS), the National COVID Cohort Collaborative (N3C), and the Rapid Acceleration of Diagnostics (RADx) initiative.

NIH has posted 11 requests for application, 9 notices, and 1 program announcement related to COVID that mention PhenX in 2020 from the Office of the Director, National Institute on Drug Abuse, National Institute on Minority Health and Health Disparities (NIMHD), and National Institute of Mental Health (NIMH). These funding opportunities from the Office of the Director, NIMHD, and NIMH encourage the use of both the PhenX COVID-19 library and collections.

For the PhenX COVID-19 collections, the process is an evolution of the PhenX consensus process that allows for more rapid identification of high-quality measures suitable for release in the Toolkit. PhenX will add a variable compare tool that is envisioned to assist investigators in finding protocols best suited for their research needs.

The COVID-19 collections will allow researchers to combine and compare studies with increasing ease. Studies need to be compared in order to replicate results. In addition, combining studies increases the sample size, thus providing the statistical power needed. Widespread adoption of PhenX measures will have a dramatic impact on biomedical research and ultimately on the health and well-being of the population. Using the NIH-recommended COVID-19 efforts alongside PhenX will help increase understanding of the COVID-19 pandemic and its related socioeconomic effects.

Time Considerations

The time required to collect data on a particular measure is addressed in the Requirements Table (see Fig. 5) for each specific protocol. Time as it pertains to use of the PhenX Toolkit is difficult to estimate. It could take as little as an hour for an investigator to select a few measures and request a Report, DCW, and DD. However, it is more likely that an investigator would spend an hour or two becoming familiar with the Toolkit, select a few measures, and then return to the Toolkit later to select additional measures. The PhenX Toolkit should be used as an integral part of the development of a new study.

Acknowledgments

Research reported in this publication was supported by the National Human Genome Research Institute (NHGRI) of NIH under award numbers U41HG007050 and 3U41HG007050-08S1. Collectively, these awards were funded by NHGRI, the National Institute on Drug Abuse (NIDA), the National Heart, Lung, and Blood Institute (NHLBI), the Office of Behavioral and Social Sciences Research (OBSSR), the Office of the Director, National Institutes of Health (OD), the National Cancer Institute (NCI), and the National Institute on Minority Health and Health Disparities (NIMHD). The authors would like to acknowledge: Dr. William T. Riley of NIH for his contributions in the development of this project in its early stages and his continued work in NIH COVID-19-related research efforts; Dr. Erin M. Ramos of NHGRI for her contributions to the initiation and development of this project; the PhenX Steering Committee for their guidance: Drs. Mary L. Marazita and Cathy A. McCarty (Co-Chairs), Drs. Lindsay A. Farrer, Elaine M. Faustman, Jonathan L. Haines, Tabitha P. Hendershot, Erin M. Ramos, Marylyn D. Ritchie, Sharon F. Terry, Marsha J. Treadwell, and Rosalind J. Wright; Dr. Michael Spittel of OBSSR, Dr. Chandra Keller of National Institute on Aging, and Dr. Aubrey K. Miller and Dr. Richard Kwok of National Institute of Environmental Health Sciences for their help with the intake of COVID-19 measurement protocols. The authors also thank Wally Campbell, Jennifer Drolet, and August Gering for editorial assistance, and RTI's Creative Services group for graphics assistance. The content is solely the responsibility of the authors and does not necessarily represent the official views of NIH, NIDA, NHBLI, OBSSR, OD, NCI, NIMHD, or any of the sponsoring organizations and agencies, or the U.S. government.

Author Contributions

Michelle C. Krzyzanowski: Conceptualization, Data curation, Funding acquisition, Methodology, Software, Supervision, Writing-original draft, Writing-review & editing, Ian Terry: Data curation, Formal analysis, Methodology, Software, David Williams: Data curation, Software, Writing-review & editing, Pat West: Writing-review & editing, Lauren N. Gridley: Data curation, Resources, Carol M. Hamilton: Conceptualization, Funding acquisition, Investigation, Project administration, Supervision, Writing-original draft, Writing-review & editing

Conflict of Interest

The authors declare no conflict of interest.

Data Availability Statement

No new data is reported in this protocol paper, and so none is being made available.

• Fortier, I., Burton, P. R., Robson, P. J., Ferretti, V., Little, J., L'Heureux, F., … Hudson, T. J. (2010). Quality, quantity and harmony: The DataSHaPER approach to integrating data across bioclinical studies. International Journal of Epidemiology , 39(5), 1383–1393. doi: 10.1093/ije/dyq139
• Fortier, I., Doiron, D., Burton, P., & Raina, P. (2011). Invited commentary: Consolidating data harmonization—How to obtain quality and applicability? American Journal of Epidemiology , 174(3), 261–264. doi: 10.1093/aje/kwr194
• Hendershot, T., Pan, H., Haines, J., Harlan, W. R., Marazita, M. L., McCarty, C. A., … Hamilton, C. M. (2015). Using the PhenX Toolkit to add standard measures to a study. Current Protocols in Human Genetics , 86, 1.21.1–1.21.17. doi: 10.1002/0471142905.hg0121s86
• Maiese, D. R., Hendershot, T. P., Strader, L. C., Wagener, D. K., Hammond, J. A., Huggins, W., … Hamilton, C. M. (2013). PhenX—Establishing a consensus process to select common measures for collaborative research. Research Triangle Park, NC: RTI Press. doi: 10.3768/rtipress.2013.mr.0027.1310
• Riley, W. T., Borja, S. E., Hooper, M. W., Lei, M., Spotts, E. L., Phillips, J. R. W., … Perez-Stable, E. (2020). National Institutes of Health social and behavioral research in response to the SARS-CoV2 pandemic. Translational Behavioral Medicine , 10(4), 857–861. doi: 10.1093/tbm/ibaa075
• U.S. Department of Health and Human Services, National Institutes of Health (HHS/NIH). (2020, July). NIH-wide strategic plan for COVID-19 research. Bethesda, MD: U.S. Government Printing Office. Retrieved from https://www.nih.gov/research-training/medical-research-initiatives/nih-wide-strategic-plan-covid-19-research.
• Windle, G., Bennett, K. M., & Noyes, J. (2011). A methodological review of resilience measurement scales. Health and Quality of Life Outcomes , 9, 8. doi: 10.1186/1477-7525-9-8

Number of times cited according to CrossRef: 8

• Cataia L. Ives, Michelle C. Krzyzanowski, Vanessa J. Marshall, Keith Norris, Myles Cockburn, Keisha Bentley‐Edwards, Dinushika Mohottige, Keshia M. Pollack Porter, Denise Dillard, Yochai Eisenberg, Monik C. Jiménez, Eliseo J. Pérez‐Stable, Nancy L. Jones, Jyoti Dayal, Deborah R. Maiese, David Williams, Tabitha P. Hendershot, Carol M. Hamilton, The PhenX Toolkit: Recommended Measurement Protocols for Social Determinants of Health Research, Current Protocols, 10.1002/cpz1.977, 4 , 3, (2024).
• Lucy W. Kibe, Mohsen Bazargan, Adaobi Bosah, Katrina M. Schrode, Yufu Kuo, Edward Andikrah, Magda Shaheen, Diet Quality of Older African Americans: Impact of Knowledge and Perceived Threat of COVID-19, International Journal of Environmental Research and Public Health, 10.3390/ijerph20075274, 20 , 7, (5274), (2023).
• Tavonia Ekwegh, Sharon Cobb, Edward K. Adinkrah, Roberto Vargas, Lucy W. Kibe, Humberto Sanchez, Joe Waller, Hoorolnesa Ameli, Mohsen Bazargan, Factors Associated with Telehealth Utilization among Older African Americans in South Los Angeles during the COVID-19 Pandemic, International Journal of Environmental Research and Public Health, 10.3390/ijerph20032675, 20 , 3, (2675), (2023).
• Edward K. Adinkrah, Sharon Cobb, Mohsen Bazargan, Delayed Medical Care of Underserved Middle-Aged and Older African Americans with Chronic Disease during COVID-19 Pandemic, Healthcare, 10.3390/healthcare11040595, 11 , 4, (595), (2023).
• Jacqueline H. Becker, Jenny J. Lin, Akosua Twumasi, Ruchir Goswami, Fernando Carnavali, Kimberly Stone, Monica Rivera-Mindt, Minal S. Kale, Georges Naasan, Joanne R. Festa, Juan P. Wisnivesky, Greater executive dysfunction in patients post-COVID-19 compared to those not infected, Brain, Behavior, and Immunity, 10.1016/j.bbi.2023.08.014, 114 , (111-117), (2023).
• Michelle C. Krzyzanowski, Cataia L. Ives, Nancy L. Jones, Barbara Entwisle, Alicia Fernandez, Theresa A. Cullen, William A. Darity, Mark Fossett, Patrick L. Remington, Maile Taualii, Consuelo H. Wilkins, Eliseo J. Pérez-Stable, Nishadi Rajapakse, Nancy Breen, Xinzhi Zhang, Deborah R. Maiese, Tabitha P. Hendershot, Meisha Mandal, Stephen Y. Hwang, Wayne Huggins, Lauren Gridley, Amanda Riley, Erin M. Ramos, Carol M. Hamilton, The PhenX Toolkit: Measurement Protocols for Assessment of Social Determinants of Health, American Journal of Preventive Medicine, 10.1016/j.amepre.2023.03.003, 65 , 3, (534-542), (2023).
• Lotte van Dammen, Tor T. Finseth, Bethany H. McCurdy, Neil P. Barnett, Roselynn A. Conrady, Alexis G. Leach, Andrew F. Deick, Allissa L. Van Steenis, Reece Gardner, Brandon L. Smith, Anita Kay, Elizabeth A. Shirtcliff, Evoking stress reactivity in virtual reality: A systematic review and meta-analysis, Neuroscience & Biobehavioral Reviews, 10.1016/j.neubiorev.2022.104709, 138 , (104709), (2022).
• Rebecca T. Emeny, Kai Zhang, Daisy Goodman, Alka Dev, Terri Lewinson, Kristina Wolff, Carolyn L. Kerrigan, Sally Kraft, Inclusion of Social and Structural Determinants of Health to Advance Understanding of their Influence on the Biology of Chronic Disease, Current Protocols, 10.1002/cpz1.556, 2 , 10, (2022).