Skip to main content

Data in the MMWR: A purposive review with recommendations

7 Additional practices: How else might MMWR modernize?

This section recommends ways to enrich the resources available to readers outside of published reports, such as additional technical details, supporting data, and analytic code, while further enhancing focus in the report itself.

7.1 Intensive exposition of methods

Principle: Sophisticated, unfamiliar, or unconventional methods should be presented in balance with other detail while supporting focus on the public health issue and impact.

Observations: Some reports have extensive exposition on methods, either in the body of the report or in footnotes. Other reports might benefit from additional details, including reports mm6802a1 (García et al. 2019), mm6935a2 (Moreland et al. 2020), mm7001a4 (Leidner et al. 2021), mm7006e2 (Joo et al. 2021), mm7010e3 (Guy et al. 2021), mm7010e4 (Kompaniyets et al. 2021), mm7032e3 (Moline et al. 2021), and mm7110e1 (Donovan et al. 2022).

Recommendations:

  1. MMWR should allow for technical supplements, just as supplemental tables and figures are supported. In addition, MMWR should develop guidance on how to ensure that reports that warrant technical supplementation maintain proper focus and scope within the body of the report. Alternatively, MMWR could develop guidance for authors to place methods in a technical report posted to a public resource besides CDC Stacks, such as prepublication archives.

7.2 Incorporate less familiar methods

Principle: Public health practice can and should gather information and learn through the full variety of available data types and analytic methods, whether familiar or unfamiliar. Examples of ascendent sources include electronic health records and wastewater. Methods with potential to enrich public health findings within the MMWR include, for example, economic methods and prevention effectiveness as well as machine learning, especially as applied to textual and image-based data. In all cases, data uses in MMWR should be rigorous and focus on public health relevance.

Observations:

  1. Several reports analyze data from electronic health record services or vendors, including athenahealth (mm6802a1 (García et al. 2019)) and Cerner (mm7212e1 (Chun et al. 2023)), as well as systems fed by electronic health records or laboratory records as part of networks or programs (such as PCORnet and the National Syndromic Surveillance Program), state systems, and hospital systems.
  2. Report mm6935a2 (Moreland et al. 2020) analyzed anonymized location data from mobile devices.
  3. None of the reports reviewed here include economic analysis, but a few published reports do, including mm6646a3 (Marx et al. 2017), mm7015a1 (Luo et al. 2021), mm7048a1 (Peterson, Miller, et al. 2021), and mm7048a2 (Peterson, Luo, et al. 2021).
  4. None of the reports reviewed here include machine learning as applied to text or image data, but a few published reports mention machine learning (e.g., mm7044e1 (Bozio et al. 2021)) or cite references pertaining to machine learning (e.g., mm7049a3 (Miller et al. 2021)).

Recommendations: Stimulate and cultivate a richer set of methods, perhaps through focused solicitation with guest editors, keeping tight focus on public health relevance and credibility.

7.3 Open data and analytic code

Principles: The Executive Branch has encouraged making data publicly available since 2013 and has signaled that federally supported publications will also be expected to make associated data and analytic code available in tandem with peer-reviewed publications. In addition, academia, government, and even the commercial sector have been advancing access to analytic code through public repositories, and many peer-reviewed journals require a statement regarding availability. (White House Office of Science and Technology Policy 2022)

Observation: While many reports cite publicly available data sources used within those respective reports, there is no current practice for sharing or describing availability of data or analytic code.

Recommendations:

  1. Require a brief statement on availability of data and analytic code, including whether items are unavailable, available with restriction, or available without restriction.
  2. Develop boilerplate text to describe common modes of availability.
  3. Allow and encourage publicly available data, or data available through a specific, stable request process; data might be limited to machine-readable elements sufficient to reconstruct published and supplemental tables and figures.
  4. Allow and encourage publicly available analytic code, or code available through a specific, stable request process, usually posted on a repository like GitHub.

References

Bozio CH, Grannis SJ, Naleway AL, Ong TC, Butterfield KA, DeSilva MB, Natarajan K, Yang D-H, Rao S, Klein NP, et al. 2021-11. Laboratory-confirmed COVID-19 among adults hospitalized with COVID-19–like illness with infection-induced or mRNA vaccine-induced SARS-CoV-2 immunity — nine states, January–September 2021. MMWR Morb. Mortal. Wkly. Rep. 70(44):1539–1544. https://doi.org/10.15585/mmwr.mm7044e1
Chun HM, Dirlikov E, Cox MH, Sherlock MW, Obeng-Aduasare Y, Sato K, Voetsch AC, Ater AD, Romano ER, Tomlinson H, et al. 2023-03. Vital Signs : Progress toward eliminating HIV as a global public health threat through scale-up of antiretroviral therapy and health system strengthening supported by the U.S. President’s emergency plan for AIDS relief — worldwide, 2004–2022. MMWR Morb. Mortal. Wkly. Rep. 72(12):317–324. https://doi.org/10.15585/mmwr.mm7212e1
Donovan CV, Rose C, Lewis KN, Vang K, Stanley N, Motley M, Brown CC, Gray FJ, Thompson JW, Amick BC, et al. 2022-03. SARS-CoV-2 incidence in K–12 school districts with mask-required versus mask-optional policies — Arkansas, August–October 2021. MMWR Morb. Mortal. Wkly. Rep. 71(10):384–389. https://doi.org/10.15585/mmwr.mm7110e1
García MC, Heilig CM, Lee SH, Faul M, Guy G, Iademarco MF, Hempstead K, Raymond D, Gray J. 2019-01. Opioid prescribing rates in nonmetropolitan and metropolitan counties among primary care providers using an electronic health record system — United States, 2014–2017. MMWR Morb. Mortal. Wkly. Rep. 68(2):25–30. https://doi.org/10.15585/mmwr.mm6802a1
Guy GP, Lee FC, Sunshine G, McCord R, Howard-Williams M, Kompaniyets L, Dunphy C, Gakh M, Weber R, Sauber-Schatz E, et al. 2021-03. Association of state-issued mask mandates and allowing on-premises restaurant dining with county-level COVID-19 case and death growth rates — United States, March 1–December 31, 2020. MMWR Morb. Mortal. Wkly. Rep. 70(10):350–354. https://doi.org/10.15585/mmwr.mm7010e3
Joo H, Miller GF, Sunshine G, Gakh M, Pike J, Havers FP, Kim L, Weber R, Dugmeoglu S, Watson C, Coronado F. 2021-02. Decline in COVID-19 hospitalization growth rates associated with statewide mask mandates — 10 states, March–October 2020. MMWR Morb. Mortal. Wkly. Rep. 70(6):212–216. https://doi.org/10.15585/mmwr.mm7006e2
Kompaniyets L, Goodman AB, Belay B, Freedman DS, Sucosky MS, Lange SJ, Gundlapalli AV, Boehmer TK, Blanck HM. 2021-03. Body mass index and risk for COVID-19–related hospitalization, intensive care unit admission, invasive mechanical ventilation, and death — United States, March–December 2020. MMWR Morb. Mortal. Wkly. Rep. 70(10):355–361. https://doi.org/10.15585/mmwr.mm7010e4
Leidner AJ, Barry V, Bowen VB, Silver R, Musial T, Kang GJ, Ritchey MD, Fletcher K, Barrios L, Pevzner E. 2021-01. Opening of large institutions of higher education and county-level COVID-19 incidence — United States, July 6–September 17, 2020. MMWR Morb. Mortal. Wkly. Rep. 70(1):14–19. https://doi.org/10.15585/mmwr.mm7001a4
Luo F, Li M, Florence C. 2021-04. State-level economic costs of opioid use disorder and fatal opioid overdose — United States, 2017. MMWR Morb. Mortal. Wkly. Rep. 70(15):541–546. https://doi.org/10.15585/mmwr.mm7015a1
Marx GE, Chase J, Jasperse J, Stinson K, McDonald CE, Runfola JK, Jaskunas J, Hite D, Barnes M, Askenazi M, Albanese B. 2017-11. Public health economic burden associated with two single measles case investigations — Colorado, 2016–2017. MMWR Morb. Mortal. Wkly. Rep. 66(46):1272–1275. https://doi.org/10.15585/mmwr.mm6646a3
Miller MF, Shi M, Motsinger-Reif A, Weinberg CR, Miller JD, Nichols E. 2021-12. Community-based testing sites for SARS-CoV-2 — United States, March 2020–November 2021. MMWR Morb. Mortal. Wkly. Rep. 70(49):1706–1711. https://doi.org/10.15585/mmwr.mm7049a3
Moline HL, Whitaker M, Deng L, Rhodes JC, Milucky J, Pham H, Patel K, Anglin O, Reingold A, Chai SJ, et al. 2021-08. Effectiveness of COVID-19 vaccines in preventing hospitalization among adults aged ≥65 years — COVID-NET, 13 states, February–April 2021. MMWR Morb. Mortal. Wkly. Rep. 70(32):1088–1093. https://doi.org/10.15585/mmwr.mm7032e3
Moreland A, Herlihy C, Tynan MA, Sunshine G, McCord RF, Hilton C, Poovey J, Werner AK, Jones CD, Fulmer EB, et al. 2020-09. Timing of state and territorial COVID-19 stay-at-home orders and changes in population movement — United States, March 1–May 31, 2020. MMWR Morb. Mortal. Wkly. Rep. 69(35):1198–1203. https://doi.org/10.15585/mmwr.mm6935a2
Peterson C, Luo F, Florence C. 2021-12. State-level economic costs of fatal injuries — United States, 2019. MMWR Morb. Mortal. Wkly. Rep. 70(48):1660–1663. https://doi.org/10.15585/mmwr.mm7048a2
Peterson C, Miller GF, Barnett SBL, Florence C. 2021-12. Economic cost of injury — United States, 2019. MMWR Morb. Mortal. Wkly. Rep. 70(48):1655–1659. https://doi.org/10.15585/mmwr.mm7048a1
White House Office of Science and Technology Policy. 2022-08-25. Ensuring free, immediate, and equitable access to federally funded research. Executive Office of the President of the United States. https://bidenwhitehouse.archives.gov/wp-content/uploads/2022/08/08-2022-OSTP-Public-Access-Memo.pdf