A Societal Outcomes Map for Health Research and Policy*

by Michele S. Garfinkel,^a, * Daniel Sarewitz,^b and Alan L. Porter^c

For copies of article reprints, please contact michele.garfinkel@gmail.com

PDF Version

Keywords: Research policy, science policy, knowledge sharing, societal benefit

*The majority of this work was completed while the first two authors were at Columbia University’s Center for Science, Policy, & Outcomes. The Web site that is the physical manifestation of the societal outcomes map described here was maintained at Columbia University until recently, when the Center moved to Arizona State University. The Web site is currently being migrated to Arizona State and will be available shortly. The site is not integral to the interpretation of the work described here. Illustrations of the information available on the web site is included in this paper, these illustrations are screen shots of the actual web-based output.

^aProject Director, J. Craig Venter Institute, Rockville, MD

^bSenior Research Scholar and Managing Director, Center for Science, Policy, &            Outcomes, Arizona State University

^cProfessor of Engineering and of Public Policy, Georgia Institute of Technology.

Abstract

As a contribution to a more open and knowledgeable policy debate about the roles of science and policy in the health system of the United States, we are using modified roadmapping and technology assessment techniques to construct a prototype societal outcomes map for health research and policy. The map will be a robust and straightforward tool linked to a rigorous and updateable data set; thus, policymakers and stakeholders will be able to comprehensively view outcome-oriented policy options and tradeoffs. Users of such maps will be able to contribute their knowledge to the construction of alternative policy pathways to achieve desired health-related outcomes.

 

1. Introduction
 

The health system of the United States comprises manifold realities: high levels of spending on health care paralleling increasing rates of several diseases; the most-respected research enterprise in the world feeding into a broken public health infrastructure; an insurance net that, despite its size, fails to catch 40 million Americans. These quandaries have been dissected at length in popular and academic venues, resulting in broad and deep agreement that the problems are real, and little agreement as to what to do about them.

To take one relatively simple example, the budget for the Department of Health and Human Services for fiscal year 2004 was over $550 billion; the FY2005 request is nearly $575 billion (DHHS, 2004). This level of funding, nearly synonymous with a comprehensive accounting of overall federal health investment, continues a trend of increases over the past several decades. Nearly 90% of this spending was for Medicare, Medicaid, the State Children’s Health Insurance Program, and other mandated spending. The remaining 11.7% (in FY2004) was not, however, a negligible amount. This $65 billion of discretionary spending included nearly $28 billion (over 40% of all HHS discretionary spending) for the National Institutes of Health; the remaining 60% went to the other HHS offices and agencies to support both research and services (including the Indian Health Service, the Food and Drug Administration and the Centers for Disease Control and Prevention). At the same time, NIH’s appropriation was approximately half of all federal non-defense research and development spending in the United States (AAAS, 2004a and 2004b).

Yet the connections between this research investment and the overall health of Americans are largely unknown. For example, over the last 60 years most gains in average life span and in health as expressed as disability-adjusted life expectancy can be attributed to changes in systems, technologies, and behaviors that have little relation to the nation’s post-World War II investments in basic research in health. Indeed, approximately 25 years of the 30-year gain in life expectancy since 1900 is attributable to advances in public health (CDC, 1999). Among these advances are improved environmental quality, reductions in smoking, and control of infectious diseases. Even those gains in life span that are a result of some kind of basic biomedical advance are at best indirectly related to the bulk of health research effort and spending (McKinlay and McKinlay, 1977; CDC, 1999). Currently, health research policy in the United States does not fully take into account the roles of, for example, public health or behavioral sciences, nor of the powerful social sciences. While basic research (for example, as practiced by the NIH and its grantees) has a place in advancing health, questions are beginning to be raised even in a sympathetic Congress about the imbalance favoring funding for this kind of research over other approaches to health attainment (Southwick, 2001).

It is in this context that the disconnect between the stated missions of the various health agencies, the actual distribution of funding to those agencies, and the reality of some worsening health indicators (such as increasing rates of obesity and diabetes; the uptick in the number of new AIDS cases; an increase in smoking among 18-34 year olds; etc.) becomes apparent. Even in cases where new technologies work, the growing gap between those who benefit from increasingly expensive technologies and those who do not continues to increase. In short, societal health outcomes do not in any straightforward way reflect the level of investment in research. Investigator-initiated research is highly successful by the internal standards of the research community. Specifically, articles published, students trained, and patents filed are easily quantifiable. Because these absolute outputs are large and countable, it is easy to argue that continually increasing funding for basic research is a good investment; these arguments become drivers of public policy, despite the lack of evidence of the connection between this investment and the overall health of Americans.

 
Indeed, Congress has just completed its five-year doubling of the NIH budget. But can such investments, which treat basic research as the driver of medical progress, redress persistent sub-optimal population-level health outcomes? (Vehorn et al., 1982). Specific problems within the system have been discussed at length and in some cases underlying causes are very well understood. The problems of large numbers of Americans without insurance, increasing rates of diabetes, the disturbing phenomenon of re-emerging vaccine-preventable infectious diseases, for example, can be described in detail. There has been only modest success, though, in understanding such problems in the context of desired societal health outcomes, and little effort to link decisions about research approaches to the achievement of these outcomes.

In other sectors of society, where the need to achieve specific outcomes is critical, decision-support tools have been developed and implemented successfully. One of the most effective of these tools, technology roadmaps for research, can clarify and enhance the connections between inputs and outcomes (Sandia National Laboratories, no date). This approach, founded in engineering theory, is thought to be particularly successful in agencies and firms that are focused on security or on technologies that are expensive to develop or that are potentially dangerous, ranging from consumer goods (for example, computers and other electronics) to space exploration technologies. These maps can aid decision making about research and development when the relations bewteen inputs (e.g., money, human resources) and desired outcomes is complex. Thus, the realm of health research and policy seems ripe for a mapping exercise: it is complicated, expensive to maintain, and the cost of inefficiency is high and measurable in morbidity and mortality.

2. Paths to societally-relevant health outcomes
 

Here, we are not attempting to illustrate a research roadmap per se. Our goal is broader: we wish to describe a guide that lays out possible paths, research-driven or not, to achieving societally-desired health outcomes. We are not proposing this approach as a solitary determinant of health policy. Rather, we were inspired by the general characteristics of roadmaps as described by Robert Galvin: “... an extended look at the future of a chosen field of inquiry composed from the collective knowledge and imagination of the brightest drivers of change in that field.... (comprising) statements of theories and trends, the formulation of models, identification of linkages among and within sciences, identification of discontinuities and knowledge voids, and interpretation of investigations and experiments.... (a means to) communicate visions….” (Galvin, 1998). Policymakers currently have few resources that allow them to comprehensively view the multitude of paths leading to a desired outcome; the ability to envision the tradeoffs necessary to achieve a combination of outcomes is even more limited. We thus suggest this map as an addition to the collection of tools that individuals or agencies might use in the process of formulating policy and setting priorities.

One example of the type of map we looked to as a model is the International Technology Roadmap for Semiconductors (commonly known as the Sematech Map). A public/private consortium, Semiconductor Manufacturing Technology (now International SEMATECH) was conceived in 1986 to improve the design (mostly speed) of semiconductors (SEMATECH, 2001). By using an iterative process of surveying for customer needs and inquiring as to what was available and which actors could carry out appropriate research to achieve those desires, foresight maps have been constructed that range from the early versions of a few printed pages to the current CD-ROM version. In all cases, what these maps show are outcomes (what is desired) and inputs (what is needed; for Sematech, this was usually a specific research plan; SEMATECH, 2002; see also Sandia National Laboratories, no date; Steel Industry, 2001). Other public/private enterprises are using this approach (Commission on Engineering and Technical Systems, 2000); private companies, for example, Motorola, have used this technique very successfully (Richey and Grinnell, 2004; Motorola, 2002). These maps frequently extend over long time horizons—ten years or more. Intensive involvement with the intended users can help assure the maps address critical elements (Sandia National Laboratories, 2002; Agriculture Roadmap, 1998).

In contrast to the engineering community, the biomedical research community as a whole has held fast to the notion that the use of such mapping to “pull” research would result in “directed research” or “priority setting” and is therefore repugnant (Bloom, 1998). The fear is that such exercises would somehow fetter research and stifle curiosity and creativity. As Galvin has pointed out, though, “roadmaps are working now in industry and they are beginning to gain a stronghold in science. Just as engineers first scoffed at them, so will some scientists. But who better than scientists to experiment with an experiment that can strengthen sciences’ support and accelerate its generation of knowledge” (Galvin, 1998). Further, “does it make sense to be scientific about everything in our universe except the future course of science?” (Bloom, 1998) The National Science Foundation invokes the Sematech map in its discussions of directions for nanoscale science and engineering (NSF, 2000) and has been generally positive about the experimental use of maps (Galvin, 1998). Priority setting as reflected in exercises such as those from Sematech, Motorola, and the Department of Energy’s Sandia Laboratory can hardly be accused of being fuzzyheaded pursuits aimed at undermining scientific creativity. These experiments are relevant for policy research communities as well; the need for such experiments and for the modified policies that would result has been alluded to by several commentators, but is just beginning to be addressed explicitly (Fischoff, 2000; Garber and Romer, 1996; Gibbons, 1999; Iglehart, 1996).

In the case of health research and policy generally, by having both experts and the public define what “health” is and which societal-level health outcomes are desired, public policies can be structured to help shape an overall health system in all its complexity and diversity. This includes basic research, applied research, prevention techniques, cultural and social conditions, economics, etc. The concept of including “consumer” opinion in the formation of policy, even at the level of individual research projects, should no longer be considered radical. Although solicitation of public opinion is sometimes considered to be in the realm of marketing, several august organizations include public opinion explicitly in their decision-making processes. For example, the Department of Defense’s breast cancer grant process includes representatives of the public on its review panels. The Intergovernmental Panel on Climate Change includes public representatives in its policy discussions.

Thus, the overall analysis approach here is somewhat related to technology assessment exercises (Bloom, 1999; Tijink, 1996; Williamson, 1998); in particular, the Royal Netherlands Academy of Arts and Sciences has proposed to offer a mechanism to measure the effects of applied health research on society as a whole (KNAW, 2002). Our work is related but complementary, and unique in this sense: it is driven by a consideration of societal outcomes in the first instance. In order to enable a more open and knowledgeable policy debate about the roles of various players in the health system, we are initiating a modified foresight mapping technique, incorporating well-understood aspects of technology assessment and coupled to a useful graphical guide. The guide should be usable as a research tool, a policymaking outline, a teaching aid, etc. The map, and the information underlying it, will allow stakeholders in and caretakers of the nation’s health system to view outcome-oriented options and tradeoffs comprehensively. As the map grows, stakeholders and their representatives will also be able to critique and contribute to it.

3. Creation of a guidance tool

3.1. Data collection and dataset management

The availability of a tool to illustrate information in a way that is flexible, easily updateable and expandable, and user-friendly will encourage the use of such analyses for actual policy setting. As a base for our analysis we are assembling data sets that will describe the state of knowledge about a sector of concern, and by their rigor will indicate what is missing (what needs to be known). The information for these data sets is being drawn from a number of sources, and these sources include a good number that have been used successfully in other kinds of technology assessment exercises (Goodman, 1998; Smith, 2001).

The key theoretical and practical features of these roadmapping and technology assessment exercises are beginning to appear in technology policy literature (Peet, 1998; Schaller, 1999; Galvin, 2004; Kostoff et al., 2004). We are using a combination of “push” and “pull” evaluations, coupled with expert input and some computer-driven analysis (Goodman, 1998). For our maps, we identify the key features as the identification of outcomes, identification of necessary (relevant) technologies (specifically, research and policy inputs), identification of possible pathways to achieving the societal outcome, and a plan for developing these technologies; i.e., policy recommendations.

We are assembling multiple large data sets that will be available to all users of the final web-based map. Individual and summary data tables will be publicly available along with the map for comment and for use by others in the community. For this paper, we limited our sources to mission statements, surveys of expert literature, meeting agendas, and expert statements to the popular press to initially define a constellation of societal health outcomes. Further, we looked at major newspapers and magazines (Lexis-Nexis scans) for an approximation of “popular opinion” of desired health outcomes. For long range studies, we anticipate that Delphi surveys for expert opinion and scientific surveys popular opinion will be desirable. For this pilot, however, the information we were able to glean from publicly available sources was sufficient to initiate the maps discussed here. We listed as many specific outcomes as we could gather from the literature, and then in each case chose two or three of them to follow in detail.

For the initial definition of societally-desired outcomes (perinatal health, pain control, etc.) we looked primarily at abstracts of papers from specialty journals and agency mission statements to begin to define desired outcomes. We recognize that to some degree the content of the journals will be linked to agency mission statements as a result of grant processes for federal funding. In order to confirm that we were not limiting the full constellation of desired outcomes by using this approach, we also looked at review articles (which tend to be more explicit at stating what is missing or what is needed whether or not it is fundable) and at expert statements to non-peer reviewed outlets (such as newspapers and magazines). We were also able to look at some publicly-available national meeting agendas, which also tend to address overarching problems. Together, these approaches allowed us to begin to identify societal desires for health research. Content analysis of these sources proved relatively straightforward and we believe we have a good accounting of the majority of expert-defined societal health outcomes dependent on health research or policy. These “high level” outcomes correspond to the dark blue nodes on the maps.

The map will be updated iteratively and interactively in response to criticism from users, both those in health research and policy whose input was overlooked in this first pass, and also individual stakeholders: private citizens, specific disease lobbies, regulators, legislators, etc. In order to take a small accounting of these opinions before the map is posted publicly, we looked at output from non-governmental organizations, especially disease-specific groups. We looked at their respective mission statements, transcripts of congressional testimony where available, and statements to the general press. We also looked for similar statements in testimony or to the press from legislators or other policymakers. At the high level (outcomes nodes) there was no significant difference between the desires of experts and of non-experts; there was some difference in statements of “what is needed” to accomplish these outcome goals (see text).

We then needed to determine exactly how these health outcomes might be achieved, whether society already has the resources to do accomplish this, and where the “white spaces” (opportunities for new research or novel policies) are. In other words, what inputs would be necessary to achieve a specified outcome? This was accomplished by a survey of the review literature in the relevant field, from media statements, and from agency mission statements. In this case, we did not try to approximate user (public) input.

We are using commercial software for information management (Inxight, 2004). By arraying the data in a pre-determined matrix, this proprietary software allows data from a variety of sources (in this case, Excel spreadsheets, but other sources can be used) to be converted to an interactive graphic representation.

For now, we are not attempting to assign weight or any other value to a particular input. We understand that, for example, a single published article and a stated policy do not have the same inherent value. Rather, we are expecting that the “value” of an outcome (as evidenced by how widely it is desired) or of a research or policy input (i.e., does more research need to be done, or is the information already available and needs to be implemented?) will be determined from user commentary, coupled to the expert opinion and literature searches described here.

For the purposes of developing prototype maps, the analyses we are conducting are small enough to do “by hand.” Proceeding backward from the last desired act, i.e., the societal outcome, we will be able to outline the know-how that is already available, and the “white spaces” where more research or other inputs are needed. As the map grows it may be necessary to devise semi-automated analyses accompanied by expert “reality checks.” The data are given physical representation using information-management software. For the prototype map discussed here, we are only presenting two high level outcomes, or “nodes.” A complete collection of such nodes would describe the universe of societal-level health outcomes; our interpretation of these nodes will appear on our Web site, will be modified by information received from users, and will be combined with data from any other groups that might take on such a mapping project.

3.2. Flow of the inquiry
 

1. Identification of desired societal outcomes. The first step in industrial roadmapping processes such as Sematech is to exhaustively describe the product(s) desired by consumers (be they individual consumers or corporate consumers). For our map, we are compiling a list of desired societal outcomes from the realm of health research and health policy. Because societal health outcomes encompass such a large number of possibilities (anything on a gradient from, for example, “a long and vital life” to specific characteristics of drugs), we are initially breaking the map into nodes of fairly high order: perinatal health, pain management, infectious disease control, cancer control, etc. Experts are consulted initially to define the parameters (research, policy, etc.) needed to achieve each outcome. For the purposes of this pilot study, we used mission statements of federal, non-governmental, and private research and policy groups; surveys of expert literature; major meeting agendas; and expert statements to the popular press. As the map grows, Delphi surveys (structured expert elicitation) can be used to understand less-accessible niches of health research and policy. As we receive input from the public and from workers in a variety of health-related fields, we will define new nodes to explore. Ultimately, we will be able to assign quantities of improvement as disability-adjusted life-years. Measuring health in terms of DALYs (or a similar scale), while not perfect, is generally accepted as a reasonable standard. The map will be flexible enough to accommodate other interpretations of “health” as the need arises.

2. Identification of research and policy inputs. The next step is to determine what factors are necessary to achieve the desired health outcomes, and whether or not these factors are available. All possible approaches are considered and the “state of know-how” for each of them is evaluated. From this evaluation, lists of “what is known” and “what needs to be known” can be generated. This is done through literature searches to evaluate primary sources, secondary (review) source evaluations, and some exploration of statements made by researchers and policymakers to the media. Reviews and statements to the media seem to be especially useful for defining what is not yet known. In some cases we are using a Technology Opportunities Analysis to supplement our qualitative analyses. This overview provides one measure of the “sense of the community” as to how realistic an achievement of that outcome might be, given available inputs.

3. Identification of appropriate research and policy paths. In a world of unlimited resources, every possible path could be the subject of a policy experiment. In a world of homogeneous human beings, one policy could be scripted from this accumulation of knowledge that would make everyone healthy. Neither of these worlds exists. Thus, we need to choose research and policy approaches carefully, and with an eye to tradeoffs that will need to be made. By explicitly illustrating alternative paths toward an outcome, mapping can help inform such tradeoffs. At this stage, we will begin to identify alternative paths that might be taken to achieve identical or similar results. However, policy creation for societal health outcomes will differ from policy for technology innovation generally. The politicization and personalization of health is striking. As far as we know, there has never been a gathering on the National Mall to protest the slow speed of computer chips. There have been innumerable gatherings, though, noting the perceived slow research on or lack of attention paid to specific diseases. Individual citizens/consumers increasingly demand input into health policy priorities, directly and via their elected representatives.

From this perspective, the democratizing potential of a World Wide Web-based mapping tool becomes manifest. In addition to outlining policy recommendations based on the results of mapping projects, those constructing maps can solicit public opinion via the Web (and other outlets) to determine if policy recommendations are reasonable in light of public understanding of health, and so that health outcomes that have been overlooked can be considered as well. As a proxy for public opinion in this study, we have looked at statements made by individuals in various media (for example, relatives of people with specific diseases, statements from legislators, etc.). We are using a commercial data management system created by Inxight Software, Inc.(Inxight, 2004) to organize possible pathways into easily viewed maps that will be available on the Web (animated and annotated) as well as in printed form.

4. Development of policy recommendations. In roadmapping for technology development, one approach is eventually chosen to develop the relevant technologies. That decision is based on a combination of the strengths of the companies involved, standard management theories, and, frequently, an executive decision to organize the actors. Here, it might be possible (and in some cases, preferable) to chose one policy recommendation from the several that will be identified from our mapping. However, we presume that one of the strengths of this kind of tool is its flexibility and usefulness for diverse stakeholders, in support both of informed public discourse and actual decision making. A number of recommendations could be made for the achievement of each societal health outcome; each recommendation could have advantages for different bodies, and could inform higher-level resource allocation strategies. While the targets of the initial analyses are macro-level outcomes, an iterative analysis could easily be used by bodies concerned with health outcomes from the community- to the global level. In all cases, social equity and the social purposes of research and policy can be addressed as appropriate or needed for the purpose of the user. We have chosen two areas for pilot studies. First, we looked at the general area of perinatal health. We chose this as an example of an outcome that is universally understood to be of high priority, influences both the quality and quantity of life, is well studied and understood, and yet is still not a guarantee in the contemporary American health system. Then, we explored pain as an example of a condition that clearly reduces the quality of life, and which is just beginning to engage both researchers and policymakers. Here, we show small sections of the analysis to illustrate its value and versatility.

4. Specific Examples
 

We looked at two specific outcomes, “perinatal health” and “pain management.” By scouring the professional literature, we were able to define a number of lower-level outcomes that contribute to the achievement of the higher-level outcome. For example, “healthy (genetic) development” and “healthy birth weight” are necessary for perinatal health, as defined in the expert literature.

4.1.1. Identification of desired societal outcomes.

Perinatal health is generally accepted as setting the stage for subsequent robust growth and development of the child, and is echoed in adult health as well. Although the definitions vary, we limited our analysis to the period of 28 weeks pregnancy to 7 days after birth (DHHS, 2000). In simplest terms, “healthy babies” is the goal. But achieving even this apparently straightforward and universally valued objective has been mired in confusion and controversy. Why do such large disparities in maternal mortality between black and white women, and in their children, still exist? Is it possible or desirable to deliver prenatal care to those mothers who seem not to want it? These can be overwhelming, even paralyzing problems for setting policy and suggesting research directions. We carried out a detailed analysis to determine if we could identify a set of policy and research paths that could result in improved indicators for perinatal health.

4.1.2. Identification of research and policy inputs.

We first did an inspection to determine as comprehensively as possible recent research and policy approaches to perinatal health (“what is known,” or nearly so), and as well by expert statements (mission statements, statements to media, etc.) what knowledge seems to be “missing,” if any. Again, we recognize that any examination, be it of experts, literature, consumers, mass media, etc., will itself have gaps. By looking as widely as possible, most of these gaps should be reduced, if not eliminated. As the map evolves through iterative incorporation of data, opinions, and “field notes” from users, we will be able to identify real strengths of any research or policy approach, and will be able to be sure of our identification of gaps that need to be closed through research and policy. We identified several possible inputs (illustrated in Figure 1). A few of these are noted in detail here.

The Women, Infants, and Children Supplemental Feeding Program (WIC) is known to improve health outcomes for newborns (and later, developmental outcomes of children; DHHS, 2000; USDA, 2002a). Specifically, its concerns (outcomes) include healthy birth weight, subsequent healthy growth, and disease prevention. This is a successful program in terms of coverage and outcomes, yet it is known that many of those eligible are not receiving these benefits (USDA, 2002b). For pregnant mothers who are malnourished, this can result in low birth weight infants who may suffer other health problems later that may need uncomfortable, invasive, or expensive interventions. There is a large professional literature on the topic and much discussion in the general press. Thus, much is “known” about this input. Further, based on literature from governmental and non-governmental workers, it appears that this is an approach that is relatively efficient (intervention yields a large benefit), and probably functions to reduce recognized health disparities. But there is an apparent lack of knowledge as to why all those who are eligible do not receive benefits. Do they not know of the existence of the program? Do they not want to participate? Do they want to participate but feel constrained from doing so? This indicates a significant research opportunity that is supported by a good deal of knowledge and the opportunity to increase the just and equitable distribution of such knowledge.

 We recognized the understanding and use of principles of developmental genetics as another potential input to assuring perinatal health. This is an area that is of both ancient (the recognition of inherited disease) and recent interest (the human genome project). Several issues in healthy perinatal development link directly to an individual’s genomic content. The key effort in perinatal genetics now is at the level of newborn screening for known diseases. Several of these tests are accepted as being critical for assuring subsequent health (the most well-known example being the screen for the metabolic disorder phenylketonuria), although in several states screens are conducted for disorders that have less severe impacts. Should effort be put into expanding this screening to diseases that are not proximally lethal (or may never be lethal) such as hemochromatosis? Or perhaps effort (and funds) should be exerted to assure that tests that are already available are accessible to every newborn: currently, the number of tests varies by state. This is an example of an area that developed somewhat ad hoc and where “what is known” (in terms of which tests might yield the best outcomes) is quite fragmented. Anecdotal evidence indicates that a better distribution of these tests would lead to more equitable (disparity-reducing) outcomes, but there is little research to show how. Thus, this is an area ripe both for research (which tests are most useful?) and policy (how does society assure that all newborns receive them?).

At the same time, the National Institute for Child Health and Human Development devotes a significant portion of its budget justification to a discussion of the need to understand the genetic underpinnings of various developmental abnormalities or disorders. “What is known” in this case is relatively restricted. The March of Dimes estimates that 150,000 babies with birth defects are born in the United States each year, reflecting an astounding 4,000 known unique defects (March of Dimes, 1997, 1999). These defects as a group are responsible for 20% of first-year mortality (around 8000 deaths) (CDC, 2004). This number includes a number of abnormalities that are preventable, or whose severity can be lessened with good prenatal care (e.g., control of a diabetic condition, folic acid supplementation, chickenpox vaccinations for mothers who did not previously have the disease, etc.). A few birth defects have a known genetic lesion (e.g., trisomy 21, the cause of Down Syndrome). Still, 70% of all cases have no apparent cause. The largest single group of birth defects, heart defects, afflicts 1 out of every 125 newborns, yet there is virtually no understanding of why. In this case particularly, and for complex diseases generally, even if the underlying causes are “genetic” it is likely that they will be so in a complex way that will also invoke environmental factors. This is not to say that the research should not be done, nor to say that it is not a serious problem. It is rather to note that possible research directions are unclear: should effort be concentrated on identifying and characterizing relevant genes? Would a better understanding of environmental factors lead to a decrease in heart defects? Ideally, these would be studied in concert. Although there is some “cross-talk” between agencies (e.g., HHS and EPA), it is worth noting that even the National Institutes of Health’s environmental health sciences institute (NIEHS) is located physically away (in North Carolina) from the NIH main campus. This is just one issue in a laundry list that could be considered in crafting research and policy directions for this portion of perinatal health. This appears on the map as the large arrays of research directions with a fairly large “white space” where policy needs to be explicitly considered.

Because the area of perinatal health is large, complex, and historically of long-standing importance, we carried out a Technology Opportunities Analysis (TPAC, no date; Porter et al., 2002). In contrast to the routine literature assessments that we carried out, TOA uses commercial software (VantagePoint, no date) to tabulate and explore patterns and associations within the abstracts retrieved electronically (in our example, from PubMed/Medline). The resulting “research profile” includes sophisticated analyses of the relationships between, for example, research activities, workers, geographic clusters of research, etc (Porter et al., 2002).

Using PubMed (NLM, 2004) as our initial data set, we were able to describe a general topography of perinatal-related research. First, we note that the number of articles containing the MeSH (Medical Subject Headings; NLM, 2004) phrases related to perinatal health is relatively small (ranging from 137 to 295 per year from 1992 to 2001; a total of just over 2000 articles through the beginning of 2002), and rises slowly but steadily over this time. At least in the period 1992-2001 where we focused, research is overwhelmingly academic (947 publications; cf. 22 corporate. We recognize that this does not account for the original source of funding). Further, the research was concentrated in the United States and England (1234 of the nearly 2000 articles originated in those two countries). Most interestingly, we found that of 5661 unique MeSH headings that occurred in these publications, only the first 300 have more than 10 mentions. Compared to, for example, general engineering literature (which frequently yields “integrated” or “hard” opportunities maps), this is “fragmented” (or “squishy”) knowledge. This echoes the finding above of the fragmented nature of research in perinatology; the TOA analysis thus confirms an emergent property of the map. This state of the knowledge is neither good nor bad, but is worth noting in terms of constructing a research policy: uncertainty about connections between research choices and particular outcomes is likely to remain high for some time.

While we recognize literature-based analyses for the most part cannot account for research with “negative” results and that it is in this case limited by the MeSH thesaurus, we believe that it is a useful accounting of the general types of research that are found to be worthwhile by that particular research community. It thus points to a number of sub-fields within that research area that are underrepresented in research effort (and likely, in research dollars).

4.1.3. Identification of relevant research and policy paths.

When presented in the context of a World Wide Web page, the nodes (boxes) represented in Figure 1 (Panels A-D) become live links that can connect to another Web page, to the original dataset, and to any additional annotations that could be useful to the user. Paths are easily visualized as the user can “pull” subsequent nodes from an initial node (that is, the connection between any two nodes or group of nodes can be visualized). This interactive form, however, is not necessary for information to be gleaned from the map. Here, we show static versions of two prototype maps; by showing the static maps scanned from left to right, the paths are apparent as well. In general, in moving from left to right (from Panel A to Panel D), desired outcomes are represented from general to specific levels, and eventually connect to the inputs required to achieve the outcomes. Thus, for example, in the initial map, the top-level outcome, “Health” (as indicated by relative quality- or disability-adjusted life years) has lower level outcomes that feed in: Perinatal Health, Pain Management, Cancer Control, Infectious Disease Control, etc. In the Perinatal Health example under discussion here, two specific outcomes, “healthy birth weight” and “healthy development” (proximally, an outcome of the lack of birth defects) were followed in detail. Outcomes are indicated with dark (black) link lines, inputs or sources of inputs (agencies, etc.) with light (gray) link lines. Although the high level outcomes shown here are represented as different maps it should be kept in mind that one global could comprise all of these outcomes and would be navigable as a whole.

Figure 1 illustrates the nodes that were identified by the type of analysis described above (panels A-D show the map from left to right). In addition to the WIC, metabolic testing, and genomics inputs mentioned, several other inputs were identified. By coupling these with the list of outcomes (perinatal health, healthy birth weight, healthy development, etc.) we can outline a number of research and policy combinations that could effect these outcomes. For example, the WIC program seems to improve the chances of both healthier born babies, and of subsequent development. Further, it improves the distribution of vaccines to pre-school age children. So by increasing an investment in WIC, both the immediate perinatal health outcome and a seemingly unrelated societal health outcome, infectious disease control, could be improved. This investment would be both in policy (service) and in research (understanding why many who are eligible do not receive the benefit). Alternatively, a biomedical research path could be taken. Understanding the mechanisms underlying birth defects or low birth weight could lead to a medical solution for these problems. Another path would be essentially a legal one: Public Law 105-168 was meant specifically to establish federal infrastructure to “prevent birth defects.” The law mandates the CDC to collect and analyze data, operate regional centers, and inform and educate the public. Several non-governmental entities have interests in this law as well. From a policy perspective, it might be worth having the provision of services clearly defined, or it might be the case that this overlap and fuzziness is desirable. This would require discussion among policymakers, health care workers, scientists, and the public. Many arenas for such a discussion are imaginable. We propose that at a minimum, the interactive nature of the map would allow us to begin to collect research and policy anecdotes and data that would be used to refine policy recommendations.

4.1.4. Policy recommendations.

Specific policy recommendations will then vary depending on the needs of the organization. Based on this analysis, access to programs is a key factor. Biochemical and molecular explorations of genetic developmental diseases has yielded large amounts of data, but some states are not able even to carry out the most basic newborn screening tests right now. Thus, for example, at the highest levels (as a recommendation for federal funding for research, and for federal policies) more research on the non-participation of women eligible for WIC, some research on the value of screening newborns for specific diseases, and continued funding (but perhaps ratcheted down somewhat) via NICHD might be warranted. At a state level, more value might be derived from direct health care worker intervention with WIC-eligible women. In those states where the number of newborn screening tests is minimal, more money might be put into that sphere. In either case, the cost of such programs would be subject to tradeoff: either through other programs judged to be less effective, through higher taxes, etc. More important for our purposes, decision makers can develop a more comprehensive view of the relations between factors that contribute to a vitally important social outcome—perinatal health.

4.2. Pain management
 

The management and prevention of pain is one of medicine’s most important objectives and one of its largest challenges. Decades of descriptive work concerned with the social implications of acute, chronic, and disease-specific pain speak to poor performance on the part of doctors, researchers, and policymakers. Only recently has a questioning of basic principles (e.g., that pain sufferers would necessarily become addicted to painkillers; or even more radically, that certain kinds of addiction during severe pain treatment are necessarily bad) begun in a public manner. Part of this opening has been driven by activists, particularly those concerned with the reduction in quality of life due to cancer pain; part has been driven by the growing realization that pain may interfere actively with healing processes. Thus, this is a significant medical and economic problem. We carried out an analysis of the research and policy aspects of pain management. To illustrate the flexibility of the mapping process for specific applications, here we looked at inputs pooled by type, and quantified these by numbers of papers available in PubMed. We generated the map shown in Figure 2.

This relatively simple map potentially indicates a well-focused problem (although by no means implying an “easy” solution). In this figure, the colored bars at the bottom of the node indicate relative current efforts. For example, the “Treatment” nodes reflect that about 700 entries in the entire PubMed database (1966-present) are studies of the undertreatment of pain or the failure of pain treatments. Nearly 5000 describe the cell biology, molecular biology, or biochemistry of pain processes. In other words, basic biomedical research is the dominant approach to the desired outcome of pain management, at least for the specific outcomes considered here (it is worth noting that clinical research for cancer pain control has expanded greatly in just the last several years; this appears to be a result, at least in part, of the actions of individuals and non-governmental activists). While it is difficult to compare the relative value of each kind of research as reflected in a single published paper, this kind of comparison does highlight the comparative underrepresentation of certain kinds of research (i.e., clinical) in a research framework that emphasizes basic research.

Several potential policies are identifiable in this analysis: more money to clinical rather than basic aspects of pain (which has the political advantage of moving money within NIH rather than between agencies), more aggressive treatment of pain (which might require some law or at least guidance to protect doctors who might then be perceived as overtreating pain), and some continuation of basic research, perhaps particularly on long-term prevention (particularly relevant for back pain) and to some degree on short term prevention (e.g., preventing cancer pain from ever starting).

5. Discussion and broad policy implications
 

Health research policy in the United States is supply oriented. The single largest policy consideration for federal agencies is the amount of funding, and success for the most part is measured by year-to-year funding increases; that is, by inputs. Some accounting based on quantifiable outputs (such as numbers of papers or patents) is taken into account as well. Still, evidence that these and related approaches to policymaking result in improved outcomes, for individuals or for society as a whole, is scant at best. We have proposed here one component of a policy system that can explicitly include outcomes-driven research and policy. This does not assume a need to change fundamentally the current systems of health policy or research policy. Rather, it expands mechanisms for governance of health research policy generally.

This approach to health research policy should be acceptable even to the strongest adherents of the outmoded but classical ideal of the linear model of research, leading inexorably from “basic research” to “applied research” to product or process. The result of our approach to health research policy would be change at the institutional, system, or governmental levels. Changes are not directed at individual researchers; laboratories would not be directed to conduct particular research programs—this level of activity is too small-scale to appear on the map. Rather, decision makers will be able to view synoptically the complex, interconnected, multiple pathways that lead to a desired outcome. This, in turn, will reveal unexamined cause-and-effect assumptions, and previously unrecognized connections and synergies. While a map cannot, of course, ensure better decisions and enhanced outcomes, the lack of such a tool is assuredly and obstacle to those desirable ends.

This is not to say that such maps will not engender controversy. For example, our analysis does not indicate a priori any particular relation between basic research and enhanced health outcomes—a controversial notion. Our analysis of pain management, for example, suggests that the fundamental causes of pain may be “over-researched” relative to other aspects of the problem. On the other hand, inspection of the general areas that will be mapped as part of this project indicates that many problems need enormous amounts of basic research before the structuring of useful polices can take place; for example, many tropical diseases have been vastly understudied at the biochemical and clinical levels. At the same time, though, a map may reveal that new policies for addressing tropical diseases can be emplaced based on what is already understood and where no further research is necessary. Our preliminary analyses do indicate that the universe we are looking at is fairly tightly circumscribed: not only are resources (the pools of talent, space, and money) limited, the ways they can be used are restricted. Specifically, democratic processes have resulted in funding and regulatory processes that buffer the rate of policy change. Thus, one particularly useful application for these maps is likely to be in identifying areas ripe for policy experimentation as a first step toward broader change.

Although economic considerations must occupy a good deal of the effort expended in framing health and health research policies, these cannot be the sole consideration, nor in many cases should they be the primary consideration. Virtually all prioritization documents, particularly those concerned with health-related technologies, mention cost savings as a driver for research and development (for example, NIGMS, 1997). In some cases such savings may ensue; in others, not. Even public health approaches, which are perceived as being inexpensive to implement (“just do it:” eat less, exercise more, use a seat belt, get children vaccinated) may in fact be costly to realize (Rettig, 1994). In the same vein, as a society we may want to prevent certain diseases in the earliest stages via expensive technologies, if these technologies work and even if they might be more expensive than downstream treatment. Workers in public health and preventative medicine can attest to the power of such treatments, though, regardless of cost. Again, the goal here is to enable informed and open discussions about how to best achieve health for the greatest number of people.

The map will be most pertinent when diverse stakeholders can contribute to and extract lessons and information from it. As has been seen with the experience of the World Health Organizations report on health systems, the circulation and subsequent critique of new frameworks for assessing health outcomes is crucial for producing a more useful product (WHO, 2000; Blendon et al., 2001; Murray et al., 2001). By making the map publicly available (both interactively through the World Wide Web and through distribution of semi-customized products such as brochures for users with specific needs in distinct health policy areas) and updateable, we hope to involve citizens and their representatives in defining and achieving desired societal health outcomes. We also hope this stimulates other groups to develop and their own maps and mapping methodologies, and to contribute these to the health research policy community.

Acknowledgements

We thank Karen Antman, Susan Cozzens, Annetine Gelijns, Richard Nelson, and Bhaven Sampat for valuable discussions; and Ramana Rao for initial thoughts on the StarTree software. This work was supported in part by a grant from the V. Kann Rasmussen Foundation.

References

See PDF version
 

Figures
 

Figure 1: Outcomes map for perinatal health research and policy. The map for perinatal outcomes is shown from top to bottom in panels A-D as a sector of the overall map for societal health outcomes (centrally represented as quality-adjusted life years, surrounded by high level outcomes such as cancer control, infectious disease control, etc.), moving essentially from left to right. High- and middle level outcomes are shown by black link lines; inputs are shown with gray (light) link lines. Many factors a re known or thought to contribute to perinatal health; here, we indicate only three: healthy development; healthy birth weight of the baby; and several “unknowns” indicated here under the rubric “research” to indicate where more needs to be known. Note the wide variety of contributory inputs that may impact the desired outcome. In Figure 1D, “Health (QALY increase)” appears as a filled red square, “Perinatal Health” appears as an open red square.

Abbreviations and acronyms:

QALY:  Quality-adjusted life years

WIC:  Special Supplemental Nutrition Program for Women, Infants, and Children

Public Law 105-168:  The Birth Defects Prevention Act of 1998

PHS: Public Health Service

PHPPO:  Public Health Practice Program Office

ACoObstet:  American College of Obstetricians and Gynecologists

AAPediatricians: American Association of Pediatricians

CDC: Centers for Disease Control and Prevention

NIH: National Institutes of Health

ONDCP:  Office of National Drug Control Policy

Figure 1A

Figure 2B

Top

Middle

Bottom