Pros and Cons: Observational and Simulation Economic Studies

Over the last several years, our company has been commissioned to carry out studies whose primary purpose is to provide evidence to support reimbursement of a specific healthcare technology (either pharmaceutical or device).  In general, for both the US and abroad, there has been an increase in health economic literature because there has been a greater demand for this type of information from the reimbursement agencies, whether they are public or private.  In our experience, these studies have typically taken one of two forms: simulation modeling or an observational study.  This blog explores some of the trends in terms of publications; as well as the pros and cons of the respective approaches. 

Over the past five years, there has been an increase in publications associated with the health economics in general.  For instance, an advanced search (i.e. All Fields and Date of Publication) of PubMed for “cost” or “cost-utility” from 2007 to 2013 increased from 23,919, to 37,624 (57% increase) and from 151 to 294 (95% increase) [1].  Similarly, for the same time period, there has been an increase in studies associated with the keyword “observational” from 4,915 to 13,622.  Although a crude method of categorizing these types of studies, the approach indicates an increase of 57% for cost studies and an increase of 177% for observational studies which suggests a greater increase in observational studies.

This trend has paralleled some of our own experience over the past few years.  The limitations of cost-effectiveness analyses (i.e. simulation modeling) have been often mentioned [2].   The modeling effort itself depends upon data from previously published studies, and the data quality is an important factor to consider.  While RCTs are generally considered the gold standard for data quality there are other factors to consider when utilizing the efficacy results from RCTs for subsequent economic modeling efforts.  The similarity in study design, completeness and consistency of reporting, and protocol driven resource utilization are all factors to consider when compiling the evidence which will be the foundation of the simulation model. 

Under “good circumstances”, there may be several high quality studies available for each intervention of interest such as the reported outcomes (e.g. treatment success) are very similar for the set of studies.  Definition of the patient populations should be as similar as possible.  Also, if a cost-utility analysis is desired, it is necessary that generic health preference scores be available, or there be a mapping algorithm from a disease specific instrument [3].

However, in our experience, as well as the experience of others, there are many limitations in this type of approach [2].  In many cases, there are significant differences with respect to study design for the studies used to inform the models.  In many cases the QALY information is not ideal.  While it is becoming more common to consider quality of life information earlier in the research process, very often the generic health preferences are not available from the studies which were used to inform the efficacy parameters.  In some cases, it is necessary to map scores from one instrument to another, such as a mapping from the SF-12 to the EQ-5D [4].

While the above discussion covers some of the key limitations associated with economic modeling (i.e. simulation modeling), there are potentially additional limitations.  In summary, there are situations where the data quality is high, and the required data is available, and the similarity between the key studies is high, then the resulting economic model has a high likelihood of being both reasonably accurate and persuasive.  However, more often than not, the actual situation is more of a mixed bag of higher quality data and other data which is not ideal or from small samples.  In these cases, when the full set of assumptions and limitations are considered, the resulting analysis may not be as persuasive as would be desirable.

While observational studies have limitations as well, some of the limitations which are part and parcel of simulation modeling can be overcome.  Since the studies are generally prospective, there is the opportunity to design the study and capture all of the necessary data elements.  Also, since the study is indeed “observational”, the limitations associated with protocol driven resource utilization (i.e. from RCTs) is overcome.  The other factor to consider is that the resulting dataset is an original dataset, as opposed to an amalgamation of data elements from a series of previously published set of studies; this factor increases the interest and publishability of the work. 

In summary, while it is somewhat more expensive and time-consuming to conduct a prospective observational study, the benefits of this type of effort may be well worth it.  The trends in publication numbers indicate that more sponsors are likely choosing to shift from health economic modeling toward observational studies, in order to support the reimbursement of their technologies.

References:

1. http://www.ncbi.nlm.nih.gov/pubmed/
2. Weintraub WS, Cohen DJ.  The limits of cost-effectiveness analysis.  Circulation: Cardiovascular Quality and Outcomes. 2009; 2:55-58.
3. Parker M, Haycox A, Graves J. Estimating the relationship between preference-based generic utility instruments and disease-specific quality-of-life measures in severe chronic constipation: challenges in practice. Pharmacoeconomics. 2011 Aug; 29(8):719-30.
4. Sullivan PW, Ghushchyan V. Mapping the EQ-5D index from the SF-12: US general population preferences in a nationally representative sample. Med Decis Making. 2006 Jul-Aug; 26(4):401-9.