Published

2022-07-14

Measuring Individual Benefits of Medical Treatments Using Longitudinal Hospital Data with Non-Ignorable Missing Responses Caused by Patient Discharge: Application to the Study of Benefits of Pain Management Post Spinal Fusion

Medición de los beneficios individuales de tratamientos médicos a partir de datos hospitalarios longitudinales con respuestas faltantes no ignorables causadas por la alta del paciente: Aplicación al estudio de los beneficios del tratamiento contra el dolor después de una fusión espinal

DOI:

https://doi.org/10.15446/rce.v45n2.101597

Keywords:

Electronic health records, Empirical Bayesian prediction, Joint mixed models, Non-ignorable missing data, Observational data, Random effects (en)
Datos faltantes no ignorables, Datos observacionales, Efectos aleatorios, Modelos mixtos conjuntos, Predicción Bayesiana empírica, Registros de salud electrónicos (es)

Downloads

Authors

  • Francisco J. Diaz Department of Biostatistics, The University of Kansas Medical Center, Kansas City, KS, United States http://orcid.org/0000-0003-4090-228X
  • Xuan Zhang Boston Strategic Partners
  • Nikos Pantazis National and Kapodistrian University of Athens, Medical School
  • Jose de Leon University of Kentucky

Electronic health records (EHR) provide valuable resources for longitudinal studies and understanding risk factors associated with poor clinical outcomes. However, they may not contain complete follow-ups, and the missing data may not be at random since hospital discharge may depend in part on expected but unrecorded clinical outcomes that occur after patient discharge. These non-ignorable missing data requires appropriate analysis methods. Here, we are interested in measuring and analyzing individual treatment benefits of medical treatments in patients recorded in EHR databases. We present a method for predicting individual benefits that handles non-ignorable missingness due to hospital discharge. The longitudinal clinical outcome of interest is modeled simultaneously with the hospital length of stay using a joint mixed-effects model, and individual benefits are predicted through a frequentist approach: the empirical Bayesian approach. We illustrate our approach by assessing individual pain management benefits to patients who underwent spinal fusion surgery. By calculating sample percentiles of empirical Bayes predictors of individual benefits, we examine the evolution of individual benefits over time. We additionally compare these percentiles with percentiles calculated with a Monte Carlo approach. We showed that empirical Bayes predictors of individual benefits do not only allow examining benefits in specific patients but also reflect overall population trends reliably.

Los registros de salud electrónicos (RSE) suministran recursos valiosos para estudios longitudinales y para comprender los factores de riesgo asociados con pobres resultados clínicos. Sin embargo, estos podrían no contener seguimientos completos, y los datos faltantes podrían no ser al azar, debido a que el alta hospitalaria puede depender en parte de resultados clínicos esperados pero no registrados que ocurren después de dar de alta al paciente. Esta ausencia de datos no ignorables requiere métodos apropiados de análisis. Aquí estamos interesados en medir y analizar beneficios individuales de tratamientos médicos en pacientes consignados en bases de datos RSE. Proponemos un método para predecir beneficios individuales el cual maneja los datos faltantes debidos al alta hospitalaria. La respuesta clínica longitudinal de interés se modela junto con el tiempo de estadía en el hospital usando un modelo conjunto de efectos mixtos, y los beneficios individuales se predicen por medio de un enfoque frecuentista: el enfoque Bayesiano empírico. Nuestro enfoque es ilustrado evaluando los beneficios individuales del tratamiendo para el dolor en pacientes que fueron sometidos a cirugía de fusión espinal. Aquí examinamos la evolución de los beneficios individuales a través del tiempo mediante el cálculo de los percentiles muestrales de los predictores de Bayes empíricos de los beneficios individuales. También comparamos estos percentiles con percentiles calculados mediante un enfoque Monte Carlo. Los resultados mostraron que los predictores de Bayes empíricos de beneficios individuales no sólo permiten examinar beneficios en pacientes específicos sino que también reflejan confiablemente las tendencias poblacionales globales.

References

Adogwa, O., Parker, S. L., Shau, D. N., Mendenhall, S. K., Bydon, A., Cheng, J. S., Asher, A. L. & McGirt, M. J. (2013), 'Preoperative Zung depression scale predicts patient satisfaction independent of the extent of improvement after revision lumbar surgery', The Spine Journal 13, 501-506. DOI: https://doi.org/10.1016/j.spinee.2013.01.017

Albers, D. J., Elhadad, N., Claassen, J., Perotte, R., Goldstein, A. & Hripcsak, G. (2018), 'Estimating summary statistics for electronic health record laboratory data for use in high-throughput phenotyping algorithms', Journal of Biomedical Informatics 78, 87-101. DOI: https://doi.org/10.1016/j.jbi.2018.01.004

Anderson, J. T., Haas, A. R., Percy, R., Woods, S. T., Ahn, U. M. & Ahn, N. U. (2015), 'Clinical depression is a strong predictor of poor lumbar fusion outcomes among workers' compensation subjects', Spine 40, 748-756. DOI: https://doi.org/10.1097/BRS.0000000000000863

Andrews, N. & Cho, H. (2018), 'Validating effectiveness of subgroup identification for longitudinal data', Statistics in Medicine 37, 98-106. DOI: https://doi.org/10.1002/sim.7500

Armero, C., Forte, A., Perpiñán, H., Sanahuja, M. J. & Agustí, S. (2018), 'Bayesian joint modeling for assessing the progression of chronic kidney disease in children', Statistical Methods in Medical Research 27, 298-311. DOI: https://doi.org/10.1177/0962280216628560

Arnold, L. M., Palmer, R. H., Gendreau, R. M. & Chen, W. (2012), 'Relationships among pain, depressed mood, and global status in -bromyalgia patients: post hoc analyses of a randomized, placebo-controlled trial of milnacipran', Psychosomatics 53, 371-379. DOI: https://doi.org/10.1016/j.psym.2012.02.005

Botts, S., Diaz, F. J., Santoro, V., Spina, E., Muscatello, M. R., Cogollo, M., Castro, F. E. & de Leon, J. (2008), 'Estimating the effects of co-medications on plasma olanzapine concentrations by using a mixed model', Progress in Neuro-Psychopharmacology & Biological Psychiatry 32, 1453-1458. DOI: https://doi.org/10.1016/j.pnpbp.2008.04.018

Cho, H., Wang, P. & Qu, A. (2017), 'Personalized treatment for longitudinal data using unspecified random-effects model', Statistica Sinica 27, 187-205. DOI: https://doi.org/10.5705/ss.202015.0120

Crowther, M. J., Abrams, K. R. & Lambert, P. C. (2012), 'Flexible parametric joint modelling of longitudinal and survival', Statistics in Medicine 31, 4456-4471. DOI: https://doi.org/10.1002/sim.5644

De Gruttola, V. & Tu, X. M. (1994), 'Modelling progression of CD4-lymphocyte count and its relationship to survival time', Biometrics 50, 1003-1014. DOI: https://doi.org/10.2307/2533439

de Leon, J. (2012), 'Evidence-based medicine versus personalized medicine: are they enemies?', Journal of Clinical Psychopharmacology 32, 153-164. DOI: https://doi.org/10.1097/JCP.0b013e3182491383

Diaz, F. J. (2016), 'Measuring the individual benefit of a medical or behavioral treatment using generalized linear mixed-effects models', Statistics in Medicine 35, 4077-4092. DOI: https://doi.org/10.1002/sim.7005

Diaz, F. J. (2018), 'Construction of the design matrix for generalized linear mixed-effects models in the context of clinical trials of treatment sequences', Revista Colombiana de Estadística 41, 191-233. DOI: https://doi.org/10.15446/rce.v41n2.63332

Diaz, F. J. (2019), 'Estimating individual benefits of medical or behavioral treatments in severely ill patients', Statistical Methods in Medical Research 28, 911-927. DOI: https://doi.org/10.1177/0962280217739033

Diaz, F. J. (2021), 'Using population crossover trials to improve the decision process regarding treatment individualization in N-of-1 trials', Statistics in Medicine 40, 4345-4361. DOI: https://doi.org/10.1002/sim.9030

Diaz, F. J., Berg, M. J., Krebill, R., Welty, T., Gidal, B. E., Alloway, R. & Privitera, M. (2013), 'Random-effects linear modeling and sample size tables for two special crossover designs of average bioequivalence studies: the four-period, two-sequence, two-formulation and six-period, three-sequence, three-formulation designs', Clinical Pharmacokinetics 52, 1033-1043. DOI: https://doi.org/10.1007/s40262-013-0103-4

Diaz, F. J., Cogollo, M. R., Spina, E., Santoro, V., Rendon, D. M. & de Leon, J. (2012b), 'Drug Dosage Individualization Based on a Random-Effects Linear Model', Journal of Biopharmaceutical Statistics 22, 463-484. DOI: https://doi.org/10.1080/10543406.2010.547264

Diaz, F. J. & de Leon, J. (2013), 'The mathematics of drug dose individualization should be built with random effects linear models', Therapeutic Drug Monitoring 35, 276-277. DOI: https://doi.org/10.1097/FTD.0b013e318283e3c6

Diaz, F. J., Eap, C. B., Ansermot, N., Crettol, S., Spina, E. & de Leon, J. (2014), 'Can valproic acid be an inducer of clozapine metabolism?', Pharmacopsychiatry 47, 89-96. DOI: https://doi.org/10.1055/s-0034-1371866

Diaz, F. J., Rivera, T. E., Josiassen, R. C. & de Leon, J. (2007), 'Individualizing drug dosage by using a random intercept linear model', Statistics in Medicine 26, 2052-2073. DOI: https://doi.org/10.1002/sim.2636

Diaz, F. J., Santoro, V., Spina, E., Cogollo, M., Rivera, T. E., Botts, S. & de Leon, J. (2008), 'Estimating the size of the effects of co-medications on plasma clozapine concentrations using a model that controls for clozapine doses and confounding variables', Pharmacopsychiatry 41, 81-91. DOI: https://doi.org/10.1055/s-2007-1004591

Diaz, F. J., Yeh, H.-W. & de Leon, J. (2012a), 'Role of Statistical Random-Effects Linear Models in Personalized Medicine', Current Pharmacogenomics and Personalized Medicine 10, 22-32. DOI: https://doi.org/10.2174/1875692111201010022

Edelstein, C. L. (2008), 'Biomarkers of acute kidney injury', Advances in Chronic Kidney Disease 15, 222-234. DOI: https://doi.org/10.1053/j.ackd.2008.04.003

Frees, E. W. (2004), Longitudinal and Panel Data, Cambridge University Press, Cambridge. DOI: https://doi.org/10.1017/CBO9780511790928

Gaudin, D., Krafcik, B. M., Mansour, T. R. & Alnemari, A. (2017), 'Considerations in spinal fusion surgery for chronic lumbar pain: psychosocial factors, rating scales, and perioperative patient education-a review of the literature', World Neurosurgery 98, 21-27. DOI: https://doi.org/10.1016/j.wneu.2016.10.124

Gerbershagen, H. J., Pogatzki-Zahn, E., Aduckathil, S., Peelen, L. M., Kappen, T. H., van Wijck, A. J., Kalkman, C. J. & Meissner, W. (2014), 'Procedur-especific risk factor analysis for the development of severe postoperative pain', Anesthesiology 120, 1237-1245. DOI: https://doi.org/10.1097/ALN.0000000000000108

Gewandter, J. S., McDermott, M. P., He, H., Gao, S., Cai, X., Farrar, J. T., Katz, N. P., Markman, J. D., Senn, S., Turk, D. C. & Dworkin, R. H. (2019), 'Demonstrating heterogeneity of treatment effects among patients: an overlooked but important step toward precision medicine', Clinical Pharmacology & Therapeutics 106, 204-210. DOI: https://doi.org/10.1002/cpt.1372

Greden, J. F. (2009), 'Treating depression and pain', Journal of Clinical Psychiatry 70(6), e16. DOI: https://doi.org/10.4088/JCP.8005cc3c

Gronski, L., Martinson, W., Singh, K. P. & Ryan, J. (2012), 'Utility of daily troponin orders for identifying acute myocardial infarction patients for quality improvement', Critical Pathway in Cardiology 11, 74-76. DOI: https://doi.org/10.1097/HPC.0b013e31824c0459

Hedeker, D. & Gibbons, R. D. (2006), Longitudinal Data Analysis, Wiley-Interscience, Hoboken, NJ.

Hickey, G. L., Philipson, P., Jorgensen, A. & Kolamunnage-Dona, R. (2018), 'joineRML: a joint model and software package for time-to-event and multivariate longitudinal outcomes', BMC Medical Research Methodology 18, 50. [doi.org/10.1186/s12874-018-0502-1]. DOI: https://doi.org/10.1186/s12874-018-0502-1

Ibrahim, J. G. & Molenberghs, G. (2009), 'Missing data methods in longitudinal studies: a review', Test (Madr) 18, 1-43. DOI: https://doi.org/10.1007/s11749-009-0138-x

Johnson, N. L. (1949), 'Systems of Frequency Curves Generated by Methods of Translation', Biometrika 36, 149-176. DOI: https://doi.org/10.1093/biomet/36.1-2.149

Laird, N. M. (1998), 'Missing data in longitudinal studies', Statistics in Medicine 7, 305-315. DOI: https://doi.org/10.1002/sim.4780070131

Lesa-re, E., Rizopoulos, D. & Tsonaka, R. (2007), 'The logistic transform for bounded outcome scores', Biostatistics 8, 72-85. DOI: https://doi.org/10.1093/biostatistics/kxj034

Little, R. J. A. & Rubin, D. B. (2002), Statistical Analysis with Missing Data, Second Edition, Wiley, New York. DOI: https://doi.org/10.1002/9781119013563

Lotzke, H., Jakobsson, M., Brisby, H., Gutke, A., Hägg, O., Smeets, R., den Hollander, M., Olsson, L. E. & Lundberg, M. (2016), 'Use of the PREPARE (PREhabilitation, Physical Activity and exeRcisE) program to

improve outcomes after lumbar fusion surgery for severe low back pain: a study protocol of a person-centred randomised controlled trial', BMC Musculoskeletal Disorders 17(1), 349. [doi: 10.1186/s12891-016-1203-8]. DOI: https://doi.org/10.1186/s12891-016-1203-8

Miksad, R. A. & Abernethy, A. P. (2018), 'Harnessing the power of real-world evidence (RWE): A checklist to ensure regulatory-grade Data Quality', Clinical Pharmacology & Therapeutics 103, 202-205. DOI: https://doi.org/10.1002/cpt.946

Pantazis, N. & Touloumi, G. (2010), 'Analyzing longitudinal data in the presence of informative drop-out: The jmre1 command', Stata Journal 10, 226-251. DOI: https://doi.org/10.1177/1536867X1001000204

Papageorgiou, G., Mau-, K., Tomer, A. & Rizopoulos, D. (2019), 'An overview of joint modeling of time-to-event and longitudinal outcomes', Annual Review of Statistics and its Application 6, 223-240. DOI: https://doi.org/10.1146/annurev-statistics-030718-105048

Ruberg, S. J., Chen, L. & Wang, Y. (2010), 'The mean does not mean as much anymore: finding sub-groups for tailored therapeutics', Clinical Trials 7, 574-583. DOI: https://doi.org/10.1177/1740774510369350

Schluchter, M. D. (1992), 'Methods for the analysis of informatively censored longitudinal data', Statistics in Medicine 11, 1861-1870. DOI: https://doi.org/10.1002/sim.4780111408

Schluchter, M. D. & Piccorelli, A. V. (2019), 'Shared parameter models for joint analysis of longitudinal and survival data with left truncation due to delayed entry - Applications to cystic fibrosis', Statistical Methods in Medical Research 28, 1489-1507. DOI: https://doi.org/10.1177/0962280218764193

Senn, S. (2016), 'Mastering variation: variance components and personalized medicine', Statistics in Medicine 35, 966-977. DOI: https://doi.org/10.1002/sim.6739

Shardell, M. & Ferrucci, L. (2018), 'Joint mixed-effects models for causal inference with longitudinal data', Statistics in Medicine 37, 829-846. DOI: https://doi.org/10.1002/sim.7567

Shaw, A. D., Mythen, M. G., Shook, D., Hayashida, D. K., Zhang, X., Skaar, J. R., Iyengar, S. S. & Munson, S. H. (2018), 'Pulmonary artery catheter use in adult patients undergoing cardiac surgery: a retrospective, cohort study', Perioperative Medicine (Lond) 7, 24. [doi.org/10.1186/s13741-018-0103-x]. DOI: https://doi.org/10.1186/s13741-018-0103-x

Shirafkan, H., Mahmoudi-Gharaei, J., Fotouhi, A., Moza-arpur, S. A., Yaseri, M. & Hoseini, M. (2020), 'Individualizing the dosage of Methylphenidate in children with attention deficit hyperactivity disorder', BMC Medical Research Methodology 20, 56. [doi.org/10.1186/s12874-020-00934-y]. DOI: https://doi.org/10.1186/s12874-020-00934-y

Touloumi, G., Pocock, S. J., Babiker, A. G. & Darbyshire, J. H. (1999), 'Estimation and comparison of rates of change in longitudinal studies with informative drop-outs', Statistics in Medicine 18, 1215-1233. DOI: https://doi.org/10.1002/(SICI)1097-0258(19990530)18:10<1215::AID-SIM118>3.0.CO;2-6

Trivedi, M. H. (2004), 'The link between depression and physical symptoms', Primary Care Companion of the Journal of Clinical Psychiatry 6(Suppl 1), 12-16.

Urman, R. D., Boing, E. A., Pham, A. T., Khangulov, V., Fain, R., Nathanson, B. H., Zhang, X., Wan, G. J., Lovelace, B. & Cirillo, J. (2018), 'Improved outcomes associated with the use of intravenous acetaminophen for management of acute post-surgical pain in cesarean sections and hysterectomies', Journal of Clinical Medicine Research 10, 499-507. DOI: https://doi.org/10.14740/jocmr3380w

Wang, Z. & Diaz, F. J. (2020), 'A graphical approach to assess the goodness-of-fit of random-effects linear models when the goal is to measure individual benefits of medical treatments in severely ill patients', BMC Medical Research Methodology 20, 193. [doi.org/10.1186/s12874-020-01054-3]. DOI: https://doi.org/10.1186/s12874-020-01054-3

Weinmann, C., Komann, M. & Meissner, W. (2017), 'Tough cookies: the older the patients, the more pain tolerating?', European Journal of Anesthesiology 34(Suppl 55), 215.

Woodward, M. (2014), Epidemiology: Study Design and Data Analysis, Third Edition, Chapman & Hall/CRC, Boca Raton, FL. DOI: https://doi.org/10.1201/b16343

Zhang, X., de Leon, J., Crespo-Facorro, B. & Diaz, F. J. (2020), 'Measuring individual beneffets of psychiatric treatment using longitudinal binary outcomes: Application to antipsychotic benefits in non-cannabis and cannabis users', Journal of Biopharmaceutical Statistics 30, 916-940. DOI: https://doi.org/10.1080/10543406.2020.1765371

Zhu, X. & Qu, A. (2016), 'Individualizing drug dosage with longitudinal data', Statistics in Medicine 35, 4474-4488. DOI: https://doi.org/10.1002/sim.7016

How to Cite

APA

Diaz, F. J., Zhang, X., Pantazis, N. and de Leon, J. (2022). Measuring Individual Benefits of Medical Treatments Using Longitudinal Hospital Data with Non-Ignorable Missing Responses Caused by Patient Discharge: Application to the Study of Benefits of Pain Management Post Spinal Fusion. Revista Colombiana de Estadística, 45(2), 275–300. https://doi.org/10.15446/rce.v45n2.101597

ACM

[1]
Diaz, F.J., Zhang, X., Pantazis, N. and de Leon, J. 2022. Measuring Individual Benefits of Medical Treatments Using Longitudinal Hospital Data with Non-Ignorable Missing Responses Caused by Patient Discharge: Application to the Study of Benefits of Pain Management Post Spinal Fusion. Revista Colombiana de Estadística. 45, 2 (Jul. 2022), 275–300. DOI:https://doi.org/10.15446/rce.v45n2.101597.

ACS

(1)
Diaz, F. J.; Zhang, X.; Pantazis, N.; de Leon, J. Measuring Individual Benefits of Medical Treatments Using Longitudinal Hospital Data with Non-Ignorable Missing Responses Caused by Patient Discharge: Application to the Study of Benefits of Pain Management Post Spinal Fusion. Rev. colomb. estad. 2022, 45, 275-300.

ABNT

DIAZ, F. J.; ZHANG, X.; PANTAZIS, N.; DE LEON, J. Measuring Individual Benefits of Medical Treatments Using Longitudinal Hospital Data with Non-Ignorable Missing Responses Caused by Patient Discharge: Application to the Study of Benefits of Pain Management Post Spinal Fusion. Revista Colombiana de Estadística, [S. l.], v. 45, n. 2, p. 275–300, 2022. DOI: 10.15446/rce.v45n2.101597. Disponível em: https://revistas.unal.edu.co/index.php/estad/article/view/101597. Acesso em: 28 jan. 2025.

Chicago

Diaz, Francisco J., Xuan Zhang, Nikos Pantazis, and Jose de Leon. 2022. “Measuring Individual Benefits of Medical Treatments Using Longitudinal Hospital Data with Non-Ignorable Missing Responses Caused by Patient Discharge: Application to the Study of Benefits of Pain Management Post Spinal Fusion”. Revista Colombiana De Estadística 45 (2):275-300. https://doi.org/10.15446/rce.v45n2.101597.

Harvard

Diaz, F. J., Zhang, X., Pantazis, N. and de Leon, J. (2022) “Measuring Individual Benefits of Medical Treatments Using Longitudinal Hospital Data with Non-Ignorable Missing Responses Caused by Patient Discharge: Application to the Study of Benefits of Pain Management Post Spinal Fusion”, Revista Colombiana de Estadística, 45(2), pp. 275–300. doi: 10.15446/rce.v45n2.101597.

IEEE

[1]
F. J. Diaz, X. Zhang, N. Pantazis, and J. de Leon, “Measuring Individual Benefits of Medical Treatments Using Longitudinal Hospital Data with Non-Ignorable Missing Responses Caused by Patient Discharge: Application to the Study of Benefits of Pain Management Post Spinal Fusion”, Rev. colomb. estad., vol. 45, no. 2, pp. 275–300, Jul. 2022.

MLA

Diaz, F. J., X. Zhang, N. Pantazis, and J. de Leon. “Measuring Individual Benefits of Medical Treatments Using Longitudinal Hospital Data with Non-Ignorable Missing Responses Caused by Patient Discharge: Application to the Study of Benefits of Pain Management Post Spinal Fusion”. Revista Colombiana de Estadística, vol. 45, no. 2, July 2022, pp. 275-00, doi:10.15446/rce.v45n2.101597.

Turabian

Diaz, Francisco J., Xuan Zhang, Nikos Pantazis, and Jose de Leon. “Measuring Individual Benefits of Medical Treatments Using Longitudinal Hospital Data with Non-Ignorable Missing Responses Caused by Patient Discharge: Application to the Study of Benefits of Pain Management Post Spinal Fusion”. Revista Colombiana de Estadística 45, no. 2 (July 14, 2022): 275–300. Accessed January 28, 2025. https://revistas.unal.edu.co/index.php/estad/article/view/101597.

Vancouver

1.
Diaz FJ, Zhang X, Pantazis N, de Leon J. Measuring Individual Benefits of Medical Treatments Using Longitudinal Hospital Data with Non-Ignorable Missing Responses Caused by Patient Discharge: Application to the Study of Benefits of Pain Management Post Spinal Fusion. Rev. colomb. estad. [Internet]. 2022 Jul. 14 [cited 2025 Jan. 28];45(2):275-300. Available from: https://revistas.unal.edu.co/index.php/estad/article/view/101597

Download Citation

CrossRef Cited-by

CrossRef citations1

1. Francisco J Diaz. (2024). Measuring the individualization potential of treatment individualization rules: Application to rules built with a new parametric interaction model for parallel-group clinical trials. Statistical Methods in Medical Research, 33(8), p.1355. https://doi.org/10.1177/09622802241259172.

Dimensions

PlumX

Article abstract page views

235

Downloads

Download data is not yet available.

License

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

  1. Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
  2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
  3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).