Tools We Use: Performance / Quality Metrics
A number of different performance and quality metrics have been developed for our clients. Today, we use a variation of Objective and Key Results (OKR) to develop metrics (I.e., key results) to organizational objectives. This helps focus everyone on the “right” goals.
“Going Beyond the Norm: Some Ideas for the British Columbia Cancer Agency” British Columbia Cancer Agency Retreat, Vancouver, BC
Measuring Clinical Trial Performance
Thoughts on Evaluation of CTSAs, NCRR
Other Work Related to Performance / Quality Metrics
1. Dilts DM, Zell A, Orwoll E (2015) “A Novel Approach to Measuring Efficiency of Scientific Research Projects: Data Envelopment Analysis.” Clinical and Translational Science. 8(5):495-501. PMID:26243147
2. Stephenson D, (plus 14 others) (2015) Charting a path toward combination therapy for Alzheimer’s disease,” Expert Review of Neurotherapeutics, 15(1):107-13. PMID: 25540951
3. Dilts DM, (2013) “A 3+1 Level Evaluation Model for CRM”, Evaluation & the Health Professions 36(4):464-477 PMID: 23908383.
4. Dilts DM, Cheng S, Crites JS, Sander AB, Doroshow J, (2010) “Phase III Clinical Trial Development: A Process of Chutes & Ladders,” Clin Cancer Res. 16(22): 5381-89. PMID:21062928
5. Dilts DM and Sandler AB, (2006) “The “Invisible” Barriers to Clinical Trials: The impact of Structural, Infrastructural, and Procedural Barriers to Opening Oncology Clinical Trials,” J Clin Oncol, 24(28): 4545-52. PMID: 17008693
6. Dilts DM, Sandler AB, Baker M, Cheng S, George SL, Karas KS, McGuire S, Menon G, Reusch J, Sawyer D, Scoggins M, Wu A, Zhou K, Schilsky RL (2006) “Processes to Activate Phase III Clinical Trials in a Cooperative Oncology Group: The Case of the Cancer Leukemia Group B (CALGB), J Clin Oncol, 24(28): 4553-57. PMID: 17008694
7. Hackett SM, and Dilts DM (2004) “A Real Options-Driven Theory of Business Incubation,” J of Technology Transfer, 29(1): 41-54. [Reprinted in “Recent Development in the Economics of Science and Innovation”, editor: A. N Link and C. Antonelli, 2013]
8. Dilts DM and Khamalah J (1999) “A comparison of ordinal analysis techniques in medical resource usage research,” Electronic Notes in Discrete Mathematics. (http://www.elsevier.com/cas/tree/store/ disc/sub/endm/store/disc2/disc2005.ps) (invited), 2: 51-68
9. Hill T, Menda R, and Dilts DM. (1998) “Using Product Profiling to Illustrate Manufacturing-Marketing Misalignment,” Interfaces, 28(4): 47-63.
10. Menda, R, and Dilts DM, (1997) “The Manufacturing Strategy Formulation Process: Linking Multifunctional Viewpoints,” J of Operations Management, 15(4): 223-41.
11. Dilts DM, Khamalah J, and Plotkin AD, (1994) “Clinical Low Vision Resource Usage Prediction,” Optometry & Vision Science, 71(7): 422-436. PMID: 7970557
12. Dilts DM (2014) “Time has come to raise the bar in oncology clinical trials,” J Clin Onc, 30, PMID:22454422 doi: 10.1200/JCO.2013.53.8009
13. Dilts DM (2012) “Robbing Peter to Pay Paul: Financing Clinical Trial Follow-Up,” J Clin Onc, 30, PMID:22454422
14. Dilts DM, Cheng SK, Wiltrout RH, Dahut WL, Bronez M, Helman LJ (2011) “Strategic alignment of clinical trials at the National Cancer Institute Center for Cancer Research: Development of the clinical mission and baseline portfolio metrics,” J Clin Oncol 29 (suppl; abstr e16619)
15. Stahl DC, Cheng SK, Lee E, Dilts DM (2011) “Benchmarking of oncology clinical trial metrics across AACI member centers,” J Clin Oncol 29 (suppl; abstr e16613)
16. Cheng S, Dietrich, M, Finnigan S, Sandler A, Crites J, Ferranti L, Wu A, Dilts DM (2009) “A Sense of Urgency: evaluating the link between clinical trial development time and the accrual performance of CTEP-sponsored studies,” J Clin Oncol 27:15s (suppl; abstr 6509)
17. Dilts DM, Sandler AB, Cheng S, Crites J, Ferranti L, Wu A, Bookman M, Thomas J, and Ostroff J, (2008) “Accrual to Clinical Trials at Selected Comprehensive Cancer Centers,” 26(15S): 6543
18. Dilts DM, Sandler AB, Cheng S, Crites J, Ferranti L, and Wu A (2008), “Setups in Oncology Clinical Trials: a missing opportunity for improvement,” EurOMA 2008.
19. Dilts DM, Sandler AB, Cheng S, Crites J, Ferranti L, and Wu A (2008), “Are Cancer Clinical Trials in the USA Going the Way of Oldsmobile?,” Production and Operations Management Society 2008.
20. Dilts DM (2005) “Practice Variation: The Achilles’ Heel in Quality Cancer Care,” Journal of Clinical Oncology, 23:17(Sept 1): 5881-2 (invited) PMID: 16087953
21. Piasecki JK, Calhoun E, Engelberg J, Rice W, Dilts DM, Belser D, Aronsky D, Jones I, Mason D, Stead B, (2005) “Computerized provider order entry in the emergency department: pilot evaluation of a return on investment analysis instrument,” AMIA Symposium Proceedings, p. 1081. PMID: 16779368
22. Ferranti, Lori and Dilts DM (2003) “A Systematic Review of Types and Uses of Medical Registries,” 20th Annual Research Meeting of the Academy of Health. (poster) (June)
23. Khamalah, J, and Dilts DM, (1997) “The Application of ‘Traditional’ Manufacturing Management Techniques in Health Care Resource Usage Prediction,” Operation Management Association Forum, April.
24. (2015) “Going Beyond the Norm: Some Ideas for the British Columbia Cancer Agency” British Columbia Cancer Agency Retreat, Vancouver, BC(May)
25. (2014) “The Promise and Perils in Clinical Trials,” Monash University, Melbourne, Australia(August, presented twice)
26. (2014) “Observations on Cancer Research from a Non-Medical Researcher,” keynote speech, Cancer Institute of New South Wales Annual Awards Banquet(August)
27. (2014) “The Importance of Doing Trials Right While doing the Right Trials,” Ministry of Health, NSW, Australia(August)
28. (2014) “Overcoming the Barriers of Transparency in Clinical Trial Data Access,” ASCO National Meeting, Chicago IL(June)
29. (2014) “Integration Metrics: Linking Mission, Science, Resources and Education,” NICHD Division of Intramural Research, Scientific Retreat, Bethesda, MD(Feb).
30. (2013) “Prioritizing and Measuring Clinical Research,” NIH Clinical Research Subcommittee, Bethesda, MD(Sept).
31. (2013) “Cooperative Groups, Version 3.0”, Palliative Care Research Consortium meeting, Chicago, IL(Sept).
32. (2013) “Consortium or not to Consortium”, ACT-AD National Meeting on Alzheimer’s Research, Washington, DC(May)
33. (2013) “A Model for Evaluating the Efficiency and Effectiveness of Clinical Research,” Medical Executive Committee, NIH, Bethesda MD(May).
34. (2013) “Process Improvement: Linking Improvement to Strategy,” Oregon Health Financial Manager’s Association, Portland, OR(Feb)
35. (2012) “Precompetitive Collaboration & Coopetition: Lessons Learned from History and Other Industries,” ACT-AD annual meeting, Bethesda MD(Nov)
36. (2012) “Metrics for Clinical Trials: Lessons Learned”, CTSA Annual Evaluation Meeting, Bethesda MD(Oct)
37. (2012) Measuring Clinical Trial Performance
38. (2010) Thoughts on NCRR Evaluation
39. (2011) “Management Principles in Healthcare” Southern Health, Melbourne AUS(Nov) and The Albert Hospital System, Melbourne AUS(Nov)
40. (2011) “Multi-disciplinary, Useful research” Melbourne Operations Management Society(MOMS)(Nov)
41. (2011) “Process Mapping – Purpose, Method & Expected Results,” University of Cincinnati Academic Health Center, Center for Clinical & Translational Science & Training(Sept)
42. (2011) “Precompetitive Collaboration: Lessons from History and Other Industries,” UCSF Accelerating Targeted Cancer Drug-Biomarker Development Conference, San Francisco CA(May)
43. (2011) “Mis-Measurement of Quality in IRBs: Thoughts from Other Industries,” 13th annual Subject Protection conference, Schulman Group, Cincinnati Children’s and the University of Kentucky, Cincinnati, OH(Sept).
44. (2010) “Beyond Efficiency: A Strategic Approach to Clinical Research,” Keynote address, 10th Annual Oncore Users Group Madison, WI(Aug)
45. (2010) “The Science of Trial Selection, Infrastructure, and Operations,” NCI-ASCO Symposium on Accruals, Chicago IL(Apr)
46. (2009) “The Need for Operations Management in Healthcare: The Case of Oncology Clinical Trials,” Distinguished Lecture, Supply Chain Management Area, Sir Wilfrid Laurier University(Dec)
47. (2009) “Envisioning the future: Lessons from the past(& present) in Oncology Clinical Trials”, keynote speech to Johns Hopkins Sidney Kimmel Comprehensive Cancer Center strategic retreat, Baltimore MD(Mar)
48. (2008) “Building Networks: Observations from Oncology and Non-Medical Industries”, keynote address for the NHLBI Workshop on Sickle Cell Disease. Washington, DC(Oct)
49. Behringer B, Omohundro E, Boswell D, Evans D and Ferranti L. The Tennessee Department of Health WORKshops on the Use of Secondary Data for Community Health Assessment, 2012. Preventing Chronic Disease. (2014). 11:130206. DOI: http://dx.doi.org/10.5888/pdc.11.130260.
2. Stephenson D, (plus 14 others) (2015) Charting a path toward combination therapy for Alzheimer’s disease,” Expert Review of Neurotherapeutics, 15(1):107-13. PMID: 25540951
3. Dilts DM, (2013) “A 3+1 Level Evaluation Model for CRM”, Evaluation & the Health Professions 36(4):464-477 PMID: 23908383.
4. Dilts DM, Cheng S, Crites JS, Sander AB, Doroshow J, (2010) “Phase III Clinical Trial Development: A Process of Chutes & Ladders,” Clin Cancer Res. 16(22): 5381-89. PMID:21062928
5. Dilts DM and Sandler AB, (2006) “The “Invisible” Barriers to Clinical Trials: The impact of Structural, Infrastructural, and Procedural Barriers to Opening Oncology Clinical Trials,” J Clin Oncol, 24(28): 4545-52. PMID: 17008693
6. Dilts DM, Sandler AB, Baker M, Cheng S, George SL, Karas KS, McGuire S, Menon G, Reusch J, Sawyer D, Scoggins M, Wu A, Zhou K, Schilsky RL (2006) “Processes to Activate Phase III Clinical Trials in a Cooperative Oncology Group: The Case of the Cancer Leukemia Group B (CALGB), J Clin Oncol, 24(28): 4553-57. PMID: 17008694
7. Hackett SM, and Dilts DM (2004) “A Real Options-Driven Theory of Business Incubation,” J of Technology Transfer, 29(1): 41-54. [Reprinted in “Recent Development in the Economics of Science and Innovation”, editor: A. N Link and C. Antonelli, 2013]
8. Dilts DM and Khamalah J (1999) “A comparison of ordinal analysis techniques in medical resource usage research,” Electronic Notes in Discrete Mathematics. (http://www.elsevier.com/cas/tree/store/ disc/sub/endm/store/disc2/disc2005.ps) (invited), 2: 51-68
9. Hill T, Menda R, and Dilts DM. (1998) “Using Product Profiling to Illustrate Manufacturing-Marketing Misalignment,” Interfaces, 28(4): 47-63.
10. Menda, R, and Dilts DM, (1997) “The Manufacturing Strategy Formulation Process: Linking Multifunctional Viewpoints,” J of Operations Management, 15(4): 223-41.
11. Dilts DM, Khamalah J, and Plotkin AD, (1994) “Clinical Low Vision Resource Usage Prediction,” Optometry & Vision Science, 71(7): 422-436. PMID: 7970557
12. Dilts DM (2014) “Time has come to raise the bar in oncology clinical trials,” J Clin Onc, 30, PMID:22454422 doi: 10.1200/JCO.2013.53.8009
13. Dilts DM (2012) “Robbing Peter to Pay Paul: Financing Clinical Trial Follow-Up,” J Clin Onc, 30, PMID:22454422
14. Dilts DM, Cheng SK, Wiltrout RH, Dahut WL, Bronez M, Helman LJ (2011) “Strategic alignment of clinical trials at the National Cancer Institute Center for Cancer Research: Development of the clinical mission and baseline portfolio metrics,” J Clin Oncol 29 (suppl; abstr e16619)
15. Stahl DC, Cheng SK, Lee E, Dilts DM (2011) “Benchmarking of oncology clinical trial metrics across AACI member centers,” J Clin Oncol 29 (suppl; abstr e16613)
16. Cheng S, Dietrich, M, Finnigan S, Sandler A, Crites J, Ferranti L, Wu A, Dilts DM (2009) “A Sense of Urgency: evaluating the link between clinical trial development time and the accrual performance of CTEP-sponsored studies,” J Clin Oncol 27:15s (suppl; abstr 6509)
17. Dilts DM, Sandler AB, Cheng S, Crites J, Ferranti L, Wu A, Bookman M, Thomas J, and Ostroff J, (2008) “Accrual to Clinical Trials at Selected Comprehensive Cancer Centers,” 26(15S): 6543
18. Dilts DM, Sandler AB, Cheng S, Crites J, Ferranti L, and Wu A (2008), “Setups in Oncology Clinical Trials: a missing opportunity for improvement,” EurOMA 2008.
19. Dilts DM, Sandler AB, Cheng S, Crites J, Ferranti L, and Wu A (2008), “Are Cancer Clinical Trials in the USA Going the Way of Oldsmobile?,” Production and Operations Management Society 2008.
20. Dilts DM (2005) “Practice Variation: The Achilles’ Heel in Quality Cancer Care,” Journal of Clinical Oncology, 23:17(Sept 1): 5881-2 (invited) PMID: 16087953
21. Piasecki JK, Calhoun E, Engelberg J, Rice W, Dilts DM, Belser D, Aronsky D, Jones I, Mason D, Stead B, (2005) “Computerized provider order entry in the emergency department: pilot evaluation of a return on investment analysis instrument,” AMIA Symposium Proceedings, p. 1081. PMID: 16779368
22. Ferranti, Lori and Dilts DM (2003) “A Systematic Review of Types and Uses of Medical Registries,” 20th Annual Research Meeting of the Academy of Health. (poster) (June)
23. Khamalah, J, and Dilts DM, (1997) “The Application of ‘Traditional’ Manufacturing Management Techniques in Health Care Resource Usage Prediction,” Operation Management Association Forum, April.
24. (2015) “Going Beyond the Norm: Some Ideas for the British Columbia Cancer Agency” British Columbia Cancer Agency Retreat, Vancouver, BC(May)
25. (2014) “The Promise and Perils in Clinical Trials,” Monash University, Melbourne, Australia(August, presented twice)
26. (2014) “Observations on Cancer Research from a Non-Medical Researcher,” keynote speech, Cancer Institute of New South Wales Annual Awards Banquet(August)
27. (2014) “The Importance of Doing Trials Right While doing the Right Trials,” Ministry of Health, NSW, Australia(August)
28. (2014) “Overcoming the Barriers of Transparency in Clinical Trial Data Access,” ASCO National Meeting, Chicago IL(June)
29. (2014) “Integration Metrics: Linking Mission, Science, Resources and Education,” NICHD Division of Intramural Research, Scientific Retreat, Bethesda, MD(Feb).
30. (2013) “Prioritizing and Measuring Clinical Research,” NIH Clinical Research Subcommittee, Bethesda, MD(Sept).
31. (2013) “Cooperative Groups, Version 3.0”, Palliative Care Research Consortium meeting, Chicago, IL(Sept).
32. (2013) “Consortium or not to Consortium”, ACT-AD National Meeting on Alzheimer’s Research, Washington, DC(May)
33. (2013) “A Model for Evaluating the Efficiency and Effectiveness of Clinical Research,” Medical Executive Committee, NIH, Bethesda MD(May).
34. (2013) “Process Improvement: Linking Improvement to Strategy,” Oregon Health Financial Manager’s Association, Portland, OR(Feb)
35. (2012) “Precompetitive Collaboration & Coopetition: Lessons Learned from History and Other Industries,” ACT-AD annual meeting, Bethesda MD(Nov)
36. (2012) “Metrics for Clinical Trials: Lessons Learned”, CTSA Annual Evaluation Meeting, Bethesda MD(Oct)
37. (2012) Measuring Clinical Trial Performance
38. (2010) Thoughts on NCRR Evaluation
39. (2011) “Management Principles in Healthcare” Southern Health, Melbourne AUS(Nov) and The Albert Hospital System, Melbourne AUS(Nov)
40. (2011) “Multi-disciplinary, Useful research” Melbourne Operations Management Society(MOMS)(Nov)
41. (2011) “Process Mapping – Purpose, Method & Expected Results,” University of Cincinnati Academic Health Center, Center for Clinical & Translational Science & Training(Sept)
42. (2011) “Precompetitive Collaboration: Lessons from History and Other Industries,” UCSF Accelerating Targeted Cancer Drug-Biomarker Development Conference, San Francisco CA(May)
43. (2011) “Mis-Measurement of Quality in IRBs: Thoughts from Other Industries,” 13th annual Subject Protection conference, Schulman Group, Cincinnati Children’s and the University of Kentucky, Cincinnati, OH(Sept).
44. (2010) “Beyond Efficiency: A Strategic Approach to Clinical Research,” Keynote address, 10th Annual Oncore Users Group Madison, WI(Aug)
45. (2010) “The Science of Trial Selection, Infrastructure, and Operations,” NCI-ASCO Symposium on Accruals, Chicago IL(Apr)
46. (2009) “The Need for Operations Management in Healthcare: The Case of Oncology Clinical Trials,” Distinguished Lecture, Supply Chain Management Area, Sir Wilfrid Laurier University(Dec)
47. (2009) “Envisioning the future: Lessons from the past(& present) in Oncology Clinical Trials”, keynote speech to Johns Hopkins Sidney Kimmel Comprehensive Cancer Center strategic retreat, Baltimore MD(Mar)
48. (2008) “Building Networks: Observations from Oncology and Non-Medical Industries”, keynote address for the NHLBI Workshop on Sickle Cell Disease. Washington, DC(Oct)
49. Behringer B, Omohundro E, Boswell D, Evans D and Ferranti L. The Tennessee Department of Health WORKshops on the Use of Secondary Data for Community Health Assessment, 2012. Preventing Chronic Disease. (2014). 11:130206. DOI: http://dx.doi.org/10.5888/pdc.11.130260.