Evidence Briefings: Towards a Medium to Transfer Knowledge from Systematic Reviews to Practitioners

ABSTRACT

Context: Integrate research evidence with practice is one of the main goals of evidence-based software engineering. However, recent studies show that the connection between systematic reviews and practitioners has not fully established.

Goal: This paper presents the first steps towards a medium to transfer knowledge acquired from systematic reviews to practitioners.

Method: We selected a set of systematic reviews identified by a tertiary study and extracted their findings to generate one-page Evidence Briefings to serve as mediums. A design specialist defined the briefings structure based on information design and gestalt principles. To evaluate the format and content of the briefings we conducted personal opinion surveys based on two groups: StackExchange users that posted questions in topics related to the reviews, and the authors of the selected reviews themselves. The former had a response rate of 21.9% (32 out 146) and the latter 31.8% (7 out of 22).

Results: Practitioners rarely use systematic review research papers as mediums to acquire knowledge, since just 9% have told to do so. Both researchers and practitioners positively evaluated the evidence briefings, since 71% and 82% of the StackExchange users and systematic review authors, respectively, agreed or strongly agreed that the briefings' interface is clear.

Conclusions: Researchers and practitioners were positive about the content and format of the evidence briefings we proposed. It is also possible to say that there is a gap between practitioners and systematic reviews due to the low percentage of practitioners that consume systematic reviews. The good reception of the evidence briefings from both sides show a possible route to reduce that gap.

References

1. A. Abran, P. Bourque, R. Dupuis, and J. W. Moore, editors. Guide to the Software Engineering Body of Knowledge - SWEBOK. 2001.
2. G. Ambrose and P. Harris. Basics Design 08: Design Thinking. Basics Design. AVA Publishing, 2009.
3. J. Batten. Comment on editorial literature reviews as a research strategy, 2012.
4. S. Beecham, P. O'Leary, S. Baker, I. Richardson, and J. Noll. Making software engineering research relevant. Computer, 47(4):80--83, 2014.
5. L. Best, A. Stevens, and D. Colin-Jones. Rapid and responsive health technology assessment: the development and evaluation process in the south and west region of england. Journal of Clinical Effectiveness, 1997.
6. F. O. Bjørnson and T. Dingsøyr. Knowledge management in software engineering: A systematic review of studied concepts, findings and research methods used. IST, 2008.
7. D. Budgen, B. Kitchenham, and P. Brereton. The case for knowledge translation. In ESEM, 2013.
8. A. Cameron. Rapid versus full systematic reviews: an inventory of current methods and practice in Health Technology Assessment. Technical report, 2007.
9. D. Chambers and P. Wilson. A framework for production of systematic review based briefings to support evidence-informed decision-making. Systematic reviews, 2012.
10. F. Q. B. da Silva, A. L. Santos, S. Soares, A. C. C. FranÃğa, C. V. Monteiro, and F. F. Maciel. Six years of systematic literature reviews in software engineering: An updated tertiary study. IST, 2011.
11. R. M. de Mello, K. T. Stolee, and G. H. Travassos. Investigating samples representativeness for an online experiment in java code search. In ESEM, 2015.
12. P. Devanbu, T. Zimmermann, and C. Bird. Belief & evidence in empirical software engineering. In ICSE, 2016.
13. D. J. Dubois and G. Tamburrelli. Understanding gamification mechanisms for software development. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2013, pages 659--662, 2013.
14. T. Dybå and T. Dingsøyr. Empirical studies of agile software development: A systematic review. IST, 2008.
15. I. D. Graham, J. Logan, M. B. Harrison, S. E. Straus, J. Tetroe, W. Caswell, and N. Robinson. Lost in knowledge translation: Time for a map? Journal of Continuing Education in the Health Professions, 2006.
16. F. Grigoleit, A. Vetro, D. M. Fernandez, W. Bohm, and P. Diebold. In quest for proper mediums for technology transfer in software engineering. In ESEM, 2015.
17. D. Hailey. A preliminary survey on the influence of rapid health technology assessments. International journal of technology assessment in health care, 2009.
18. D. Hailey, P. Corabian, C. Harstall, and W. Schneider. The use and impact of rapid health technology assessments. International journal of technology assessment in health care, 2000.
19. J. E. Hannay, T. Dybå, E. Arisholm, and D. I. Sjøberg. The effectiveness of pair programming: A meta-analysis. IST, 2009.
20. L. Hartling, J.-M. Guise, E. Kato, J. Anderson, S. Belinson, E. Berliner, D. M. Dryden, R. Featherstone, M. D. Mitchell, M. Motu'apuaka, H. Noorani, R. Paynter, K. A. Robinson, K. Schoelles, C. A. Umscheid, and E. Whitlockg. A taxonomy of rapid reviews links report types and methods to specific decision-making contexts. Journal of Clinical Epidemiology, 2015.
21. E. Hassler, J. C. Carver, N. A. Kraft, and D. Hale. Outcomes of a community workshop to identify and rank barriers to the systematic literature review process. EASE, 2014.
22. B. Haugset and G. Hanssen. Automated acceptance testing: A literature review and an industrial case study. In AGILE, 2008.
23. W. Hordijk, M. L. Ponisio, and R. Wieringa. Harmfulness of code duplication: A structured review of the evidence. In EASE, 2009.
24. E. Hossain, M. Babar, and H. young Paik. Using scrum in global software development: A systematic literature review. In ICGSE, 2009.
25. A. Jedlitschka, N. Juristo, and D. Rombach. Reporting experiments to satisfy professionals' information needs. Empirical Software Engineering, 19(6):1921--1955, 2014.
26. M. Jørgensen. Forecasting of software development work effort: Evidence on expert judgement and formal models. International Journal of Forecasting, 2007.
27. S. Khan, M. Niazi, and R. Ahmad. Critical barriers for offshore software development outsourcing vendors:A systematic literature review. In APSEC, 2009.
28. S. Khan, M. Niazi, and R. Ahmad. Critical success factors for offshore software development outsourcing vendors: A systematic literature review. In ICGSE, 2009.
29. S. Khangura, K. Konnyu, R. Cushman, J. Grimshaw, and D. Moher. Evidence summaries: the evolution of a rapid review approach. Systematic Reviews, 2012.
30. B. A. Kitchenham, T. Dyba, and M. Jorgensen. Evidence-based software engineering. ICSE, 2004.
31. E. Lupton and J. C. Phillips. Graphic Design: The New Basics. Princeton Architectural Press, 2nd edition edition, 2015.
32. L. Mamykina, B. Manoim, M. Mittal, G. Hripcsak, and B. Hartmann. Design lessons from the fastest Q&A site in the west. In SIGCHI, 2011.
33. M. McGregor and J. M. Brophy. End-user involvement in health technology assessment (hta) development: a way to increase impact. International Journal of Technology Assessment in Health Care, 2005.
34. P. Mohagheghi, V. Dehlen, and T. Neple. Definitions and approaches to model quality in model-based software development -- a review of literature. IST, 2009.
35. J. Nicolás and A. Toval. On the generation of requirements specifications from software engineering models: A systematic literature review. IST, 2009.
36. G. Pinto, F. Castor, and Y. D. Liu. Mining questions about software energy consumption. 11th MSR, 2014.
37. G. H. Pinto and F. Kamei. What programmers say about refactoring tools?: An empirical investigation of stack overflow. WRT, 2013.
38. D. Posnett, E. Warburg, P. Devanbu, and V. Filkov. Mining stack exchange: Expertise is evident from initial contributions. In SocialInformatics, 2012.
39. M. Reboucas, G. Pinto, F. Ebert, W. Torres, A. Serebrenik, and F. Castor. An empirical study on the usage of the swift programming language. In 23rd SANER, 2016.
40. R. Santos and F. da Silva. Motivation to perform systematic reviews and their impact on software engineering practice. In ESEM, 2013.
41. J. Singer, S. E. Sim, and T. C. Lethbridge. Software engineering data collection for field studies. In Guide to Advanced Empirical Software Engineering. Springer, 2008.
42. E. Smith, R. Loftin, E. Murphy-Hill, C. Bird, and T. Zimmermann. Improving developer participation rates in surveys. In CHASE, 2013.
43. D. Spencer. Card sorting: Designing usable categories. Rosenfeld Media, 2009.
44. B. Tondreau. Layout Essentials: 100 Design Principles for Using Grids (Design Essentials). Design Essentials. Rockport Publishers, 2011.
45. A. Trendowicz and J. Münch. Factors inuencing software development productivity-state-of-the-art and industrial experiences. Advances in Computers. 2009.
46. A. Tricco, J. Antony, W. Zarin, L. Strier, M. Ghassemi, J. Ivory, L. Perrier, B. Hutton, D. Moher, and S. E. Straus. A scoping review of rapid review methods. BMC Medicine, 2015.
47. S. Van de Velde, E. De Buck, T. Dieltjens, and B. Aertgeerts. Medicinal use of potato-derived products: conclusions of a rapid versus full systematic review. Phytotherapy Research, 2011.
48. B. Vasilescu, V. Filkov, and A. Serebrenik. Stack overflow and github: Associations between software development and crowdsourced knowledge. In 2013 IEEE International Conference on Social Computing (SocialCom), pages 188--195. IEEE, 2013.
49. W. Wang and M. W. Godfrey. Detecting api usage obstacles: A study of ios and android developer questions. In MSR, 2013.
50. I. Young, J.-M. Guise, E. Kato, J. Anderson, S. Belinson, E. Berliner, D. M. Dryden, R. Featherstone, M. D. Mitchell, M. Motu'apuaka, H. Noorani, R. Paynter, K. A. Robinson, K. Schoelles, C. A. Umscheid, and E. Whitlockg. A guide for developing plain-language and contextual summaries of systematic reviews in agri-food public health. Foodborne Pathogens and Disease, 2014.
51. I. Zechmeister and I. Schumacher. The impact of health technology assessment reports on decision making in austria. International journal of technology assessment in health care, 2012.