definition of evaluation by different authors

Here we outline a few of the most notable models that demonstrate the contrast in approaches available. In the educational context, the . A discussion on the benefits and drawbacks of a range of evaluation tools (bibliometrics, economic rate of return, peer review, case study, logic modelling, and benchmarking) can be found in the article by Grant (2006). 2007). In developing the UK REF, HEFCE commissioned a report, in 2009, from RAND to review international practice for assessing research impact and provide recommendations to inform the development of the REF. This involves gathering and interpreting information about student level of attainment of learning goals., 2. The criteria for assessment were also supported by a model developed by Brunel for measurement of impact that used similar measures defined as depth and spread. More details on SROI can be found in A guide to Social Return on Investment produced by The SROI Network (2012). 0000334683 00000 n To evaluate impact, case studies were interrogated and verifiable indicators assessed to determine whether research had led to reciprocal engagement, adoption of research findings, or public value. The basic purpose of both measurement assessment and evaluation is to determine the needs of all the learners. The case study approach, recommended by the RQF, was combined with significance and reach as criteria for assessment. A collation of several indicators of impact may be enough to convince that an impact has taken place. The ability to record and log these type of data is important for enabling the path from research to impact to be established and the development of systems that can capture this would be very valuable. The Payback Framework systematically links research with the associated benefits (Scoble et al. The quality and reliability of impact indicators will vary according to the impact we are trying to describe and link to research. Although it can be envisaged that the range of impacts derived from research of different disciplines are likely to vary, one might question whether it makes sense to compare impacts within disciplines when the range of impact can vary enormously, for example, from business development to cultural changes or saving lives? What are the methodologies and frameworks that have been employed globally to assess research impact and how do these compare? Without measuring and evaluating their performance, teachers will not be able to determine how much the students have learned. In endeavouring to assess or evaluate impact, a number of difficulties emerge and these may be specific to certain types of impact. HEIs overview. Assessment refers to the process of collecting information that reflects the performance of a student, school, classroom, or an academic system based on a set of standards, learning criteria, or curricula. Definition of Evaluation by Different Authors Tuckman: Evaluation is a process wherein the parts, processes, or outcomes of a programme are examined to see whether they are satisfactory, particularly with reference to the stated objectives of the programme our own expectations, or our own standards of excellence. Its objective is to evaluate programs, improve program effectiveness, and influence programming decisions. Figure 2 demonstrates the information that systems will need to capture and link. In the UK, more sophisticated assessments of impact incorporating wider socio-economic benefits were first investigated within the fields of Biomedical and Health Sciences (Grant 2006), an area of research that wanted to be able to justify the significant investment it received. Clearly the impact of thalidomide would have been viewed very differently in the 1950s compared with the 1960s or today. Downloadable! Again the objective and perspective of the individuals and organizations assessing impact will be key to understanding how temporal and dissipated impact will be valued in comparison with longer-term impact. Measurement assessment and evaluation helps the teachers to determine the learning progress of the students. A university which fails in this respect has no reason for existence. Published by Oxford University Press. This is recognized as being particularly problematic within the social sciences where informing policy is a likely impact of research. The process of evaluation involves figuring out how well the goals have been accomplished. Studies (Buxton, Hanney and Jones 2004) into the economic gains from biomedical and health sciences determined that different methodologies provide different ways of considering economic benefits. In this case, a specific definition may be required, for example, in the Research Excellence Framework (REF), Assessment framework and guidance on submissions (REF2014 2011b), which defines impact as, an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia. Table 1 summarizes some of the advantages and disadvantages of the case study approach. For example, the development of a spin out can take place in a very short period, whereas it took around 30 years from the discovery of DNA before technology was developed to enable DNA fingerprinting. Where narratives are used in conjunction with metrics, a complete picture of impact can be developed, again from a particular perspective but with the evidence available to corroborate the claims made. Despite many attempts to replace it, no alternative definition has . As Donovan (2011) comments, Impact is a strong weapon for making an evidence based case to governments for enhanced research support. 0000348082 00000 n While looking forward, we will be able to reduce this problem in the future, identifying, capturing, and storing the evidence in such a way that it can be used in the decades to come is a difficulty that we will need to tackle. Developing systems and taxonomies for capturing impact, 7. Dennis Atsu Dake. Where quantitative data were available, for example, audience numbers or book sales, these numbers rarely reflected the degree of impact, as no context or baseline was available. Assessment for learning is ongoing, and requires deep involvement on the part of the learner in clarifying outcomes, monitoring on-going learning, collecting evidence and presenting evidence of learning to others.. 4. The term comes from the French word 'valuer', meaning "to find the value of". 0000001883 00000 n Not only are differences in segmentation algorithm, boundary definition, and tissue contrast a likely cause of the poor correlation , but also the two different software packages used in this study are not comparable from a technical point of view. By evaluating the contribution that research makes to society and the economy, future funding can be allocated where it is perceived to bring about the desired impact. 5. Incorporating assessment of the wider socio-economic impact began using metrics-based indicators such as Intellectual Property registered and commercial income generated (Australian Research Council 2008). This distinction is not so clear in impact assessments outside of the UK, where academic outputs and socio-economic impacts are often viewed as one, to give an overall assessment of value and change created through research. The following decisions may be made with the aid of evaluation. Attempts have been made to categorize impact evidence and data, for example, the aim of the MICE Project was to develop a set of impact indicators to enable impact to be fed into a based system. For example, following the discovery of a new potential drug, preclinical work is required, followed by Phase 1, 2, and 3 trials, and then regulatory approval is granted before the drug is used to deliver potential health benefits. 0000008675 00000 n The Oxford English Dictionary defines impact as a 'Marked effect or influence', this is clearly a very broad definition. n.d.). Indicators were identified from documents produced for the REF, by Research Councils UK, in unpublished draft case studies undertaken at Kings College London or outlined in relevant publications (MICE Project n.d.). %PDF-1.4 % This is a metric that has been used within the charitable sector (Berg and Mnsson 2011) and also features as evidence in the REF guidance for panel D (REF2014 2012). Why should this be the case? The first attempt globally to comprehensively capture the socio-economic impact of research across all disciplines was undertaken for the Australian Research Quality Framework (RQF), using a case study approach. There is a great deal of interest in collating terms for impact and indicators of impact. These sometimes dissim- ilar views are due to the varied training and background of the writers in terms of their profession, concerned with different aspects of the education process. To allow comparisons between institutions, identifying a comprehensive taxonomy of impact, and the evidence for it, that can be used universally is seen to be very valuable. 2007). In this sense, when reading an opinion piece, you must decide if you agree or disagree with the writer by making an informed judgment. Teresa Penfield, Matthew J. Baker, Rosa Scoble, Michael C. Wykes, Assessment, evaluations, and definitions of research impact: A review, Research Evaluation, Volume 23, Issue 1, January 2014, Pages 2132, https://doi.org/10.1093/reseval/rvt021. There is a distinction between academic impact understood as the intellectual contribution to ones field of study within academia and external socio-economic impact beyond academia. Evaluative research has many benefits, including identifying whether a product works as intended, and uncovering areas for improvement within your solution. They are often written with a reader from a particular stakeholder group in mind and will present a view of impact from a particular perspective. Assessment for Learning is the process of seeking and interpreting evidence for use by learners and their teachers to decide where the learners are in their learning, where they need to go and. 4 0 obj It can be seen from the panel guidance produced by HEFCE to illustrate impacts and evidence that it is expected that impact and evidence will vary according to discipline (REF2014 2012). 2010). 0000342798 00000 n This petition was signed by 17,570 academics (52,409 academics were returned to the 2008 Research Assessment Exercise), including Nobel laureates and Fellows of the Royal Society (University and College Union 2011). 0000008241 00000 n While aspects of impact can be adequately interpreted using metrics, narratives, and other evidence, the mixed-method case study approach is an excellent means of pulling all available information, data, and evidence together, allowing a comprehensive summary of the impact within context. It is possible to incorporate both metrics and narratives within systems, for example, within the Research Outcomes System and Researchfish, currently used by several of the UK research councils to allow impacts to be recorded; although recording narratives has the advantage of allowing some context to be documented, it may make the evidence less flexible for use by different stakeholder groups (which include government, funding bodies, research assessment agencies, research providers, and user communities) for whom the purpose of analysis may vary (Davies et al. It is very important to make sure people who have contributed to a paper, are given credit as authors. Metrics in themselves cannot convey the full impact; however, they are often viewed as powerful and unequivocal forms of evidence. While valuing and supporting knowledge exchange is important, SIAMPI perhaps takes this a step further in enabling these exchange events to be captured and analysed. It is acknowledged in the article by Mugabushaka and Papazoglou (2012) that it will take years to fully incorporate the impacts of ERC funding. The Goldsmith report concluded that general categories of evidence would be more useful such that indicators could encompass dissemination and circulation, re-use and influence, collaboration and boundary work, and innovation and invention. Impact is not static, it will develop and change over time, and this development may be an increase or decrease in the current degree of impact. The definition of health is not just a theoretical issue, because it has many implications for practice, policy, and health services. Reviewing the research literature means finding, reading, and summarizing the published research relevant to your question. The first category includes approaches that promote invalid or incomplete findings (referred to as pseudoevaluations), while the other three include approaches that agree, more or less, with the definition (i.e., Questions and/or Methods- (2008), and Hanney and Gonzlez-Block (2011). 10312. Although some might find the distinction somewhat marginal or even confusing, this differentiation between outputs, outcomes, and impacts is important, and has been highlighted, not only for the impacts derived from university research (Kelly and McNicol 2011) but also for work done in the charitable sector (Ebrahim and Rangan, 2010; Berg and Mnsson 2011; Kelly and McNicoll 2011). One of these, the RQF, they identified as providing a promising basis for developing an impact approach for the REF using the case study approach. Concerns over how to attribute impacts have been raised many times (The Allen Consulting Group 2005; Duryea et al. An evaluation essay is a composition that offers value judgments about a particular subject according to a set of criteria. n.d.). To understand the socio-economic value of research and subsequently inform funding decisions. As a result, numerous and widely varying models and frameworks for assessing impact exist. 6. If knowledge exchange events could be captured, for example, electronically as they occur or automatically if flagged from an electronic calendar or a diary, then far more of these events could be recorded with relative ease. While the case study is a useful way of showcasing impact, its limitations must be understood if we are to use this for evaluation purposes. A Preferred Framework and Indicators to Measure Returns on Investment in Health Research, Measuring Impact Under CERIF at Goldsmiths, Anti-Impact Campaigns Poster Boy Sticks up for the Ivory Tower. Thalidomide has since been found to have beneficial effects in the treatment of certain types of cancer. In viewing impact evaluations it is important to consider not only who has evaluated the work but the purpose of the evaluation to determine the limits and relevance of an assessment exercise. The Payback Framework enables health and medical research and impact to be linked and the process by which impact occurs to be traced. The University and College Union (University and College Union 2011) organized a petition calling on the UK funding councils to withdraw the inclusion of impact assessment from the REF proposals once plans for the new assessment of university research were released. Such a framework should be not linear but recursive, including elements from contextual environments that influence and/or interact with various aspects of the system. Scriven (2007:2) synthesised the definition of evaluation which appears in most dictionaries and the professional literature, and defined evaluation as "the process of determining merit, worth, or significance; an evaluation is a product of that process." . This distinction is not so clear in impact assessments outside of the UK, where academic outputs and socio-economic impacts are often viewed as one, to give an overall assessment of value and change created through research.

definition of evaluation by different authors 2023