Reviewing the Methodology of the Paper - Example

A CRITICAL REVIEW OF THE METHODOLOGY 20th October, 2008 EXECUTIVE SUMMARY The detailed analysis or critical review of an evaluation of a report involves an in-depth approach to each and every aspect of the report that has an impact on the practicality or the effectiveness of the report. Evaluation can help us to understand that whether the specific aims and objectives of the evaluation have been actually achieved in the true sense or not. One such critical review would be carried further in this report which relates to the development of rural farmers in Thailand. For a holistic review, a formative evaluation would be preferred which is a continuous process of critical reflections on the past experiences leading to a specific result or action. INTRODUCTION The report ‘Targeting Poor Farmers: Contributions to Rural development in Thailand’ was evaluated by the three eminent personalities Mr. John Wrigley, Dr Jim Taylor and Ms Walaitat Worakul to study the impact of three main projects which were initiated to uplift the rural life in Thailand, with lot many components under consideration such as environmental degradation, irrigation issues, methods of agricultural extension, capacity development and few more elements in relation to the life of the urban people in Thailand. The following analysis deals intricately with the analysis of various structural and functional aspects involved in the report. The study would try to figure out the critical review of the methodology chosen in the report with special emphasis in drawing out the political nature of the report presented on the targeted farmers. The main focus of the evaluation to be done would be on finding out the genuity of the effectiveness and efficiency of the report as claimed by the evaluators of the Aus AID, moreover approaches applied and methodology adopted would also be highlighted in the analysis. The evaluation in the following study aims to understand the outcomes of the various practices entailed in the report. As far as an overall performance of the project undertaken by the AusAID was concerned, a moderate to low achievements was reported which included objectives such as extension methods and skills applied, communication, interest groups, marketing systems, demonstration diversified procedures etc which revealed that the report was more inclined towards the economic aspects rather than applying a holistic approach. A detailed analysis of each objective and the values attached with each of them would be discussed in the following sections. SUMMARY OF THE PURPOSE OF THE EVALUATION The purpose of the evaluation undergone by the AusAID projects that were focusing on the rural development in Thailand as poor farmers being their major target was basically to assess the effectiveness and the impact that the three main rural development projects had on the village family and the community life in general, also including issues on environmental and social perspectives. The aim of the evaluation done by the AusAID was basically to assess the feasibility of the three projects, HASD II, , Thai-Australia Agricultural and Social Development Project, implemented by the Hill Tribe Welfare Division, Department of Public Welfare, Ministry of Labour and Social Welfare in the year1988 and accomplished in 1993, secondly a project named TAAEP, Thai-Australia Agricultural Extension Project, implemented by the Department of Agricultural Extension, Ministry of Agriculture and Cooperatives in the year 1991 and extended till 1995 and lastly to determine the effectiveness of project ULRAP, Ubon Ratchathani Land Reform Area Project, implemented by Agricultural Land Reform Office, Ministry of Agriculture and Cooperatives in the same year 1991 till 1995. The revaluation also aims to figure out the lessons drawn from project experience so as to incorporate the necessary changes in the rural development future projects undertaken and used by AusAID and RTG. The evaluation also highlights on the social, economic, institutional and environmental impacts of all three projects under consideration, along with the gender impact, complexity of the project designs and sustainability of the project benefits. DISCUSSION OF MAJOR ISSUES CRITICAL REVIEW OF THE THREE PROJECTS IN GENERAL STRUCTURE OF EVALUTION After examining the report, various issues pertaining to matters such as apt methodology, purpose and many others have to be dealt with; some of them have a positive correlation with the evaluation while few others need a deeper conceptualization and thought. To start with the purpose of the evaluation which is basically that is to assess the effectiveness of the three main projects targeting the poor farmer is absolutely clear as the evaluation highlights the performances and impact of each report on the rural development in Thailand. The evaluation does not support any literature review to a great extent which if would have been included, could have added to the extent of the success of the rural development projects. The economic, social, political conditions of the people who have been affected by the implementation of these projects could have been better supported through exact comparison of the past figures and the present quantitative data which is very less in the supported document. Moreover, the exact statistics of people defined and classified under the poor category and the basis on which the people have been marked as poor has not been defined and justified which definitely creates a gap in the literature review and the present analysis and findings of the projects. The evaluation which has been undertaken has revealed a systematic approach in analyzing aspects such as defining of objectives, projecting a background for the project, highlighting the achievements of the project through project design, extension methods, institutional aspects, communication and cooperation aspects, achievement of Demonstration of Diversified Production Enterprises Objective, Achievement of Self-Sustaining Interest Groups Objective, Achievement of Strengthened Marketing System Objective and the project impact on the family incomes, community developments, environmental impact, institutional impact, sustainability and project lessons. Though the literature review has not being included, yet whatever existing information has been used in the evaluation has been done keeping in mind the purpose and essence of the evaluation. The review after each project gives a detailed future indications to the government about the design and implementation of the project, the sustainability factor, participation of the targeted poor segment and the cross cutting issues. METHODOLGY USED The use of methodology in the evaluation comprises of both the quantitative as well as the qualitative aspects. The quantitative aspects basically include the targeted rural population, the expenditure incurred on the project, the human resource involved in the accomplishment of the project including various departments such as agricultural, social, institutional, management and monitoring departments. The evaluation also includes a quantitative aspect of growth in village revolving funds, family incomes etc. on the other hand, a qualitative interpretation of the evaluation is a predominant areas which is mainly focusing on the impact of the projects on the community life of the targeted poor farmer segment, major impacts being social, institutional, environmental and income covering local organizational influence including local government, women’s participation, village groups etc , natural resource strategies including project capacity, management tools etc, infrastructure including water, road construction, management including training, institutionalization etc. The above methodology implemented in the evaluation of the report seems to be quite effective as it reveals a comprehensive study of the impact of the projects on the community life of the poor in all aspect, right from social to environmental. The choice of methodology for evaluation depends upon the availability of the data and the problem under review. In the present context, more emphasies on qualitative aspects are justified while quantitative approaches are good for larger populations and getting more generalized results what is often needed is a more in-depth view but as the qualitative data is based on personal perceptions and view points, it may add certain biasness in the evaluation of the project .in relation with the current evaluation contemporary approaches related with poverty alleviation have also been included. The quantitative method is mainly concerned with the questions like the quantity of products or people or any material or non-material things which have been sufficiently and satisfactorily mentioned in the evaluation referring to aspects such as ‘How much’ and ‘What kind’? The main sources for the secondary data collected and analyzed in the report are other AusAID projects and other kind of similar projects with have been undertaken in the past such as HTWD (Hill Tribe Welfare Division, Department of Public Welfare, Ministry of Labour and Social Welfare data), DIS data (Development Information System of HTWD), thee current agricultural database (KPG), Kor Por Gor, Agricultural Data Base used by DOAE (Department of Agricultural Extension, MOAC) was derived from the project management information system. Not much use of secondary data has been seen in the evaluation; no such graphs or pictorial forms of comparisons between the past and the present scenario have been included. The evaluation could have been more transparent if figures and charts could have been embodied making use of secondary data available in the past projects under in Thailand for rural development and other such related projects. The language used in the evaluation report is pretty simple, using the terminology related with the project topic, not much use of jargon is noticed which adds up to the effectiveness and understandability of the report and idea projected. The recommendation proposed or the lessons learnt which are specific to the evaluation have been deeply thought over after considering all the impacts and procedures related with the different aspects of the projects such as the feasibility, practicality, impact, sustainability, participation, expenditures, performance, replicability, gender empowerment etc . The evaluation has included a glossary of terms which have simplified the interpretations and understanding of unfamiliar terms. The areas of knowledge, subject and disciplines which the evaluators have focused on are economics; hydrology; anthropology; social psychology and crop science to analyze various factors such as family income, community development and participation impact , environmental achievements and impact of each project such as strategies adopted, omissions and achievements etc, institutional impact, gender and rural impact which have significant influence on the projection and justification of the project title which is the rural development in Thailand, targeting poor farmers but the evaluation lacks a major focus on the political and ethical aspects of the project which could have certainly supported the arguments (Missingham, B, 2000). SCOPE FOR IMPROVIZATION The evaluation interpretation so far does have scope for amendments and changes as far as the methodological approach is concerned. Recent studies have found out that Q methodology is one of the best methods to carry on a rural search. Q methodology was developed by psychologist and physicist William Stephenson in the 1930s which provides an ontological perspective to the analysis. Its aim in general is to study people’s own perspectives, meanings and opinions but in practical terms it has much more to offer as some scholars, believe the Q methodology allows a ‘scientific’ study of the problem or project stated. Q method provides an opportunity to shift focus from a particular individual narrative to an analysis of the range of viewpoints that is shared by a particular group of participants. The advantage the Q methodology could have extended in the current evaluation was focus on the subjective experiences of participants, its prominence on context, and its privileging of the everyday and local, secondly the methodology has an immense capacity to challenge the power of the researcher, and thirdly it helps in acknowledging how troublesome the concepts such as truth and language are. The Q methodology helps in producing more nuanced understandings of social phenomenon. Including the study of how stakeholders view the role of development trusts and projects, this important angle to the evaluation is so far missing, that is the role of stakeholders who are affected by the development of the three main projects undertaken for the upliftment of the rural class in Thailand (Previte J, Pini B, Haslam-McKenzie F, 2007). Though the evaluation has satisfactorily explained the role of infrastructure and the projects undertaken but a detailed view of infrastructure and the household welfare needs to be highlight as well (Songco A J, 2002). Watershed activities, irrigation and development projects could have added a secondary data research and past irrigation projects initiated in Northern Thailand (IFAD Evaluation, 2007). The evaluation does highlight impacts socially and environmentally but ethical considerations are also a prominent part of an evaluation which should have been focused. The evaluation has tremendous scope to add the different perspective where a wide range of stakeholders who are directly or indirectly being affected by the projects initiation are considered (Rossi H P, Freeman E H, Lipsey W M, 1999). One of the main targeted areas of each project was poverty alleviation for which promotion by many NGOs and support by socially and environmentally sensitive urban elites in Thailand and elsewhere was initiated along with the World Bank assistance which Northern Thailand has been constantly getting in order to secure land for the poor (Leonard R, retrieved on 20th Oct, 2008), however a detailed SPOT check analysis could have added on to the findings of the rural project. SPOT tends to elaborate the evaluation by including a Strategic Distance which reveal direct or indirect interventions, Poverty dynamics that delve deeper into the relation of interventions to poverty causes and effects. Operational level which embodies a wider perspective of international, macro/national, micro/local research factors. Targeting where the focal point is intended towards direct or indirect beneficiaries identified in the project. New approaches such as Sector Wide Approaches and the Comprehensive Development Framework can also be useful (Aus AID, 2000) The evaluation should also focus on the Australian and Thai relationships and there impact on the rural development and the targeted poor families. Australia has supported the projects both by providing high skilled labour and sufficient funds; however the economic problems of Australia might have an impact on the projects (Perlez J, 1987). CRITICAL EVALUATION OF EACH PROJECT Gender and Rural Development: Various objectives related to the three different projects such as HASD II, TAAEP and ULRAP have been underlined which mainly focus on the women having a say in the decision making and women’s participation in the implementation process, for which the results have ranged from relatively low to moderate levels. Though the findings of the project have been able to justify the effectiveness and impact of women participation and involvement in the project yet it does not focus whether the involvement of women would lead to any kind of improvement in efficiency of the project or not. The assessment stands positive on the front that apart from the economic aspect, other approaches such as leadership traits, ethnicity, and culture also have been reflected. One limitation in assessing the gender importance was lack of secondary data analysis. References from many other AusAID projects which have focused on Evaluation and gender should have been a part of the evaluation which could have further aided in thematic and non thematic evaluations (AusAID, 2002). A major impact of the project can be seen on the TAO which has taken firm steps in encouraging the women of the rural areas to come forward and refocus development funds on social development programmes rather than roads and infrastructure (International Development Research, 1997) HASD II was a five year lengthy integrated area development project with many constraints at a large scale with a target of 102000 people in 579 villages. AusAID has not funded another rural project of this scale or complexity since HASD II was completed in 1993. Phase I of the project focused on environmental objectives while phase II focused on the social issues. The project achieved most of its objectives at a moderate level with a moderate impact and sustainability and low replicability. HASD II successfully provided improved health and education services in project villages, constructed roads and brought hill tribe peoples into planning processes and group activities. Project training and systems development were the main factors of the project. The project reduced a remarkable rate of environmental degradation and extended social linkages with the rest of the nation. Greater improvements within rainfed cropping systems and high budgets would have yielded wider impacts. TAAEP was extended over 3.5 years with lower outcomes than expected, though the project did contribute to the development of an agricultural database/management information system. Cooperative rice marketing was successfully introduced and supported with infrastructure and training in one district with major focus on gender or environmental issues. The availability of DOAE budget was enough to sustain use of extension methodology and the database developed with project assistance. ULRAP benefited from consistent Australian staffing from design to completion and implementation of innovative method of collaboration between government, NGO and Australian Contributions to Rural Development in Thailand .ULRAP was highly successful in achieving women’s participation There was greater awareness regarding environmental issues in the community . ULRAP has had a high impact on the capacity of local men and women whereas it had a moderate impact on the local environment, infrastructure and incomes and had a low impact on ALRO approaches due to lack of funds. CONCLUSION The AusAID evaluation report on the contributions to the rural development fund in Thailand has been done meticulously, involving all aspects which make the report a complete authentic evaluation from a political perspective. It can be said that both effectiveness and affectivity of the objectives of the three projects undertaken for the development have in some manner or the other helped in enhancing the quality of life of people in Thailand with a strong supported aided by Australia in order to improve the social and economic situation of people in the northern provinces . The overall impact of the three projects resulted in two of the projects having a moderate overall impact in improving the quality of life, especially where major contribution was made through changes in land use possible as a result of water development activities and one project was highly successful in escalating local development capacity and in increasing the status of women and their participation in social affairs. The overall evaluation of the report was systematic, clear in objectivity and comprehensive in nature. REFERENCES Perlez J. February 2, 1987. Special to the New York times, retrieved on 20th Oct, 2008, http://query.nytimes.com/gst/fullpage.html?res=9B0DEED6143CF931A35751C0A961948260&partner=rssnyt&emc=rss AusAID, 36th meeting, 29-30 May 2002, progress report: review on gender and evaluation, Evaluation and gender, retrieved on 20th Oct, 2008, http://www.oecd.org/secure/pdfDocument/0,2834,en_21571361_34047972_31705464_1_1_1_1,00.pdf Aus AID, 32nd Meeting 10-11 February 2000, Evaluation of development assistance for poverty reduction workshop report, retrieved on 20th Oct, 2008, http://www.oecd.org/secure/pdfDocument/0,2834,en_21571361_34047972_31967651_1_1_1_1,00.pdf Missingham, B. iCAM Working Paper 2000/01. Participatory Development in Thailand: A Review of Some Relevant Literature, retrieved on 20th Oct, 2008, https://dspace.anu.edu.au:8443/bitstream/1885/40082/1/icam01_MissinghamWorkingPaper2.PDF Aus Report.16th June, 1999. Quality Assurance Series, Targeting Poor Farmers: Contributions to Rural Development in Thailand, Commonwealth of Australia 1999, Retrieved on 20th Oct, 2008, http://www.ausaid.gov.au/publications/pdf/qas16_thailandfarmers.pdf Leonard R. Thailand’s Land Titling Programme: Securing land for the poor? , retrieved on 20th Oct, 2008, http://dlc.dlib.indiana.edu/archive/00001150/00/Rebeca_Leonard.pdf International Development Research Centre. 21 July 1997. Evolution of Decentralization: Documents and Literature Review, retrieved on 20th Oct, 2008, https://idlbnc.idrc.ca/dspace/bitstream/123456789/23558/2/ch1.html Songco A J. February 2002. Do Rural Infrastructure Investments Benefit the Poor? Evaluating Linkages: A Global View, a Focus on Vietnam, retrieved on 20th Oct, 2008, http://wwwwds.worldbank.org/servlet/WDSContentServer/WDSP/IB/2002/03/22/000094946_02031010384555/Rendered/PDF/multi0page.pdf IFAD Evaluation.26th Feb, 2007. Thailand: Agricultural Diversification and People's Irrigation Project in the North, retrieved on 20th Oct, 2008, http://www.ifad.int/evaluation/public_html/eksyst/doc/prj/region/pi/thailand/r201thae.htm Thailand, National Report, Nov, 2003, Review of Protected Areas and Development, retrieved on 20th Oct, 2008, http://www.mekong-protected areas.org/thailand/docs/thailand_nr.pdf Previte J, Pini B, McKenzie H F.2007. Q Methodology and Rural Research, European Society for Rural Sociology. Sociologia Ruralis, Vol 47, Number 2. Rossi H P, Freeman E H, Lipsey W M.1999.The social context of evaluation, Evaluation: A Systematic Approach, Sage, Newbury Park, Calif., pp. 397–416. Read More