Evaluation of Integrated Humanitarian Settlement Strategy - Case Study Example

A Critical Analysis of the Methodology Applied in Reporting xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Evaluation of Integrated Humanitarian Settlement Strategy xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Name xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Tutor xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Date Executive Summary Evaluation is a very important step in planning and actualizing any program or project. It acts as a means through which errors and any form of inconsistency is identified and a plan on how to rectify it is developed. IHSS has relies on evaluations to redefine her operation and solve potential problems. This report focuses on the effectiveness of methodology applied during the combined evaluation of IHSS (Integrated Human Settlement Strategy) and commonwealth funded services as indicated in the report. This evaluation was conducted by Urbis Keys Young commissioned by DHA (Department of Health and Aging) in conjunction with DIMIA (Department of Multicultural and Indigenous Affairs). The concentration is on the methodology applied and its effectiveness. Introduction Project evaluation is a very important step in running any project. It involves the assessment of how well the needs of the stakeholders are met and the ability of a developed plan to enhance the attainment of set goals and objectives. The main purpose of conducting an evaluation is to assess the performance of a program for accountability, program development, program clarification, program improvement or enlightment (Owen 1993, p. 14). The success of an evaluation is determined by the method of evaluation applied, availability of resources and the ability of evaluators to keenly apply accurate and precise methods. This means that evaluators must be very keen in determining the methodology to be applied as well as methods of analyzing data to enhance precision and accuracy (Owen 1993, p. 18). An applied method can be said to be effective when it results to environmental benefits, operating efficiencies, improved mobility and when it is cost effective. A very costly and lengthy evaluation does not necessarily mean that the evaluation is accurate and precise. The evaluation of commonwealth funded services and IHSS (Integrated Humanitarian Settlement Strategy) is an example of a precise evaluation. Assessment of IHSS and services that are funded for the purpose of assisting trauma and torture survivors was commissioned by DHA (Department of Health and Aging) in conjunction with DIMIA (Department of Multicultural and Indigenous Affairs). This report was aimed at the capacity of the services to meet set objectives, identify the need for modifications, determining client satisfaction and determination of the way forward. To begin with, Urbis Keys Young had developed a very clear purpose of the evaluation. According to the report, the purpose of the evaluation was to how well services offered met set objectives while at the same time identifying any possible modification that would enhance distribution of equitable, effective and efficient services (DIMIA 2003, p. 4). The evaluation also aimed at assessing whether the needs of torture and trauma victims were met to a point that would be termed as satisfactory and give recommendation as to how this could be achieved. This explanation made the purpose of the evaluation clear hence enhancing the possibility of producing accurate results. Literature Review An in-depth literature review on IHSS has been given. This dates from the 1970 explaining how settlement of refugees and victims of torture and trauma has been transformed over time (DIMIA 2003, p. 8). It includes an explanation of factors that lead to the transformation from hostels to self-contained accommodation. According to the literature review, it is not clear whether the previous applied methods of assisting entrants were effective. The evaluator simply explains what happened in the previous years but does not explain the short comings of previous systems. Some parts of the literature review are based on assumptions which limits ones ability to draw conclusions as to whether current strategies are better as compared to the initial ones. Failing to identify the cause of failure in the past limits the ability of an evaluation to determine the effectiveness of current strategies in relation to previous strategies (Abe & Wiseman 1983, p. 7). For instance, it is indicated that it was assumed that entrants were supported by the proposers. However, the literature review does not confirm whether this actually happened. Identifying existing gaps and the possible causes of project failure is vital so as to avoid a repetition of similar mistakes. Urbis Keys Young should have identified the effects of such assumption and whether the proposers actually funded entrants. This could have been important when giving recommendations since if a particular effective method has been done away with, it could be reintroduced. The literature review provides key details which are necessary in determining the method to be applied during an evaluation. For instance, it is clear that entrants are retained in the IHSS program for a period of six months unless there is a special need that would take longer to handle. This means that an effective evaluation would have to be done with this period of time in mind. However, information on what happens to these entrains after the six months is missing. Though it is important to offer humanitarian support for the most vulnerable, it is also important to identify a means of enhancing their sustainability so as to reduce the dependency level (Wedeen 2000, p.16). This will ensure that clients are eager and ready to leave the program after six months since they know that they can make end meet through the support given to them. It would even result to more people joining the volunteers group thus increasing the possibility of client satisfaction. Such information would have been important when redefining the program to address existing and potential problems. Conclusion drawn from this data would be inefficient since though those in the program may be satisfied; those who have already left may be suffering. The literature review also fails to provide information on cultural barriers that may be faced by entrants which would hinder them from receiving satisfying servicers (Abe & Wiseman 1983, p.23). Since IHSS recognizes the basic needs for refugees and other entrants, it is also important to identify how best these needs can be met without violating a person’s belief or stand as far as religious and cultural matters are concerned. This means that, though some individuals may not be satisfied, it is not necessarily because they do not have the basic needs but because they may not be in a position to utilize what has been provided due to cultural and religious barriers (Kerlinger 1986, p.31). The best method of addressing a problem is addressing the root cause of that particular problem. Though Urbis Keys Young does not explain main cause of torture and trauma among entrants, it is evident that these causes are on the increase resulting to increased number of entrants. This gap can have a great effect on the conclusions drawn as well as the recommendations given. Collecting information on such matters would be important for budgeting and distribution of funds. However, the overall literature review is well developed as Urbis Keys Young look into the main stakeholders and their contribution to the project. An in-depth explanation on the purpose of the evaluation, main stakeholders and upheld principles has been developed explaining the role of each stakeholder in the project (Torraco1997, p.6). The literature review is well organized as each stakeholder is tackled as a unit. Both IHSS and commonwealth funded services for survivors of torture and trauma are well explained. The methodology applied in the evaluation was an effective one since major stakeholders were well represented. However, this method is biased since Urbis Keys Young has to consult other organizations and service providers before picking the people to be involved in the interviews. The aim of most service providers in humanitarian organizations is to reduce dependency and promote self reliance. From the literature review, it is not clear what happens to entrants who are educated after receiving assistance for the first six months. Assisting them find jobs would go a long way in reducing the dependency level (Torraco 1997, p.34). In addition, there are no clear guidelines as to whether entrants interested in education are given an opportunity to go back to school and develop important skills and careers. These gaps render the literature review inadequate. Secondary data has been used to explain the situation of trauma and torture clients as in the past. This information is useful in understanding the changes that have taken place and in explaining whether the program is moving forward or backwards. In this case, IHSS has improved her services over time though there is still more that is to be done. The data collected is skewed since it concentrates more on matters relating to the client rather than a combination of all stakeholders. It can be said that the data is positively skewed. There is also little graphical representation of the data. Graphical representation is important as it gives one a better understanding of the issue being discussed at a glance (Owen 1999, p.23 ). One has to read the report so as to understand the purpose and results of the evaluation. Stakeholders The main stakeholder in this evaluation is the client who draws the attention of evaluators as they seek to understand his or her level of satisfaction. This means the evaluators assume that the success of the project is exclusively determined by client satisfaction. Bending an assessment on one side may result to neglecting some important factors which could have attributed to poor performance including cultural factors (Ferber 1977, p.23). With this in mind, evaluators should have given a similar weight to surrounding communities, volunteers and project managers in evaluating the effectiveness of services provided. Project beneficiaries tend to be reluctant in criticizing offered services for fear of being denied such services in the future. On the other hand, they may tend to exaggerate matters as a result of having high expectations that more will be provided if they proof how bad the situation is. It is therefore important to seek for the opinion of an independent group and draw a comparison of the results. Methodology Sampling The sampling applied in this report was targeted sampling which is important in determining specific information from a specified group of people. In this case, information can only be collected from stakeholders in IHSS, DIMIA and Commonwealth funded services. These include the clients, service providers, staff members, managers and volunteers who assist in delivering services to torture and trauma victims in Australia. Evaluators had to eliminate any person who did not have any experience dealing with IHSS and Commonwealth funded services. However, the sampling method applied may have been biased since Urbis Keys Young requested AMEP (Adult Migrant Education Program) and service providers to find clients on their behalf. The participants may therefore have responded to the questions in favor of the group that contacted them. A clear inclusion and exclusion criteria had been developed to ensure that only persons with required important information were involved. This reduces wastage of time and other resources (Patton 1990, p.12). Data collection Evaluation websites were utilized in determining the effectiveness of IHSS in meeting set objectives. The use of websites in conducting researches has been found to b effective in that it enhances wide coverage. However, this method has several shortcomings which include collection of limited and incorrect information (Patton 1990, p.12). The interviewee interprets questions according to their understanding which may not necessarily be the information required. During face to face interviews, it is possible to clarify any vague ideas hence assisting the respondent answer according to expected answers rather than according to their understanding of the questions (Fraser 2000, p.17). In addition, collecting information is time consuming as compared to face to face responses since the interviewer has a chance to summarize a response from an interviewee. Important aspects in communication such as body language, intonation and voice are thus lost. This makes it hard to identify when one is giving incorrect information (Denzin & Lincoln 1994, p.65). The issues discussed in the report are very sensitive and there is a high tendency of people giving inaccurate information in such instances. This affects the quality of information collected as well as the conclusions drawn. As one directs the respondent on questions to be answered, in-depth information is collected within a very short duration (Clark 2003, p.54). In addition, some potential participants may be eliminated due to their inaccessibility of adequate internet services and lack of motivation to participate in the evaluation. The language used may also limit their participation unlike in face to face interviews where one is in a position to change the language used in accordance with the age, gender and immediate evaluation of a person’s level of understanding of particular terms (Miles and Huberman 1994, p.32). The application of web evaluation however eliminates the possibility of incurring high travel cost and time wastage as the interviewee and the interviewer seek for an appropriate meeting time and place. Apart from web evaluation, group discussions were held with IHSS clients. These groups received financial compensation though this was not referred to as payment. According to Abe & Wiseman (1983, p.13) offering financial assistance may result to one receiving wrong information as the clients seek to impress the researcher. In addition, this increases the cost of running an evaluation and may negatively affect future evaluations as people participate with the aim of gaining some financial assistance. The language used in the report is technical especially when it comes to referring to different groups of people and services offered. However, understanding the report has been simplified by the fact that a glossary of terms has been provided. Terms have been explained as used in the report to avoid misunderstanding making it easy for the reader to understand what the report is all about. The evaluator draws on human rights and how best IHSS offers its services to torture and trauma victims. This affects the mode of data collection as well as reporting since the issues tackled are very sensitive (Calhoun 1994, p.123). Since it involves a collection of people’s opinions of how best IHSS and commonwealth funded services meet the needs of their clients, qualitative method had to be used. This limits one from collecting quantitative data. Certain methods cannot be used such as random sampling since the research targets information that can only be received from IHSS stakeholders. The evaluator focuses on clients’ satisfaction. He seeks to understand their satisfaction in relation to services provided. Research Method Applied The research method applied was qualitative research. This is because the evaluators sought for people’s opinions rather than statistical data. Application of qualitative research methods may have been driven by the fact that required information was subjective. For instance, the evaluators sought the opinions of clients as to whether the services provided to them were satisfactory (Hall & Hord 2001, p.76). People’s opinions are subjective and this affects the kind of recommendations given. The evaluators also assumed that available secondary data and information received during the consultation phase was objective. This may not be so bearing in mind that the researches were carried out by other people and it is not clear as to whether they accurately applied scientific methods. The results achieved after the group discussions and the interviews could have been affected to a greater extent by people’s behavior and attitudes towards the services offered by IHSS. This method is very advantageous in gaining light on complex phenomena such as level of satisfaction. In this case, application of qualitative method enhances the ability to understand level of satisfaction among the clients despite their past experiences. It seeks to understand this issue from all dimensions including the effectiveness of service providers in delivering services to the clients and the effectiveness of IHSS management. This method can also result to the development of a theory aimed at explaining why a situation is as it is (Neuman, 2003, p.23). For instance, it can be said from the report that service providers fail to satisfy the client due to poor management. In this regard, when the management fails, to compensate service providers in time or to give them a clear guideline of the services that they are required to offer, the service providers automatically offer poor services to the clients. This means that delivery of high quality services to clients can be interpreted to mean that service providers are satisfied with the treatment they receive from the management (Kirkpatrick 1994, p.54). These theories make it easier to interpret a situation in the future. Qualitative methods also give the interviewer an opportunity to probe in-depth explanation based on previous researches. One disadvantage of this method is that it is time consuming resulting to a very low number of samples. The samples involved in the evaluation are not representative. The results obtained cannot therefore be generalized to a wide population (Wilsdon 2001, p.23). The data is also subjective and bears no scientific evidence that the situation is as depicted in the report. Data collected form this method only presents possibilities and never explains the best alternative. Recommendations To eliminate biasness in sampling, Urbis Keys Young should have given the criteria of selection to AMEP to identify all individuals with such characteristics and later collect samples randomly from the list provided. The recommendations given are clear and link very well with the evaluations. They have not been drawn from the blues but are as a result of findings. All recommendations rhyme with the literature review and the findings. This means that the evaluator clearly understood the situation and was accurate in analyzing the data. This adds value to information collected (Owen and St Leger 2004). Stakeholders The main stakeholders in the evaluation included clients or victims of tourture and trauma who had benefited from the program, key staff members from DHA and DIMIA, volunteers, service providers (EHAI/PASTT, IHSS), AMEP,NSW ( Multicultural Disability Advocacy Association). These stakeholders were involved in the research though not fully. In my view, the evaluator sought for the most convenient methods therefore leaving out some important details (Ferber 1977). Conclusion Though evaluation is a very important step in project formulation, it constantly fails to accomplish its purpose when wrong methodologies are applied. Individuals working for organizations and companies should therefore seek to the application of accurate, precise and scientific methods in evaluation so as to enhance success. Without this, similar mistakes will keep reoccurring hence resulting to a wastage of resources. In evaluation, one should not seek to follow the most convenience method only. In fact, the most important aspect of an evaluation is the quality of information to be gained and its value in future development and solving existing problems. Bibiograhy Owen, J. M. 1999. Program Evaluation : Forms and Approaches, 2nd Ed. St Leonards, N.S.W.: Allen & Unwin. Department of Immigration and Multicultural and Indigenous Affairs, 2003, Evaluation of the integrated humanitarian settlement strategy (IHSS). http://www.immi.gov.au/media/publications/pdf/evaluation_report.pdf< retrived in 24 July 2010>. Neuman, WL 2003, Social Research Methods,25 Qualitative and Quantitative Approaches, 5th edn, Allyn and Bacon, Boston, pp. 1–19. Owen, J. M. and St Leger, P. (2004). Evaluating Policy: Trends and Issues. Workshop presented at the Annual conference of the Australasian Evaluation Society, Adelaide. W Wedeen, RP 2000, ‘Consent in epidemiology: Implications of history for public policy’, Archives of Environmental Health, vol. 55, iss. 4, pp. 31–39. Wholey, J., Hatry, P., Newcomer, K., 2004, Handbook for practical program evaluation. San Francisco, John Wiley sons. Abe, H, & Wiseman, R, 1983, A cross-cultural confirmation of the dimensions of intercultural effectiveness. International Journal of Intercultural Relations Kerlinger, N, 1986 Foundations of behavioral research, Orlando, FL:Holt, Rinehart and Winston, Inc. Denzin, N & Lincoln, Y, 1994, Handbook of Qualitative Research Methods, Newbury Park, CA: Sage. Miles, B and Huberman, M, 1994, Qualitative Data Analysis, 2nd edition, Thousand Oaks, CA:Sage. Patton, M, Qualitative evaluation and research methods, Newbury Park, CA:SAGE Publications, Inc. 1990.pp. 1-506. Torraco, R, 1997, Theory building research methods, In R. A. Swanson & E. F. Holton III (Eds.) HRD Research Handbook. San Francisco: Jossey-Bass. Ferber, Robert, 1977, “Research by Convenience,” Journal of Consumer Research, 4, 57-58. Clark, M, 2003, Data networks, IP, and the internet: protocols, design, and operation, Oxyford: Wiley Publishers. Fraser, M, 2000, Free-for-all: the struggle for dominance on the digital frontier, New York: Stoddart Publishing. Wilsdon, J, 2001, Digital futures: living in a dot.com world, London: Earthscan Publications. Calhoun, E, 1994, How to use action research in the self-renewing school, Alexandria, VA: ASCD. Hall, G. & Hord, S, 2001, Implementing change: Patterns, principles, and potholes, Boston. MA: Allyn & Bacon. Kirkpatrick, D,1994, Evaluating training programs, San Francisco, CA: Berrett-Koehler Publishers. Guskey, T. (2000). Evaluating professional development. Thousand Oaks, CA: Corwin Press Inc. Read More