Evaluation of Turkey ’ s PISA Reading Literacy Scores

The purpose of the current study is to evaluate Turkey’s International Student Assessment Program (PISA) reading literacy scores in view of Turkish Language teachers. For this reason the study was designed as a case study, one of qualitative approach. In the research, the data were collected from 15 Turkish language teachers through interviews. An interview form was developed and used as a data collection tool. Content analysis techniques were used to evaluate the research data obtained. As a result of the research, it was identified that most of the Turkish teachers did not follow the PISA results. Teachers regarded to the missing information and indifference as a reason for their incomplete command of this subject. The reason why teachers follow PISA results is to evaluate Turkey’s success in the field of education and to see the educational situation of other countries. Teachers also stated that it is necessary to follow the current developments in education as a requirement of their profession. A number of teachers assessed Turkey’s reading literacy scores in PISA as relatively low by pointing out the society’s and students’ lack of reading habit. In addition, they stated that the education system is not suitable for being successful in PISA. They offered that the education system, curriculum and course books should be updated in order to reach the desired score in PISA. They also suggested that teachers should be informed about PISA and its measurement and evaluation processes.


INTRODUCTION
The Organization for Economic Cooperation and Development (OECD) has implemented the International Student Assessment Program (PISA) for the international assessment of reading, science and mathematics skills of fifteen-year-old students. First administered in 2000 and conducted every 3 years, PISA claims to measure students' academic success on an international scale. The major domain of assessment rotates between reading, mathematics and science in each cycle; however, in 2000, 2009 and 2018, students' reading literacy was aimed to be measured.
In PISA Reading Field, the reading literacy competencies of 15-year-olds, who are either at the end of elementary education or enrolled in the first grade in secondary education, are measured. Within the framework of the reading skills field assessment framework at PISA, answers to the following questions are sought: Is the student able to construct meaning from different text types and formats? Is the student able to search for and find the specific information across the texts? Is the student able to critically assess the given text? (Ministry of National Education, 2010;Tuzlukaya, 2018). The above-mentioned questions guide the assessment of success of students, namely countries in education.
are evaluated in three ways as continuous, non-continuous and mixed texts. The inclusion of different text types and text formats in PISA is due to the fact that students are required to evaluate a broad range of texts which they encounter in their daily lives (Bozkurt, 2016).
In 2003, Turkey participated for the first time in PISA conducted with the participation of approximately 75 countries. In PISA 2003 Reading Literacy Assessment, the OECD average was 494 and Turkey's average was 441 score points. Accordingly, Turkey ranked 34 th among other countries. However, in PISA 2006 Reading Literacy Assessment, the OECD average was 492 and Turkey's average was 447 score points. Accordingly, Turkey ranked 38 th among other countries. In PISA 2009 Reading Literacy Assessment, the OECD average was 493 and Turkey's average was 464. With this result, Turkey ranked 39 th among other countries. In PISA 2012 Reading Literacy Assessment, the OECD average was 496; however, Turkey ranked 42 nd among all countries with 475 points. In PISA 2015 Reading Literacy Assessment, the OECD average was 493 and Turkey's average 428. With this result, Turkey ranked 50 th among all countries in 2015. Finally, in PISA 2018 Reading Literacy Assessment, the OECD average 486. Turkey ranked 40 th among other countries with 466 points. It is seen hat Turkey is below the OECD average in PISA Reading Literacy Assessments. The majority of Turkish students were able to perform at the proficiency level 2 or below in PISA testing (Bozkurt, 2016). Even though there is a point-based increase in students' performance in reading literacy assessment, Turkey's performance has still been seen to fall behind the OECD average in this area. The Turkish education system was updated according to the constructivist learning approach in 2005. However, when the PISA reading results of Turkish students are examined, it is seen that there is no significant development. The reason of this can be shown that the constructivist system cannot be applied fully and in all its aspects. In addition, the fact that teachers carry out measurement and evaluation activities traditionally can be shown as a reason for this situation (Dilekçi and Çiçek;Karatay and Dilekçi 2019). The OECD and Turkey's average in PISA Reading Literacy Assessment by year are illustrated in Figure 1.
PISA was administered with the participation of 43 countries in 2000, 41 in 2003, 57 in 2006, 65 in 2009 and 2012, 72 in 2015 and 79 in 2018. Table 1 shows that Turkey's performance in PISA Reading Literacy Assessment is not at higher levels by year.
As seen in Figure 1 and Table 1, since Turkey's average in PISA reading tests varies each year, it may be concluded that Turkey is not able to set a standard of ranking among all countries in PISA Reading Literacy Assessments.
In Turkey, language courses at schools mainly centre upon reading skills. Teachers focus on students' reading comprehension skills through course books and certain texts. Nevertheless, Turkish students are not able to acquire expected results in PISA Reading Literacy Assessments. Therefore, Turkey's performance may be said not to be higher rankings in PISA tests. In order to increase their performances in PISA, countries have amended their education systems, teacher training processes, curricula and assessment and evaluation processes (Benzer, 2020;Koç, 2021). Likewise, Turkey has made certain amendments in various educational elements with the aim of improving its performance in PISA.
There have been numerous studies concerning PISA in the literature. These studies investigated Turkish Language Course Curricula (Batur and Ulutaş, 2013;İnce, 2016;İşeri, 2019;Karabulut, 2017;Koç, 2021;Sefer et al., 2017;Temizyürek and İnce, 2019), Turkish course books (Benzer, 2019;Bozkurt et al., 2015;Coşkun, 2013;Yağmur, 2009) and transition to secondary education examinations (Aşıcı et al., 2012;Batur, Ulutaş and Beyret, 2019;Savran, 2004;Tuzlukaya, 2019) within the context of PISA reading skills criteria. In addition, reading skills course was examined based on reading comprehension skills in PISA (Batur and Alevli, 2014). Besides, a study aiming to assess Turkey's ranking in PISA 2009 among OECD and other countries was conducted (Acar, 2009). However, no study regarding teachers' views on Turkey's reading literacy scores in PISA has been encountered. As a result, it is of great importance that Turkey's success in PISA is evaluated and different views are explored in order for students to be more successful.

Objective and Research Questions
Apart from previous studies, the current study aims to reveal how Turkish Language teachers assess the situation in Turkey's reading literacy scores in PISA and to put forward proposals in order to improve Turkey's performance in PISA Reading Literacy Assessment. Within the framework of the main research question 'What are teachers' views on Table 1. Turkey's ranking in pisa reading literacy assessment by years Turkey's reading literacy scores in PISA?', within the framework of the main problem, answers to the following research questions were sought: 1. What is the status of Turkish Language teachers to follow PISA reading results? What are their justifications for this situation? 2. How do Turkish Language teachers assess the situation in Turkey's reading literacy scores in PISA? 3. What are Turkish Language teachers' proposals in order to improve Turkey's performance in PISA Reading Literacy Assessment?

Research Design
The present research is designed as a case study. In a case study, questions in an interview form are addressed to the participants. The case study approach enables the researcher to learn from the participants' experiences and, by doing so, to identify their perceptions (Gillham, 2000). A case study allows the researcher to explore and explain the case, thereby guiding further implementations (Merriam, 1990). In the research, how Turkish teachers evaluate Turkey's PISA reading literacy scores and how Turkey should perform well in the PISA reading exams were discussed.

Study Group
15 Turkish teachers working in secondary schools constitute the study group of the research. The teachers were selected for the study group according to the convenient sampling method, one of the purposive sampling methods. In the convenient sampling method, easily accessible and accessible respondents are used. For this purpose, a study was conducted with 15 Turkish teachers working in secondary schools in Bolu. Table 2 shows the information (gender, age, level of education and professional seniority) of the participants. As seen in Table 2, seven of the teachers are male and eight are female. Of all Turkish Language teachers, 2 of them are 21-30, 12 are 31-40 and one of them is 41-50 years old. As for their levels of education, 12 Turkish Language teachers have bachelor's degree and 3 have master's degree. In addition, two of them have 1-5, 2 have 6-10, 8 have 11-15 and 3 have 16-20 years of professional seniority.

Data Collection Procedure
Data were collected with a semi-structured interview form. The researcher made a review of the literature to develop the interview form. Three open-ended questions were asked in the interview form to guide explanation and detailed conversation. The interview questions were concerning the status of Turkish Language teachers to follow PISA reading results, Turkish Language teachers' opinions on the situation in Turkey's reading literacy scores in PISA and their proposals in order to improve Turkey's performance in PISA Reading Literacy Assessment. The interview questions developed by the researcher were directed to 15 teachers face-to-face. The opinions of participants were recorded through note-taking technique.

Data Analysis
The data collected from the interview form were analyzed according to content analysis, one of the qualitative data analysis methods. In content analysis, it is aimed to encode similar data and categorize them in the light of certain themes (Yıldırım and Şimşek, 2013). To this end, the main theme and sub-themes are first identified (Balcı, 2009). In the research, the data obtained from teachers were examined by two field experts and encoded under appropriate theme and sub-themes. The frequencies and percentages of themes were tabulated and interpreted. Teachers' views were presented under the themes.

Validity and Reliability
In the validity of the research, it is of importance that information is given about data collection and analysis (Yıldırım and Şimşek, 2013). The research processes were recorded in details. In order to ensure validity, the questions to be asked to the participants were determined by scanning the literature and the interview form was finalized after the evaluation of the two experts.
The collected data were evaluated as objective for the reliability of the study. In order to ensure the consistency, reliability and accuracy of the study, two field experts evaluated the data independently of each other. According to Miles and Huberman (1994), inter-rater consistency should be above 70%. In this study, it was determined that the evaluation consistency of the experts was 80%. This result shows that the study is consistent, confirmable, and reliable. For addressing the disagreement in the data, the interview forms were re-considered. In case there was a disagreement in the data, negotiations were made toward consensus, thereby ensuring the reliability of the study.

FINDINGS
The findings of research regarding the opinions of teachers on Turkey's reading literacy scores in PISA are covered in this section.

st Sub-Question
As concerned with the first sub-question, the status of Turkish Language teachers to follow PISA reading results were analysed and their justifications concerning this situation were addressed. The findings are given in Table 3. 15 teachers have been participated in the study. 8 of them stated that they did not follow PISA reading results; however, 7 of them reported that they followed PISA reading results.
As shown in Table 4, the status of Turkish Language teachers to follow PISA reading results were presented under two main themes as 'Reasons for Following' and 'Reasons for Not Following.'

Reasons for Following
Six teachers stated that they were interested in PISA reading results in order to examine Turkey's status in international education: "I think that we should follow as required by profession and to examine to what extent our country's situation is in education among other countries" (K2); "I carried out a TUBITAK project concerning this topic, thereby having the opportunity to follow" (K3); "I follow our country's rank, examine questions and, accordingly, take notes for myself about what to do" (K4); "I follow our country's position in education on an international scale " (K10); "I think that PISA is of importance in terms of assessing to what extent students are able to use their reading skills which they acquire at school" (K14); "It is important to see our country's reading skill performance over the years and this guides the precautions to take in this situation" (K15).

Reasons for Not Following
Four teachers expressed their opinions representing insufficient information and their indifference: "I did not have a full command of this subject" (K7); "It does not draw my attention" (K8); "There is not sufficient information" (K4); "I do not follow because the goal of our exams and that of PISA are different, ours is elimination-oriented, but theirs is process-oriented" (K15).

nd Sub-Question
In the second sub-question, Turkish Language teachers' views on Turkey's reading literacy scores in PISA were analysed. The findings are presented in Table 5.
When Turkish Language teachers' views on Turkey's reading literacy scores in PISA were investigated, it was revealed their opinions were defined under 'Failure' theme.
Failure 13 teachers reported that Turkey underachieved in PISA examinations: "I think that our country' scores in PISA are not satisfying and far behind other countries; Turkey's scores have clearly been seen to deteriorate year by year" (K2); "I think that Turkey's results in PISA were not at the desired level despite heavy investment into education, and the results would be better " (K3); "As far as I am concerned, our country's scores in PISA are considerably low " (K4); "In my opinion, Turkey's reading literacy scores in PISA are low and, in addition, our country experienced a decrease even in the years when its scores should be higher " (K5); "Regarded as a natural outcome of a society disliking reading, our country' s ranking is one of the lowest ones " (K6); "Despite having a slight increase till 2012, Turkey's scored low in terms of ranking in PISA " (K8); "The results show a dramatic decrease in each PISA cycle " (K9); "Turkey's ranking is significantly low in spite of an increase in its scores at first glance, referring to a deterioration in our reading literacy results …" (K10); "Except from the last PISA cycle, our country showed a significant decrease in terms of ranking…" (K13); "Turkey exhibited a gradual improvement from 2003 to 2015; on the contrary, the results in 2015 imply a considerable decline. However, our country has improved its mean score in 2018 and this may be due to the alteration of question types used in transition to secondary and higher education examinations. In the last PISA cycle, the fact that Turkey's performance and ranking have improved can be regarded as a positive situation; however, the country's being far behind the OECD countries poses a serious problem…" (K14); "It can be said that Turkey, as one of the founding members of the OECD, is far behind the OECD countries' average, referring to Turkey's insufficiency to  Examining other countries 10 1 7 As part of a project 3 1 7 Reasons for Not Following Insufficient information 7, 9 2 13 Being different from exam concept held in Turkey 13 1 7 Indifference 8 1 7 making 15-year-old students acquire higher-order reading skills…" (K15).

rd Sub-Question
As concerned with the third sub-question, Turkish Language teachers' proposals in order to improve Turkey's performance in PISA Reading Literacy Assessment were investigated. The findings are given in Table 6. Turkish Language teachers' proposals in order to improve Turkey's performance in PISA Reading Literacy Assessment were presented under four main themes identified as 'reading activities', 'the revision of the system and elements of education', 'the teacher factor' and 'environmental factors.'

Reading Activities
Nine teachers emphasized that Turkey's reading literacy scores in PISA may be increased by carrying out reading activities: "In my opinion, reading and comprehension should be centred upon in question -solving activities and the society do not have a reading culture which I think should be improved" (K2); "We should make reading more appealing" (K4); "Families should, first, be educated in order to generate reading habit; reading should be turned into a fun activity and we should advice suitable books for our students' interests" (K5); "Families should make their children acquire reading habit, certain books drawing the child's attention should be provided at early ages and rather than turn reading into a job, it should be a fun activity" (K6); "Certain educational activities in and outside the school should be arranged in order to make children acquire a regular reading habit as from the early childhood" (K10); More time should be allocated to reading and comprehension" (K11).

The Revision of the System and Elements in Education
Six teachers said that education system, curriculum and course books should be revised: "Definitely improving the institutions responsible for teacher training, and the curriculum should be more clear and relevant to daily life. Instead of continuously changing system, a national education system It is below the OECD average. 14, 15 2 13 The country's does not set a standard in ranking. 8, 15 2 13 The country's mission in education is insufficient. 7 1 7 The assessment and evaluation process is different. 13 1 7 The duration of mandatory education is too long. 9 1 7 Focusing on reading and comprehension in measurement tools. 2 1 7 The Revision of the system and elements in education Re-structuring the education system. 3, 7 2 13 Raising students with inquiry skills. 5, 14 2 13 Improving the quality of course books. 9 1 7 Improving the institutions responsible for teacher training. 3 1 7 Altering assessment and evaluation processes. 13 1 7 The Providing family support in education process. 7, 15 2 13 Diminishing the impact of socio-economic gap. 14 1 7 searched for years with an infrastructure should be launched" (K3); "We should raise generations being equipped with inquiry skills" (K5); "I think that an improvement, at the least, are made with a novel educational vision" (K7); "Mother tongue education should be restructured" (K8); "The quality of course books in terms of literature should be improved" (K9); "The curriculum should completely be altered" (K10); "Certain alterations in exam contents can be made" (K13).

The Teacher Factor
Four teachers reported that teachers should be informed about PISA and certain attempts regarding their developments should be made: "An overseas experience should be provided to the individuals gaining training in order to develop them socially and culturally and teachers' motivation concerning self-improvement can be enhanced" (K1); "The concept and significance of PISA should be explained more clearly in our schools" (K8); "Teachers should be informed about PISA questions and results" (K11); "The teaching profession should be converted into a continuously developing profession, thereby offering students the opportunity to experience an education process meeting the necessities of our time; in addition, certain educational environments should be generated in a way that the effect of socio-economic gap is eliminated and skill-oriented approach should be adopted in education " (K14).

Environmental Factors
Regarding environmental factors, four teachers stated that families and the society are required to support education process: "I think education should be regarded as a social problem" (K2); "Students should be supported by their families" (K7); "Interaction networks where a reading culture generated in pre-school period is taken into account by families, non-governmental organizations, municipalities and other segments of the society should be developed" (K15).

In this research, Turkish Language teachers' views on
Turkey's reading literacy scores in PISA were investigated by qualitative data analysis methods under certain themes. It was concluded in the study that the majority of Turkish Language teachers did not follow PISA reading results. With respect to this situation, teachers highlighted insufficient information about PISA and their indifference to this subject. As for the reasons of the status of teachers to follow PISA reading results, they reported that they followed PISA results to examine the status of Turkey and other countries in education services. Besides, teachers said that current developments should be followed as required by profession. In addition, it was concluded that the minority of teachers addressed to assessment and evaluation activities in class by making use of sample questions in the light of reports. Bozkurt (2016) assessed Turkey's reading literacy scores in PISA 2003, 2006 and 2009 and found that the general average of correct answers to questions for Turkish students was approximately 51%, which was considerably below the OECD average. Aydın et al. (2011) found that higher rate of Turkish students had lower level of reading skills, and a small rate of the students were able to accomplish high levels of reading skills. Likewise, in the current study, teachers stated that Turkey's reading literacy scores in PISA were relatively low. It may be said that Turkish students demonstrate low performance in PISA reading field and, therefore, Turkey are far behind the OECD countries participating in PISA in the field of reading skills. In addition, considering its rankings among other countries over the years, it can be seen that Turkey is not able to set a standard in PISA reading field.
When examining Turkish students' achievements based on question types, it was observed that majority of Turkish students obtained a relatively lower average of correct answer for open-constructed questions (Aydın et al., 2014). Bozkurt (2016) determined that Turkish students had difficulty particularly in short response and open-constructed questions and interpreted this situation as a deficiency in the writing skills. Demir (2010) stated that Turkish students' achievement levels in semi-structured questions were lower; on the contrary, they had higher performance with the multiple choice questions. The fact that Turkish students exhibited higher performance on multiple choice questions rather than open-constructed ones may be due to the broad use of the type multiple choice questions by teachers at schools. Nevertheless, Karatay and Dilekçi (2019) found that Turkish Language teachers used multiple choice questions most in the exams. Teachers in the present study evaluated Turkish students' performance in PISA although they did not comment regarding the questions types used in this assessment.
The previous studies investigating Turkish Language and literature curricula based on PISA reading literacy criteria have shown that learning outcomes overlapped to low-level objectives of PISA (Batur and Ulutaş, 2013;Demiral and Menşan, 2017;İnce and Gözütok, 2016;İşeri, 2019;Karabulut, 2017;Koç, 2021). Batur and Ulutaş (2013) examined to what extent reading comprehension objectives in Turkish Language teaching program overlapped to the proficiency levels defined in PISA 2009 test and determined that the majority of objectives planned for Turkish courses addressed to 1 st and 2 nd proficiency levels in PISA. Yağmur (2009) found that higher order thinking processes were not included in course book activities. Bozkurt et al. (2015) classified the comprehension questions in Turkish textbooks according to PISA cognitive levels and specified that the rate of questions representing the level of access-retrieve was 50%; the rate of questions representing the level of integrate-interpret was 27%; and the rate of questions representing the level of reflect-evaluate was 23%. Likewise, Benzer (2019) determined that basic questions with the level of access-retrieve were mainly encountered in Turkish textbooks. In this research, teachers highlighted that Turkey's education system was not suitable for being successful in PISA. In addition, they stated that education system, curriculum and course books were required to be revised in order to obtain higher results in PISA. This finding is in compatible with the results of previous studies in the literature and teachers' views on curriculum and course books. Regarded as a reason for students to be unable to get higher scores in PISA reading tests, the questions concerning curriculum, course books and texts are partially in line with PISA criteria. Despite improving schools' physical conditions, providing smart boards to schools and compensating teacher shortage, it may be said that Turkey is not at the desired level. Teachers recommended reconstructing the education system through which students with inquiry skills in accordance with 21 st century skills are raised in order to find solutions to current problems and to attain international education qualifications.
When examining the questions in transition to secondary education examination based on PISA cognitive processes, it was determined that questions mainly focused on the level of integrate-interpret; however, based on PISA proficiency levels, they were observed to be at 1a, 2 nd , 3 rd and 4 th levels (Aşıcı et al., 2012;Tuzlukaya, 2019). Assessment and evaluation and preparatory activities in schools particularly address to ensure student achievement in transition to higher education examinations (Karatay and Dilekçi, 2019). Accordingly, Demiral and Menşan (2017) teachers aimed to measure knowledge and basic language skills through the questions in order to assess 8 th grade Turkish Language course. Similarly, in the current research, it was concluded that the minority of teachers attempted to address to question types, cognitive processes and proficiency levels in PISA in assessment and evaluation activities in class by making use of sample questions in the light of reports. Teachers were revealed to have difficulty in writing questions although they exerted efforts to prepare exams addressing to different cognitive processes, proficiency levels and question types (Demiral and Menşan, 2017;Karatay and Dilekçi, 2019). The fact that assessment and evaluation activities in teaching process are completely in accordance with PISA criteria may contribute to Turkish students' achievements in PISA.
Having reading habit ensures students to be successful in exams (Acıyan, 2008;Gallik, 1999;Karatay and Dilekçi 2020;Tatar and Soylu, 2006;Sallabaş, 2008;Yaman and Süğümlü, 2010). Previous studies have shown that one of the factors affecting countries' performance in PISA is a social culture placing emphasis on literacy (Çobanoğlu and Kasapoğlu, 2010). Similar to the findings in the literature, teachers in this research referred to reading habit and culture as a main factor prepossessing student and country achievement in PISA. As a reason for Turkish students' underachievement in PISA, teachers emphasized the lack of reading habit in society. In fact, it has been determined in the literature that Turkish students have difficulty in reading comprehension (Anılan, 2004;Baydık, 2011;Çaycı and Demir, 2006;Temizkan and Sallabaş, 2011;Topuzkanamış and Maltepe, 2010). In order to eliminate the problems in reading comprehension and to get higher scores in PISA, teachers stated that efforts should be made to make students acquire reading culture. In this regard, activities in and outside the school are required to be arranged to make students acquire reading culture from the early childhood. Family support should also be provided in the process of achieving reading culture (Acıyan, 2018; Clark, 2010; Doiron and Asselin 2011; Gaona and González, 2011;Karatay and Dilekçi, 2020;Oriogu, 2015;Ivey and Broaddus, 2001;Zickuhr et al., 2012). The fact that students with reading habit encounter with different text types and formats facilitates reading continuous and non-continuous texts, generating meaning from the texts in international reading tests and retrieving relevant information from the text. Therefore, students may likely be more successful in such tests.
According to the literature, it has been seen that education expenditures have a positive effect on countries' academic achievement in international examinations (Turanlı and Deniz, 2008). Turkey is one of three OECD countries where socio-economic background affects student achievement most (Bozkurt, 2014). Moreover, equality of educational opportunity are regarded among the factors determining countries' achievements in PISA (Çobanoğlu and Kasapoğlu, 2010). In this research, teachers address to the effect of the budget allocated for education on education itself. Yet, teachers emphasized that providing appropriate educational environments to reduce the effect of socio-economic gap would improve Turkey's performance in PISA.
According to Demiral and Menşan (2017), training on writing test questions in line with students' levels and the skills required by PISA assessment can be carried out for teachers. Similarly, in this research, teachers recommended general information concerning the concept of PISA and its assessment and evaluation processes to be provided. Thus, in-class activities and assessment and evaluation processes can be comply with PISA. It may be said that informing teachers about these subjects can contribute to teaching processes. It is suggested that the format and genre of texts in the textbooks should be designed according to PISA, and the measurement tools should be arranged according to the cognitive processes and competencies of PISA.

CONCLUSION
In order to increase students' scores in PISA Reading Literacy Assessment, following recommendations may be taken into account: • The curricula and course books may be revised on the basis of PISA reading criteria explained with cognitive processes, proficiency levels, text format, and text genre. • Assessment and evaluation may be in compliance with PISA reading field. • Training on PISA may be given for teachers.
• Reading activities both in and out of the school may be carried out to increase students' scores in PISA reading field and to make them acquire reading habits.