Abstract
In this paper, we show an unintended effect of the program Ser Pilo Paga (SPP) that was a flagship program of the Colombian government between 2014 and 2018. It was designed as an intervention in the Colombian Higher Education System (CHES) by awarding, in the steady state, individual funding to about 40,000 students. Every year, 10,000 new students were chosen from the best applicants in the top decile of the population in the entrance exam to higher education in Colombia that also came from families that live under the level of poverty according to a national survey. Our approach, based on an intensive study of the changes in the statistical distributions of the exam scores during these four years, provides evidence of student performance improvements not only of the beneficiaries of the program, but also of the whole student population. This shows that the program opened similar opportunities for all the students, especially for the poorest ones. The program drove a reduction in the gap between students of the upper strata of the population and those of the lowest strata that usually did not access a high quality institution of higher education due to the lack of funding. This result has opened a debate about the optimal way of funding higher education.
1. Introduction
In October 2014, the President of Colombia announced the program Ser Pilo Paga (SPP for the Spanish acronym) (the name of the program uses the Colombian adjective “pilo” that denotes a particular characteristic of a person, which will be explained later in the introduction). SPP is one of the programs designed and implemented by the Minister of Education (Gina Parody, coauthor of this paper) to contribute to the objective of turning the country into the most educated country in Latin America by 2025. SPP focuses on a specific group of academically high achievers that had no access to a higher education due to their low income [1–3].
In this manuscript, we report an unintended effect in the Colombian education system that was produced by this program. As we will analyze below, the SPP program has had some direct and some indirect effects in the Colombian Higher Education System (CHES), such as a clear improvement in the results of the entrance examination to the CHES, not only of the SPP students, but also of the whole student population.
SPP is the result of a previous research, conducted by the authors of this article, that concluded with the design and implementation of the program. The design of the program articulates efforts from the following four institutions of the Colombian system: (i) the Colombian agency (ICETEX) created in the 1950s to offer financial loans to pay higher education of individual students; (ii) the agency (ICFES) that has designed a compulsory entrance examination to access higher education, now named SABER 11, since 1968; (iii) the accreditation commission (CNA) created by law in 1992 that accredits the quality of the academic programs of both the official and the nonprofit private academic institutions; and (iv) the National Planning Department (DNP) that has run a national survey that characterizes groups to focus the need of subsidies (SISBEN) since 1995.
The origin of this research was the observation of a reduction in the quality of the students that access the CHES. In particular, we observed that the proportion of students that entered the CHES with a high score in SABER 11 was reduced from 32% to 16% in the first 14 years of the 21st century. This tendency implied that somehow there was a reduction of the proportion of talented students that enter the CHES. Therefore, we had to find a way to reverse that tendency because we felt that a characteristic of the quality of a system of higher education is the access of highly qualified students [4]. Furthermore, from a point of view of “justice as fairness,” it is important to educate this set of students so that they can contribute to the progress and fairness of society [5].
In the context of the reduction of the proportion of students with high scores in SABER 11, we studied the set of students with a score in the entrance examination of one and a half standard deviations above the mean. We found that the number of students in this set was 7.5% of the total number of students that had taken the examination. In the subset of this 7.5% of students, we studied the access to the CHES of the students with a SISBEN score under the level of poverty. The result was that 32.5% of the students in the subset did not enter the CHES in the year immediately after they had taken the examination in spite of the fact that they had the academic merit.
Hence, the result of our analysis was that around 14,000 students did not access the CHES in an accredited institution in spite of the fact that they had the merit. With this fact, we designed the SPP program to reverse the situation. In this paragraph we will clarify our understanding of the characteristics of a student that we term “pilo,” which has been studied in a similar way in other contexts [1, 4, 6]. The Colombian expression “pilo” characterizes a student that believes that if he/she makes the necessary effort, then he/she will construct the context to achieve high academic standards. SPP means that if he/she has the capacity to make the effort to construct the context in which he/she will achieve the academic objectives that he/she has declared for him/herself, then he/she will have the financial support to achieve a professional degree.
Thus, SPP is a merit-based program that offers financial aid that covers full tuition costs and some living expenses for each student. The program benefits 10,000 new students every year for four years. To be eligible, a student should fulfill three requirements: (i) a score in SABER 11 in the top 7.5% of the population, (ii) a score in SISBEN under the level of poverty in the area where he/she lives, and (iii) acceptance into one of the Colombian accredited universities (public or private).
The financial aid is administered by ICETEX. The aid is a loan that will be condoned when the student is awarded his/her professional degree.
In this paper, we show one unintended effect of SPP that is related to the overall improvement in the scores of SABER 11. We studied the cohorts of students that took SABER 11 in the second semester of the four years from 2014 to 2017. Among others, we studied the evolution of simple statistical measurements like mean and standard deviation in the four years of the program.
With the observation of the evolution of these measurements we studied the performance, and its improvement, of three different sets of students: (i) the entire group of students that took the entrance examination; (ii) the set of students that took the entrance examination and that had a score in the SISBEN survey that set them under the level of poverty, henceforth named SISBEN students; and (iii) the set of students that could not be beneficiaries of SPP due to their SISBEN score (it was higher than the level of poverty or did not have a SISBEN score), henceforth named non-SISBEN students.
In the case of the SPP beneficiaries, we separated them into three subsets, corresponding to the three geographical areas defined by the survey: (i) 14 biggest cities of the country, (ii) urban cities and towns not included in the 14 biggest cities category, and (iii) rural areas.
With the analysis of the measurements in this article we show that the beneficiaries of SPP drove the system in two ways: (i) every year the performance of the SPP students was better than in the previous year and (ii) there was an increment of the speed of the improvement of the whole student population.
For the analysis we developed a Pareto approximation of the right tail of the distribution. In the analysis we compared the temporal variation of the performance and the improvement of the different sets of students.
After this introduction (Section 1) we have organized the article in the following three sections: (i) in Section 2, we explain the data and the methodology that we used to study the performance of the population; (ii) in Section 3, we present the results obtained from our analysis; and, (iii) in Section 4, we analyze the competition between students to show the results and possible future consequences due to this intervention.
2. Materials and Methods
2.1. Beneficiary Conditions and the Data
In the study we use the data of the results of SABER 11. ICFES (these administrative data are provided by ICFES after approval request) provided the data of the students that took the test in the second semester of each of the four years from 2014 to 2017. In each cohort there are approximately 575,000 students. We used a weighted average of the performance of each student in different components of the test. In future studies, we may analyze the student performance in individual components of the test. ICFES provides the scores in a range from the minimum of 0 to the maximum of 500, and for each cohort we found the minimum score required to be a beneficiary, namely, above one and a half standard deviations from the mean. The minimum scores in the examination were (i) 310 in 2014, (ii) 318 in 2015, (iii) 342 in 2016, and (iv) 348 in 2017.
In the database we also included scores of SISBEN. DNP provided the scores of SISBEN to determine the socioeconomic characteristic of the set of beneficiaries, as it classifies the socioeconomic situation of the student. The SISBEN is further categorized according to three geographic areas, so that the score is different for each area. The three areas and their respective SISBEN scores are (i) 57.21 for the 14 biggest cities of the country, (ii) 56.32 for the urban areas not included in the 14 biggest ones, and (iii) 40.75 for rural areas.
Each year, the CNA provides the list of institutions that have an institutional accreditation.
For the analysis of the beneficiaries we also use administrative data of the program. This information allows us to compare a variety of properties of the students in the three different areas of the SISBEN students, and their statistical analysis is presented in the article. The SPP program selects 10,000 new students every year.
2.2. The Measurements of Performance and Improvement
For the analysis we constructed the sets of the whole population that took the entrance examination (SABER 11) in the second semester of the years 2014 to 2017. In each of these sets we identified two subsets: (i) the set of students with a SISBEN equal to or less than the score for a person to become a possible beneficiary of SPP and (ii) the set of students with a SISBEN greater than the score for a person to become a possible beneficiary of SPP. For each year we identified the subset of students in set (ii) that were beneficiaries of SPP. In each of these 4 subsets, one for each year, we further separated them into the three subsets of the beneficiaries for each of the three classification areas of the SISBEN. In each of these 12 sets of students we conducted a basic descriptive statistical analysis to study the performance and the improvement of each set based on the right tail of probability distribution of the scores of the students. To study the performance and the improvement of a set of students, we proposed our own definition of these two quantities. Our assumption, as we may observe from the data, is that analyzing the set of students that are classified in the right tail of the distribution provides a strong hint of the behavior of the entire population. We also complemented our analysis using common statistical measurements like the mean and the standard deviation. Moreover, we made a special analysis of the subsets of students in the three areas of the SISBEN observing the changes in the SPP scores.
As we have said, studying the tail of the distributions allows us to measure the rate of performance and improvement in time. We consider that the right tail of the distribution of the examination scores has a statistical behavior similar to a Pareto distribution. The Pareto distribution is defined by the function . The function represents the probability density distribution of the variable , which in this case represents test scores, while represents the parameter of the distribution. In other words, gives an idea of how “extreme” the values of the scores are. The greater of the distribution is, in general, the lower the scores we may expect. In other contexts, power-law distribution has been widely accepted to describe the wealth distribution in different economies, as well as to characterize other extreme or critical phenomena like earthquakes, the Internet network, and so forth [7–29]. If for two sets we obtain , we may deduce that members of set 2 have obtained in general lower scores on the examination, while students of set 1 have obtained in general higher results because the distribution has more “extreme” scores. Therefore, we propose to use the exponents as an estimation of a measurement of performance:
According to our approach, the fraction should convey a reasonable notion of performance. We expect that populations described with a lower value of have in general better results than populations with a higher value of , so that the expression of (1) will be greater than in the second cases. In Figure 1(a) we show an example for the scores of two sets of students that can be described by Pareto distributions with and , respectively. Although both sets of students have similar mean, the set characterized with (the blue ones) has more people with higher scores than those characterized with (red). In the same way, this figure suggests that if the test would consider higher examinations scores, on average the first population (the blue ones) may obtain higher scores. Because we are considering normalized distributions (i.e., ), it is then possible to compare sets with different population sizes.
(a)
(b)
(c)
Figure 1(b) exhibits an example of the evolution of the parameter that characterizes the probability distribution of the test scores of one of the populations shown in Figure 1(a). In other words, the parameter provides a measurement of the set of students for a specific socioeconomic score of a SISBEN group that was studied during 12 years using the expression of (1). In this particular example, the scores of the population present different fluctuations indicating that the parameter is not monotonously growing. We can now define a measurement of improvement as the rate of change of the performance in time, or
In a discrete approach, the expression may be interpreted as
Assuming that for all , if for a particular examination year (the reader may notice that refers to the year in which scores are obtained; in this sense, evaluates the difference between examinations of two consecutive years, denoted at the last one), from (3) we obtain that and, henceforth, there is an improvement in performance in the test scores of a particular set. Otherwise, if , we expect that the results for the group got worse. Figure 1(c) shows the measurement based on (3) of the time series shown in Figure 1(b). As the reader may notice, the fluctuation in time of the measurement, and, therefore, in the parameter , may be interpreted as a performance speed in analogy with the cinematic concept. This suggests that other derivatives may be studied in the future, such as the acceleration, that may provide additional information.
3. Results
3.1. SISBEN vs. Non-SISBEN Performances
Figure 2 shows the probability density function of the SABER 11 examination scores from 2014 to 2017, for the overall population, for the students with a SISBEN score that would establish that they could be beneficiaries of the program and the population that would not be beneficiaries of the program. These figures summarize the temporal evolution of the distribution of the scores of the students. Figure 2(a) shows the performance of the evolution of the overall population. Macroscopically, we immediately observe that the distributions of 2014 and 2015 are different from the following two years. Although it is not shown here, it is possible to observe that the distribution for the years before 2014 exhibits a macroscopic behavior similar to that of 2014 and 2015, suggesting that the system had reached a quasi-static regime; i.e., the shape and the parameters that characterize the distribution did not change in time. However, in the following years, namely, 2016 and 2017, we note an interesting variation signaling an important change in the educational system. This effect is probably a product of the intervention developed by the SPP program. We note that the program was announced in October 2014, so the student taking the exam that year had no time to react to it. Furthermore, it seems that the student population required two years to consciously prepare for the exam, so the change in the score distribution is observed from the year 2016 onward. This effect is more significantly observed in the changes in the score distributions for the SISBEN population from the year 2016 onwards, as appreciated in Figure 2(b). This suggests that changes in the preparation period to take the test were crucial to observe this change.
(a)
(b)
(c)
Figures 2(b) and 2(c) show the temporal evolution of the test score probability distributions for SISBEN and non-SISBEN students. Similar effects to those described in Figure 2(a) are presented in the SISBEN students. This observation strongly suggests that SISBEN students have produced a structural change in the system that is ubiquitous from 2016 onwards. It is worth noticing that the left side of the distribution is invariant in time for the general population and SISBEN students, indicating that the population with lower scores for the different years presented the same behavior. The important changes of the distribution, in the case of SISBEN students, are driven by the right tail of the distribution, which turns out to be composed mainly by the pilos.
We can also observe interesting results in Figure 2(c) about the non-SISBEN students. We first observe a notorious change in the distribution for 2016. The kurtosis of the probability density function changed to the right. Left and right tails of the distribution changed, indicating that this population obtained better results, even than the pilos. Although these students have no direct incentive to study more to obtain an economic benefit, they noticed the pressure that the SPP students imposed in the institutions that are accredited. In this sense, it is possible to observe a competition among the different populations due to the restricted possibility to access a good university education. Although the acceptance to the university does not have any influence on the SPP program, its access is limited. Even more impressive, the form of the distribution of non-SISBEN students changed abruptly in 2017, with a significant increase of the left and right tails of the distribution. The changes at the left tail signal an increase in the number of non-SISBEN students with worse test results. At the same time, changes at the right tail of the distribution signal that the global incentive is to study more to be able to access a good university given the new competition.
In the three figures, and particularly in non-SISBEN students Figure 2(c), we note notorious changes that occur from the 2nd year after the program was announced. We observe that the main core, not necessarily the tails, of the distribution of the examination scores can be taken as the natural behavior of a bell-shaped distribution that might be considered as normal, or a distribution that is characterized by a mean and a standard deviation , carefully noting that, in Figures 2(a), 2(b), and 2(c), most of the data exhibits a positive change in the skewness for all years.
Figure 3 summarizes the evolution of the statistics during the four years of the program implementation. Figure 3(a) shows the evolution of the mean of the distribution for SISBEN, non-SISBEN, and the whole population. We observe a general improvement for the SISBEN population with respect to the non-SISBEN which seems to have a relatively constant performance. Although Figure 2(c) displays a significant change in the distribution of non-SISBEN students, it affects mainly the tails of the distribution, keeping the mean relatively constant over time. However, from the changes in the standard deviation of Figure 3(b), we observe an increase in the gap between the best and the worst test score performers. From this last figure we may observe that important differences or gaps have emerged in the affluent population, while there is a trend to observe a reduction of the gap between high scores and low scores of the SISBEN population. It is important to remember that this dynamic, which affect the SISBEN and non-SISBEN population, is driven by an intervention program on the SISBEN population, and it is related to the fact that the SISBEN population has benefited from this program in the last four years, as is shown in Figures 3(b) and 3(c). As these figures suggest and considering that socioeconomic conditions in Colombia remain relatively stable, there are more students requested to be surveyed by SISBEN. We believe that this may be considered as an additional effect that was not expected from this intervention; however the economic implications for the test should be analyzed in future studies in order to enhance the test conditions.
(a)
(b)
(c)
Figures 4(a) and 4(b) show the improvements and performance of the right tail of the distribution using our approach based on the Pareto power-law tail approximation. Figure 4(a) shows the distribution at the right tail of the SISBEN students, the non-SISBEN students, and the overall population in the power-law tail regime. The form of the distribution shows that the index of the distribution evolves in time and that our approach is a reasonable approximation in this regime, allowing us to make inferences about the performance of these populations. Figure 4(b) shows the evolution of the measurements of performance and Figure 4(c) shows the performance improvement for the three populations during years 2014 to 2017. Although the SPP intervention is intended for the SISBEN students, there are significant changes even in the non-SISBEN and in the whole population as well. Therefore, it seems that the SPP intervention was able to produce a structure change of the whole secondary and tertiary educational system.
(a)
(b)
(c)
It is worth focusing on the overall population. For example, the performance during the first two years (2014 and 2015) of the whole population is very low, and, therefore, the improvement () is not notorious. However, the biggest performance is obtained for the three populations during 2016, obtaining a notorious improvement during the period 2015-2016. Another interesting unintended effect is that, for 2017, the overall population decreased the performance. This fact might be explained by the increment in the educational offer and transition to other nonprofessional educational options, among others. It is worth noticing that the non-SISBEN population always had a better performance than SISBEN students; however, the measurements of improvement suggest that SISBEN population tend to have a better improvement that non-SISBEN ones.
3.2. The Pilos
In this section we study the statistical performance of the SPP beneficiaries in the three subset areas. Each student corresponds to one of the three geographical areas defined by the SISBEN: (i) 14 biggest cities of the country, (ii) urban cities and towns different from the 14 biggest cities, and (iii) rural areas. Figures 5(a)–5(d) show the probability distribution of the three SISBEN areas for the years 2014, 2015, 2016, and 2017, respectively. First, we can observe that our approximation of the Pareto power-law tail regime is valid to evaluate in time the evolution of the performance of the students. Over time, Figures 6(a) and 6(b) show the evolution of the performance in time as well as the measurement of the improvement for these populations. It is worth noticing that SISBEN areas 2 and 3, which represent the poorest populations of Colombia, have the same performance compared to those that inhabit the poorest parts of area 1 during years 2014, 2015, and 2016. However, we do observe an interesting variation during the year 2017. While the performance of SISBEN area 1 is similar to previous years, there is a considerable improvement in the performance of SISBEN areas 2 and 3. In this sense, we may deduce that the SPP program has its biggest effect in the most abandoned regions of the country. This may be interpreted as a reduction in the gap between the poorest of the large cities, who are closer to the accredited universities, and the poorest students of the other two areas, who before SPP did not have a reasonable opportunity to have access to a university education. We believe that the inclusion of payment of living expenses in the SPP program could be responsible for this effect.
(a)
(b)
(c)
(d)
(a)
(b)
4. Discussion and Final Remarks
In this paper we present some statistical results of the program SPP, which was a flag educational program of the Colombian government during 2014-2018. The statistical analysis suggests that the SPP intervention has produced a global structural change in the access to a university education. Not only did the change affect the students that benefited from SPP, but it also affected the whole educational system.
For example, we found different degrees of performance and improvement of the SISBEN and the non-SISBEN populations in the entrance examination to the CHES. The study also shows clear evidence of the competition among different groups in the educational market. The result in the test is an unintended effect of the SPP program in the access of these populations to a university education. This unintended effect shows that SPP is one of the most important interventions of the Colombian government in recent years. Perhaps, the most outstanding result reported here is the reduction in the gap between the SISBEN students, particularly from the poorest and farthest parts of Colombia, in comparison to the SISBEN students that live in the 14 biggest cities. This result suggests a reduction in the inequality gap that we hope will produce a significant reduction in the socioeconomic inequality in Colombia in the near future.
These results open an interesting discussion about some crucial aspects that could drive the educational policies in Colombia for the future. The first point is the need to produce quantitative measurements of the efficiency of these interventions to compare the economic benefits of programs that are focused on the student population and the public universities. This is a crucial point in the debate about public education in Colombia. Previous works have suggested that funding the public universities may produce better results than funding individuals with a program like SPP. However, our findings show that the intervention has reduced the gap between poor people in the cities who may have access to public education and people in the farthest and poorest parts of the country that traditionally have not had the opportunity to go to university. We have also shown that this intervention stimulated a strong improvement in test scores, and hence the opportunities to access a good university education, of the SISBEN students from areas 2 and 3 in comparison with the SISBEN students from area 1. Furthermore, the relative increase in the mean of the scores of all SISBEN students, compared with non-SISBEN students, should improve the opportunities of the SISBEN students to access other financial resources to be able to finance their university education, due to the fact that they can be accepted in better institutions than before.
The SPP intervention may have further unintended social consequences. For example, some of the students from areas 2 and 3 may work once they finish their studies and may generate progress in their region. Hence, we should prepare to develop studies to understand the complexity nature of this social intervention as these students graduate from their university programs in a few years, start working, and encourage new generations of SPP. Therefore, there can be further unexpected consequences of this program that we may find in the future.
Data Availability
The data used to support the findings of this study were supplied by ICFES under license and so cannot be made freely available. Requests for access to these data should be made to Roberto Zarama and Fernando Rojas (coauthors of the present manuscript) from CeiBA Complexity Research Center who will do the formal request to ICFES.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this manuscript.
Acknowledgments
We thank the support of Fondecyt project 1150718, Fondecyt Postdoctoral project 3180315, CEDENNA, and the support of CeiBA.