Use of Analytics in K-12 School Districts

Executive Summary

By their nature schools have always collected large amounts of data, but without an analytics program in place it is impossible to analyze the volume of data being produced by students, given the countless variables (variety) and the speed at which it is created (velocity). An analytics program allows administrators to not only understand their students and faculty across the district but proactively create policy and processes to foster growth and future success. Further, although schools have always had the ability to using reporting and analytical analysis to understand what they know about a student population, a full analytics program offers greater epistemological value by providing statistical inference in order to shed light on what we otherwise could not know. The synergetic benefits of an analytics program are further realized by implementing the complete analytics framework, addressing the culture, people, organization, process, data, and architectural needs of the school district.

Case Study

In 2015 ACT performed a study on the implementation of an Analytics program on two Texas school districts and its impact on the hierarchical leadership of (1) district leaders, (2) school leaders, and (3) classroom teachers across five practices: (a) curriculum and academic goals, (b) staff selection, leadership, and capacity building, (c) instructional tools, programs and strategies, (d) monitoring performance and progress, and (e) intervention and adjustment (Dougherty, C. 2015). The school districts served approximately between 25-75,000 and 5-25,000 students respectively, and the full demographic information of the students can be found in Table 1:

Table 1
Demographics of Study

The study specifically examined the types of data examined (e.g., test scores, attendance records, discipline records, surveys, etc.), the users which the data pertains to (e.g., students, teachers, coaches, etc.), and the objective use of the data (i.e., “identify individual student needs and place students in groups, interventions, programs, and classrooms,” “Modify curriculum and instruction,” etc.). Stakeholders interviews were conducted with school leaders, district administrators, and academic coaches, and the analyst also observed faculty meetings to determine the impact of the district’s analytics initiatives. The study explored what data was collected by the school district, the support leaders provided to enable the use of the collected data, and the impact of the implementation (Dougherty. 2015).

The Value of Implementing an Analytics Framework within K-12 School Districts.

Schools, and more contemporarily in the United States school districts have always used some form of analytics to inform teachers on the progression of their students. A 2020 survey by EdTech states that 47% of respondents school administrators are planning on investing in data analytics tools for their district in the next two years (Castelo, M. 2020). Conversely, a Glimpse K12 report from 2019 suggests that “67% of software licenses sold to schools were going unused”  and that “90% of districts (Glimpse K-12, 2019) works with that do not have clearly defined ROI goals suffer from severe underutilization of technology resources” resulting in lower student achievement (Glimpse K12, 2019). Although districts understand the importance of data analytics, there is a disconnect between perceived and realized value of these digital transformation initiatives. The following is a case study of a successful data analytics digital transformation within two Texas school districts demonstrating the value brought by implementing an analytics framework program.

Data analytics provides multiple “dimensions of value” for an organization. While various frameworks differentiating the kinds of analytics have been suggested (Davenport, T. & Harris, J.G. 2017; Eckerson, W. 2012; Gartner, 2019) in each framework the value of analytical analysis increases with complexity of analysis. There are five distinct ordered categories of analytics: (1) Descriptive, (2) Diagnostic, (3) Predictive, (4) Prescriptive, (5) Autonomous.[1] Davenport & Harris’s (2017) categories of analytics are descriptive, predictive, prescriptive, and autonomous analytics, whereas Gartner (2019) list of analytical tools contains the inclusion of diagnostic analytics and removed autonomous analytics.

(Davenport, T. & Harris, J.G. 2017., Gartner, 2019). Further, each kind of analytics produces different value based on the category which Eckerson suggests is related to a temporal relationship (i.e., past, present, and future); analytical reporting quickly communicates insight of past data and analytical analysis explains these results (Eckerson. 2012). Analytical monitoring communicates the current state of present data, and prediction and hypothesis tests attempt to uncover future data (Eckerson. 2012).

The value of analytics is much greater than temporal analysis and the value categories of analytics are essentially epistemological in nature; reporting and analysis concern what we know or can come to know whereas monitoring concerns truth values which are indeterminate or in fluctuating states, and hence capable of changing based on the period within which it is measured. Predictions and hypothesis testing attempt to give us probabilistic knowledge about what we do not know, either because it has not yet occurred or because it is missing from our knowledge. As an epistemological tool analytics is invaluable.

In this case study the analytics program created great value for the districts by providing a range of insights, from reporting to prediction and hypothesis testing. Data concerning student grades were obtained using a variety of assessments. Benchmark exams were administered by the districts at least every 9 weeks, and the data was recorded in a centralized database (Dougherty. 2015). Additional standardized tests were also administered to students, including the State Accountability Test, the College Board’s ReadiStep exam, the Iowa Tests of Basic Skills, cognitive examinations to flag students for inclusion of gifted student programs, and English language proficiency exams amongst others (Dougherty. 2015). Walk-throughs were given which largely served as a validation tool to ensure that teachers were following their lesson plan and that the lesson plan was aligned with the district plan (Dougherty. 2015).

The results of the assessments allow for advanced modeling of the students, teachers, and curriculum in the district. Reporting analytics were used to assess lagging indicators of teaching viz. student grades. Scores between students from different socio-economic backgrounds were able to be compared to ensure that students in one area of the district were receiving the same quality of education (Dougherty. 2015). Similarly, the variance between students’ scores between classroom examinations and benchmark examinations revealed possible differences between expected learning outcomes of individual teachers versus that of the district (Dougherty. 2015). These methods help to explain what has happened in leading up to these examinations.

More advanced analytics go beyond telling us past states and rather predict future states. Predictive modeling was used to flag individual students or groups of students who were in need of intervention based on their current trajectory (Dougherty. 2015). Prescriptive modeling was used to predict student scores on state standardize testing, and then adapt their curriculum in real time to better prepare students for the examinations (Dougherty. 2015). Monitoring and Analysis was also used to ensure equity between distributions of students at all levels, either between teachers at different schools or between classrooms at the same schools (Dougherty. 2015). This type of analysis demonstrated that teachers or schools were not being singled out by receiving a disproportionate number of students in any statistically monitored category (i.e., race, gender, learning disability, behavioral needs, etc.) (Dougherty. 2015).  The data from the benchmark assessments was joined with data on individual student grades by assignment and classroom assessment; as with benchmark data, district and school leaders could use these scores to compare the distributions and ensure that they were equitable both within the same school and within the same district (Dougherty. 2015).

Attendance records and disciplinary records provide another source of interesting data which enabled the districts to handle issues proactively and not reactively. Disciplinary records in particular were used to identify specific locations on school campuses where problems are likely to occur such as in one example the boy’s washroom (Dougherty. 2015. 16). One of the districts also used disciplinary data to understand issues concerning a specific teacher: “We’ll use discipline data to see if a particular teacher is really struggling. If one teacher’s referring a lot [of students and] no one else is referring those same kids… it tells us there’s an instructional classroom management problem.” (Dougherty. 2015. 16) This data can further be modeled to examine potential factors such as race, gender, orientation, etc.

Leadership & Impact

Without an analytics program it would be impossible to obtain such a wide understanding of a district of this size impacted by hundreds of variables and factors. Analytics allows us to use statistical inference to infer knowledge otherwise impossible to obtain at scale. Data alone is not enough to ensure analytical success; according to Eckerson’s Analytical Framework  the (1) right culture, (2) right people, (3) right organization, (4) right process, and (5) right architecture, in addition to having the (6) right data are all important factors (Eckerson, 2012. 56 – 58). and each of these were addressed by the district.

Culture and People. Local and district leadership fostered a data-forward culture by urging teachers to collaborate with each-other using the insights from the data, as well as create a culture where teachers are encouraged to devise their own studies using the data (Dougherty. 2015). The district actively changed the organization by created Professional Learning Communities (PLC) and data teams which help teachers to understand the results of their data and how to apply the derived insights to the classroom (Dougherty. 2015.). The districts also continue to work with teachers to strengthen their own data mining skills to learn how to analyze the data themselves (Dougherty. 2015).

Architecture. The district created the required architecture needed to support analytics reporting and this resulted in the creation of a vast electronic information system enabling broader and more efficient communication between all stakeholders from district administrators to parents. The aggregation of data amongst all schools in the district and the communication network enabled cross-department and cross-school collaboration based on subject verticals (Dougherty. 2015. 20). Leaders used this network to facilitate discussion between subject teachers at all level from k – 12^th grade across the entire district; this enables subject teachers (i.e., biology, history, etc.) to work together to plan multi-year curriculum which covers multiple grade levels (Dougherty. 2015. 20).

Process and Organization. The implementation of the Analytics program changed processes at both the district level as well as the individual teacher level. District administrators utilized the data to reduce the cost by hiring academic coaches who are able to work with teachers on a 1-on-1 or group basis (Dougherty. 2015. 21). Hiring additional an additional teacher has a localized impact which is only felt at the specific school, and only mitigates the performance of underachieving teachers; Academic coaches instead are dispatched to mentor and help struggling teachers (Dougherty. 2015. 21). Their transient nature allows limited funding to have a positive impact district wide. District administrators were also able to use the modeled data to create “pacing guides” which advised teachers on the length of time needing to be spent in each subject area (Dougherty. 2015. 19). While not mandated, the resulting analysis served as an important coaching tool to assist teachers.

References

ACT. (2015). “Use of data to support teaching and learning: a case study of two school districts.” ACT Research Report Series 2015. (1). http://www.act.org/content/dam/act/unsecured/documents/ACT_RR2015-1.pdf

Castelo, M. (2020). The IT investment priorities shaping today’s school districts. Ed Tech. https://edtechmagazine.com/k12/article/2020/07/it-investment-priorities-shaping-todays-school-districts

Davenport, T.H. & Harris, J.G. (2017). Competing on analytics: the new science of winning. Harvard Business Review Press.

Eckerson, W. W. (2012). Secrets of Analytical Leaders. Technics Publications.

Gartner Research. (2019). 4 types of analytical tools to support insight generation. Gartner inc.

Glimpse K12. (2019). Glimpse K12 analysis of school spending shows that two-thirds of software license purchases go unused. https://www.glimpsek12.com/blog/schoolsoftwarespending

[1] Davenport & Harris’s (2017) categories of analytics are descriptive, predictive, prescriptive, and autonomous analytics, whereas Gartner (2019) list of analytical tools contains the inclusion of diagnostic analytics and removed autonomous analytics