Improving Analytics Capacity

Background

While DHE has long collected data from the Commonwealth’s 29 public colleges and universities and used analysis of this data to report on institutional performance and shape policy, developments in technology in recent years have opened up possibilities to transform data collection, institutional research, and analysis to be more nimble, accessible, and expansive than ever before. DHE, working in partnership with the Executive Office of Education (EOE), the community colleges’ and state universities’ Partnership to Advance Collaboration and Efficiencies (PACE), and the public campuses, is investing in the modernization of our data and analytics systems to build a shared and improved resource to generate more useful research and analyses for and within the Massachusetts public higher education system.

Objectives

DHE’s collaboration on this modernization effort with respective partners, such as EOE, PACE and the public campuses, is framed by these shared objectives:

We Will Reduce Administrative Burden

• Eliminate data fields or collections that are no longer relevant or not used
• Explore new central data collection practices that may ease burdens on campus-level IR directors and other data providers, such as streamlining complex standard submissions and eliminating need for specialized, short-term collections

We Will Improve Data Quality & Access

• Improve data quality by revising data element definitions and methodologies for existing systemwide data collections
• Expand the scope of the Higher Education Information Resource System (HEIRS) analytic database to include the broadest possible range of relevant data for research purposes
• Provide campus access to public higher education data merged with other Commonwealth agencies’ data (e.g., public K-12 and Unemployment Insurance)

We Will Accelerate Analytics Adoption

• Provide data users on campuses with new resources in Tableau that will enhance the utility of centrally collected and campus-stored data
• Address the data needs of campus decision-makers who have traditionally had limited access to centrally collected data and analytics
• Convene regular meetings of campus-level researchers and analysts to share insights and knowledge and to foster a Community of Practice

Current Projects

This effort to improve analytics capacity is well underway through a number of complementary projects and initiatives.

Performance Measurement Reporting System (PMRS)

In May 2019, DHE launched the Performance Measurement Reporting System (PMRS), a public-facing data tool intended to prompt constructive conversations about performance and accountability between and among DHE, the campuses, boards of trustees, legislators and others. The launch followed a multi-year process engaging the Commonwealth’s community colleges and state universities in the development of a new set of metrics in each required performance area that was accepted by the BHE in December 2018. The debut of PMRS represented the first time in over a decade that DHE provided institution-level performance metrics and reports on our website, and with the introduction of the interactive capabilities of the Tableau data analytics platform, the scope and depth of these reports is unprecedented. PMRS now provides a shared language and foundation for discussions of performance and strategic planning across the community colleges and state universities.

Tableau Implementation

In December 2018, PACE, working with DHE and participating public campuses, executed a collaborative purchase of Tableau Server and software to help accelerate analytics adoption across the public higher education system. Currently, along with DHE, 12 of the community colleges and six of the state universities host their own enterprise analytics sites through this implementation. A few other campuses have their own, separate Tableau implementations but collaborate with and advise the partnership. Through support from DHE’s Higher Education Innovation Fund, all community colleges and state universities have access to DHE’s enterprise site which is home to system-level data sources and dashboards for DHE and campus use.

In summer 2019, through the Higher Education Innovation Fund, DHE funded a Tableau Accelerated Adoption initiative, supported by a management consultant, to identify and begin to fill knowledge and resource gaps that may be preventing campuses from moving up the scale toward full analytics adoption. This initiative culminated in a host of guidance documents and templates developed by the consultant that are available to our Tableau user community in an online Tableau Community Hub, as well as a series of trainings for campus staff with hands-on roles in authoring reports and managing users.

In fall 2019, using IT capital funds from the Commonwealth, DHE began working with EOE to convert systemwide data reporting tools built in antiquated technology into Tableau data sources and dashboards for the Tableau user community. As of January 2020, the first phase of this work is underway, and data sources and dashboards will soon be ready for usability testing and feedback from the user community.

Employment and Earnings Data Merge

Supported by the community colleges’ FY14–FY17 grant from the U.S. Department of Labor’s Employment and Training Administration, DHE and the Executive Office of Education (EOE) developed a data merge with the Massachusetts Division of Unemployment Assistance, which provides useful information about the wage outcomes of community college students who go on to be employed in Massachusetts in positions that require unemployment insurance. The data merge covers both certificate and associate degree students and graduates, and wage data can be disaggregated by demographics such as race/ethnicity, gender and Pell eligibility.

While there are limitations to the utility of the merged data set, including the lack of information about graduates employed out of state, employed by the federal government or military, or self-employed; a lack of specificity on the type of work being performed beyond the industry of employment; and a lack of specificity on the rate of earnings beyond a quarterly sum for each individual matched in the system, the current data merge provides a lot of value to supplement other institutional data on employment and earnings outcomes of graduates. Working with EOE, DHE expects to expand access to this data to analysts within the state university and UMass segments during FY20.

Data Collection Process Enhancement

Finally, also using IT capital funds from the Commonwealth, DHE and EOE have begun a “study year” (FY20) to plan for a significant improvement and enhancement of the benefits of the centralized HEIRS database. During the study year, we will explore the feasibility of technologies such as APIs (Application Programming Interfaces) to streamline the process through which the public campuses submit data to the centralized system. The goal is to identify technology solutions that will ease administrative burdens on campuses and increase the potential to obtain a higher quality and broader scope of data for research purposes. The study year will inform a strategy, budget and funding request to support the implementation of this effort.