The SAT (Scholastic Aptitude Test) is a standardized test widely used for college admissions in the United States. The test is divided into 3 sections, each of which has a total score of 800. High schools are often ranked by their average SAT scores, and high SAT scores are considered a sign of how good a school district is.

This project utilizes different datasets taken from the NYC OpenData initiative. The main idea here was to combine data from multiple associated data sources and perform exploratory data analysis on it to understand:

1. Are schools differentiated by SAT scores?
2. The schools which are doing best and the ones which are doing worse, what are the key differentiating factors between these schools?
3. Do demographics really play a role in SAT scores? If yes, then how and which demographic categories are doing better than others

Data pull and cleaning

For purpose of this analysis, we will be taking multiple open-source data sets available, perform some cleaning, and consolidate them into one dataset having one row per school, which would then allow us to do the rest of Exploratory Data Analysis pretty easily. As I would be using the NYC OpenData APIs to pull in the data, I have added the links to APIs for downloading the data directly into pandas dataframes.

Before we start with the datasets, I would like to take a moment and explain how the data hierarchy works in these datasets. I did some research and found out that New York is divided into districts (denoted by numbers like 01,02, etc.), which are then further divided into boroughs, which then have schools. Thus, each school is identified by its DBN (District Borough Number) which looks something like “01M015”.

The datasets that we would be using in this project are:

• Sat Scores - SAT Scores by sections for each school. Download JSON here
• Enrollment - Average attendance & Number of students enrolled at a district level. Download JSON here
• High Schools - A plethora of details about every school such as location, address, website, etc. Download JSON here
• Class Size - Details about school class sizes & education programs. Download JSON here
• Advanced Placement - Summary of students taking advanced placement exams by school. Download JSON here
• Graduation - Graduation related stats for each school by cohorts. Download JSON here
• Demographics - Student demographics information for each school. Download JSON here
• Districts - District coordinates and shape polygons. You will need to download this data in GeoJSON format in order to use it for plotting. Download them from here
• Math test results - Math test results for every school. Download JSON here
• Survey Results - Survey responses of students, teachers, and parents for each school providing overview of their perception of school quality. This is not available through APIs and has to be downloaded from here

Having gotten enough context, let’s get started with this project!!

First, we will import the required libraries using the below code:

#Importing all relevant libraries
import numpy as np
import pandas as pd
import os
import folium # For plotting chloropleth charts
from folium import plugins
import pygeoj # For reading in geoJSON District map file

Now, let’s read in all the different datasets that I described above:

#Using Department of Education APIs to read in all relevant data
sat_results = pd.read_json("https://data.cityofnewyork.us/resource/f9bf-2cp4.json")
enrollment = pd.read_json("https://data.cityofnewyork.us/resource/7z8d-msnt.json")
high_schools = pd.read_json("https://data.cityofnewyork.us/resource/n3p6-zve2.json")
class_size = pd.read_json("https://data.cityofnewyork.us/resource/urz7-pzb3.json")
ap2010 = pd.read_json("https://data.cityofnewyork.us/resource/itfs-ms3e.json")
graduation = pd.read_json("https://data.cityofnewyork.us/resource/vh2h-md7a.json")
demographics = pd.read_json("https://data.cityofnewyork.us/resource/ihfw-zy9j.json")
math_results = pd.read_json("https://data.cityofnewyork.us/resource/jufi-gzgp.json")
survey_1 = pd.read_csv("masterfile11_gened_final.txt",delimiter='\t',encoding='windows-1252')
survey_2 = pd.read_csv("masterfile11_d75_final.txt",delimiter='\t',encoding='windows-1252')

Having the data in place, its always good to check samples for each data set to understand it better

#Browsing the datasets one by one
sat_results.head()

enrollment.head()

high_schools.head()

5 rows × 69 columns

class_size.head()

ap2010.head()

graduation.head()

5 rows × 23 columns

demographics.head()

5 rows × 38 columns

math_results.head()

survey_1.head()

5 rows × 1942 columns

survey_2.head()

5 rows × 1773 columns

Things we can start noticing from previews:

1. DBN is one common field that is present in majority of the datasets. We can combine these datasets on DBN
2. All datasets do not have one unique row for each school
3. All schools listed in these datasets are not high-schools
4. We can choose the important fields from survey data and combine the two survey datasets into one

Okay, so having said that we need to create a DBN field in the class_size dataset. Looking at what DBN looks like, we can see it is a combination of district, borough, and school code. Fragments of dbn are already lying around in class_size, which we need to concatenate into one.

class_size["dbn"] = class_size.apply(lambda x: "{0:02d}{1}".format(x["csd"], x["school_code"]), axis=1)
class_size['dbn'].head(10)

0    01M015
1    01M015
2    01M015
3    01M015
4    01M015
5    01M015
6    01M015
7    01M015
8    01M015
9    01M015
Name: dbn, dtype: object  

Perfect! This looks like a dbn code now.

Let’s now combine the two survey datasets and only keep the fields seeming important

survey = pd.concat([survey_1,survey_2],axis=0)
survey_fields = ["dbn", "rr_s", "rr_t", "rr_p", "N_s", "N_t", "N_p", "saf_p_11", "com_p_11", "eng_p_11", "aca_p_11", "saf_t_11", "com_t_11", "eng_t_10", "aca_t_11", "saf_s_11", "com_s_11", "eng_s_11", "aca_s_11", "saf_tot_11", "com_tot_11", "eng_tot_11", "aca_tot_11"]
survey = survey.loc[:,survey_fields]

Before joining the datasets, let’s reduce datasets to 1 row per school dbn. Starting with class_size

#Only the grade 9-12 is relevant to our analysis 
class_size['grade_'].unique()

array([‘0K’, ‘01’, ‘02’, ‘03’, ‘04’, ‘05’, ‘0K-09’, nan, ‘06’, ‘07’, ‘08’, ‘MS Core’, ‘09-12’], dtype=object)

#Majority of the grade 9-12 programs are general education
#Here CTT stands for Collabortive Team Teaching where both Gened and Special Ed teachers teach together in the class
#For now, let's keep only the gened program
class_size[class_size['grade_']=='09-12'].groupby('program_type').count()

class_size = class_size[class_size["grade_"] == "09-12"]
class_size = class_size[class_size["program_type"] == "GEN ED"]
class_size = class_size.groupby("dbn").agg(np.mean)
class_size.reset_index(inplace=True)
class_size.rename(columns = {'grade_':'grade'}, inplace = True)

Moving on to demographics

#Let's keep only the latest year
demographics['schoolyear'].unique()

array([20052006, 20062007, 20072008, 20082009, 20092010, 20102011,
       20112012], dtype=int64)  

demographics = demographics[demographics['schoolyear']==20112012]

#Ensuring there are no duplicate dbn rows
len(demographics)-demographics['dbn'].nunique()

0

Moving on to math results

#Taking results for the highest grade
math_results['grade'].unique()

array(['3', '4', '5', '6', 'All Grades', '7', '8'], dtype=object)

#Taking results for the latest year
math_results['year'].unique()

array([2006, 2007, 2008, 2009, 2010, 2011], dtype=int64)

math_results = math_results[(math_results['grade']=='8') & (math_results['year']==2011)]

#Ensuring there are no duplicate dbn rows
len(math_results)-math_results['dbn'].nunique()

0

Moving on to graduation

#Taking the latest 2006 cohort
graduation['cohort'].unique()

array(['2003', '2004', '2005', '2006', '2006 Aug', '2001', '2002'],
      dtype=object)

#Taking the overall cohort
graduation['demographic'].unique()

array(['Total Cohort', 'Asian', 'Male'], dtype=object)

graduation = graduation[(graduation['cohort']=='2006') & (graduation['demographic']=='Total Cohort')]

len(graduation) - graduation['dbn'].nunique()

0

#This one is already reduced
len(high_schools) - high_schools['dbn'].nunique()

0

#This one requires fixing
len(ap2010) - ap2010['dbn'].nunique()

1

ap2010 = ap2010.groupby('dbn').agg(np.mean)
ap2010.reset_index(inplace=True)

#Checking if it is fixed now
len(ap2010) - ap2010['dbn'].nunique()

0

#This one is already reduced
len(survey) - survey['dbn'].nunique()

0

#Ensuring this is reduced
len(sat_results) - sat_results['dbn'].nunique()

0

Okay, now that we are done cleaning the data sets, let’s proceed to creating few fields which do not exist and might be important later followed by combining the datasets.
Also, I noticed that there are some entries in sat_results which do not have any score associated. Let’s remove them from the analysis as there is no legit way to impute these values.

Combining the datasets

sat_results = sat_results[sat_results['sat_math_avg_score']!='s']

#Converting sat scores to numeric valu
cols = ['sat_math_avg_score', 'sat_critical_reading_avg_score', 'sat_writing_avg_score']
for c in cols:
    sat_results[c] = sat_results[c].apply(lambda x: int(x))

# Adding in total average SAT score as a new field
sat_results['sat_score'] = sat_results['sat_critical_reading_avg_score']+sat_results['sat_math_avg_score']+sat_results['sat_writing_avg_score']

# Extracting latitude and longitude out of the location information provided in high school data
high_schools['Latitude'] = high_schools['location_1'].apply(lambda x: x.get('latitude'))
high_schools['Longitude'] = high_schools['location_1'].apply(lambda x: x.get('longitude'))

# Checking the number of unique DBNs in each database
data_list = {'sat_results':sat_results,
             'enrollment':enrollment,
             'high_schools':high_schools,
             'class_size':class_size,
             'ap2010':ap2010,
             'graduation':graduation,
             'demographics':demographics,
             'districts':districts,
             'math_results':math_results,
             'survey':survey}

for k,v in data_list.items():
    print(k)
    try:
        print(v['dbn'].nunique())
    except Exception as e:
        print(e)

sat_results
421
enrollment
'dbn'
high_schools
435
class_size
15
ap2010
257
graduation
154
demographics
151
districts
'dbn'
math_results
12
survey
1702

Hmm, looks like survey leads in number of dbns by a huge margin. One join approach can be to use that as base and left join every other table to that. But that will result in a high number of blank rows (Not apt for our analysis) Let’s check how many DBNs we have in common between sat_results and high_schools. Maybe we could utilize that number as the base and fetch information for those from other tables

for k,v in data_list.items():
    print(k)
    try:
        print(len(set(sat_results['dbn']).intersection(v['dbn'])))
    except Exception as e:
        print("Exception {0} caught".format(e))

sat_results
421
enrollment
Exception 'dbn' caught
high_schools
339
class_size
10
ap2010
251
graduation
148
demographics
67
districts
Exception 'dbn' caught
math_results
3
survey
418

From what I observe above, I believe it would be best to take an outer join of sat_results, high_schools, ap2010, and graduation. Then we can left join other datasets to it.

full = pd.merge(sat_results,high_schools,on='dbn',how='outer')

full = pd.merge(full,ap2010,on='dbn',how='outer')

full = pd.merge(full,graduation,on='dbn',how='outer')

for k,v in data_list.items():
    if k not in ['sat_results','high_schools','ap2010','graduation','enrollment','districts']:
        full = pd.merge(full,v,on='dbn',how='left')

np.set_printoptions(threshold=200)

cols = full.columns.tolist()

full[['ap_test_takers_','number_of_exams_with_scores_3_4_or_5','total_exams_taken']]

['dbn',
 'num_of_sat_test_takers',
 'sat_critical_reading_avg_score',
 'sat_math_avg_score',
 'sat_writing_avg_score',
 'school_name_x',
 'sat_score',
 ':@computed_region_92fq_4b7q',
 ':@computed_region_efsh_h5xi',
 ':@computed_region_f5dn_yrer',
 ':@computed_region_sbqj_enih',
 ':@computed_region_yeji_bk3q',
 'addtl_info1',
 'addtl_info2',
 'advancedplacement_courses',
 'bbl',
 'bin',
 'boro',
 'building_code',
 'bus',
 'campus_name',
 'census_tract',
 'city',
 'community_board',
 'council_district',
 'ell_programs',
 'end_time',
 'expgrade_span_max',
 'expgrade_span_min',
 'extracurricular_activities',
 'fax_number',
 'grade_span_max',
 'grade_span_min',
 'language_classes',
 'location_1',
 'nta',
 'number_programs',
 'online_ap_courses',
 'online_language_courses',
 'overview_paragraph',
 'partner_cbo',
 'partner_corporate',
 'partner_cultural',
 'partner_financial',
 'partner_highered',
 'partner_hospital',
 'partner_nonprofit',
 'partner_other',
 'phone_number',
 'primary_address_line_1',
 'priority01',
 'priority02',
 'priority03',
 'priority04',
 'priority05',
 'priority06',
 'priority07',
 'priority08',
 'priority09',
 'priority10',
 'program_highlights',
 'psal_sports_boys',
 'psal_sports_coed',
 'psal_sports_girls',
 'school_accessibility_description',
 'school_name_y',
 'school_sports',
 'school_type',
 'se_services',
 'start_time',
 'state_code',
 'subway',
 'total_students',
 'website',
 'zip',
 'Latitude',
 'Longitude',
 'ap_test_takers_',
 'number_of_exams_with_scores_3_4_or_5',
 'total_exams_taken',
 'advanced_regents_n',
 'advanced_regents_of_cohort',
 'advanced_regents_of_grads',
 'cohort',
 'demographic',
 'dropped_out_n',
 'dropped_out_of_cohort',
 'local_n',
 'local_of_cohort',
 'local_of_grads',
 'regents_w_o_advanced_n',
 'regents_w_o_advanced_of_cohort',
 'regents_w_o_advanced_of_grads',
 'school_name',
 'still_enrolled_n',
 'still_enrolled_of_cohort',
 'total_cohort',
 'total_grads_n',
 'total_grads_of_cohort',
 'total_regents_n',
 'total_regents_of_cohort',
 'total_regents_of_grads',
 'average_class_size',
 'csd',
 'number_of_sections',
 'number_of_students_seats_filled',
 'schoolwide_pupil_teacher_ratio',
 'size_of_largest_class',
 'size_of_smallest_class',
 'asian_num',
 'asian_per',
 'black_num',
 'black_per',
 'ctt_num',
 'ell_num',
 'ell_percent',
 'female_num',
 'female_per',
 'fl_percent',
 'frl_percent',
 'grade1',
 'grade10',
 'grade11',
 'grade12',
 'grade2',
 'grade3',
 'grade4',
 'grade5',
 'grade6',
 'grade7',
 'grade8',
 'grade9',
 'hispanic_num',
 'hispanic_per',
 'k',
 'male_num',
 'male_per',
 'name',
 'prek',
 'schoolyear',
 'selfcontained_num',
 'sped_num',
 'sped_percent',
 'total_enrollment',
 'white_num',
 'white_per',
 'category',
 'grade',
 'level_1_1',
 'level_1_2',
 'level_2_1',
 'level_2_2',
 'level_3_1',
 'level_3_2',
 'level_3_4_1',
 'level_3_4_2',
 'level_4_1',
 'level_4_2',
 'mean_scale_score',
 'number_tested',
 'year',
 'rr_s',
 'rr_t',
 'rr_p',
 'N_s',
 'N_t',
 'N_p',
 'saf_p_11',
 'com_p_11',
 'eng_p_11',
 'aca_p_11',
 'saf_t_11',
 'com_t_11',
 'eng_t_10',
 'aca_t_11',
 'saf_s_11',
 'com_s_11',
 'eng_s_11',
 'aca_s_11',
 'saf_tot_11',
 'com_tot_11',
 'eng_tot_11',
 'aca_tot_11']

pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)

# Adding a field for school districs
full["school_dist"] = full["dbn"].apply(lambda x: x[:2])
full.fillna(full.mean(),inplace=True)

It would be beneficial to understand the correlation between sat_score and other columns to understand what fields influence sat_score the most

# Computing correlations between Sat_score and other columns
full.corr()['sat_score'].sort_values(ascending=False)

sat_score                               1.000000e+00
sat_writing_avg_score                   9.810159e-01
sat_critical_reading_avg_score          9.747583e-01
sat_math_avg_score                      9.530106e-01
ap_test_takers_                         5.103909e-01
total_exams_taken                       5.017174e-01
number_of_exams_with_scores_3_4_or_5    4.535005e-01
N_p                                     4.281443e-01
advanced_regents_of_cohort              4.253570e-01
N_s                                     4.221618e-01
advanced_regents_of_grads               3.924787e-01
total_regents_of_cohort                 3.873194e-01
total_students                          3.845674e-01
white_num                               3.777941e-01
total_grads_of_cohort                   3.403242e-01
white_per                               3.328702e-01
N_t                                     2.940519e-01
saf_t_11                                2.757786e-01
rr_s                                    2.736355e-01
asian_num                               2.729442e-01
total_regents_of_grads                  2.697793e-01
aca_s_11                                2.669456e-01
saf_tot_11                              2.608540e-01
saf_s_11                                2.569001e-01
male_num                                2.017470e-01
total_enrollment                        2.013726e-01
asian_per                               1.906019e-01
number_of_sections                      1.726816e-01
female_num                              1.711856e-01
number_of_students_seats_filled         1.708587e-01
total_cohort                            1.619401e-01
aca_tot_11                              1.611355e-01
eng_s_11                                1.551608e-01
com_s_11                                1.512602e-01
aca_t_11                                1.216712e-01
number_programs                         1.153913e-01
number_tested                           1.109474e-01
level_4_1                               1.108240e-01
level_4_2                               1.107934e-01
level_3_4_1                             1.104739e-01
mean_scale_score                        1.072120e-01
saf_p_11                                1.066251e-01
level_3_4_2                             1.027085e-01
rr_p                                    1.021401e-01
eng_tot_11                              8.574072e-02
com_t_11                                8.197490e-02
com_tot_11                              7.766472e-02
regents_w_o_advanced_of_cohort          7.265046e-02
female_per                              6.516539e-02
size_of_largest_class                   4.760754e-02
bin                                     4.132848e-02
census_tract                            3.914908e-02
level_3_1                               3.586515e-02
average_class_size                      3.438492e-02
bbl                                     3.429191e-02
aca_p_11                                3.004570e-02
eng_p_11                                2.891362e-02
:@computed_region_sbqj_enih             1.987861e-02
grade_span_max                          1.830755e-02
:@computed_region_efsh_h5xi             1.638056e-02
rr_t                                    1.164827e-02
expgrade_span_max                       7.847064e-04
expgrade_span_min                       4.104648e-27
size_of_smallest_class                 -2.323112e-02
grade_span_min                         -2.844718e-02
csd                                    -3.282872e-02
:@computed_region_f5dn_yrer            -4.702550e-02
level_3_2                              -4.776999e-02
community_board                        -5.675188e-02
black_num                              -6.104478e-02
zip                                    -6.147304e-02
male_per                               -6.516539e-02
:@computed_region_yeji_bk3q            -6.561082e-02
level_1_2                              -6.972787e-02
level_1_1                              -7.103664e-02
:@computed_region_92fq_4b7q            -7.145365e-02
council_district                       -7.446446e-02
com_p_11                               -8.715157e-02
hispanic_num                           -8.974585e-02
sped_num                               -9.696171e-02
level_2_2                              -1.105178e-01
level_2_1                              -1.109453e-01
regents_w_o_advanced_of_grads          -1.130022e-01
Latitude                               -1.155862e-01
Longitude                              -1.288842e-01
ell_percent                            -1.299355e-01
local_of_cohort                        -2.226132e-01
black_per                              -2.295139e-01
sped_percent                           -2.306069e-01
still_enrolled_of_cohort               -2.361166e-01
local_of_grads                         -2.697793e-01
hispanic_per                           -3.222427e-01
dropped_out_of_cohort                  -3.397260e-01
frl_percent                            -3.546909e-01
schoolwide_pupil_teacher_ratio                   NaN
schoolyear                                       NaN
year                                             NaN
eng_t_10                                         NaN
Name: sat_score, dtype: float64

Let’s take a moment here to note down anything interesting we can find from the above. We can use that later to create an analysis viewpoint.

• Total number of students correlated with Sat Scores. Interesting, that would mean than large schools perform better than small schools
• Number of parents, teachers, and students responding to the survey also highly correlates with sat scores
• FRL (Free or Reduced Lunches) and ELL (English Language Learners) percent correlate strongly negatively with sat score
• Female_num correlates positively whereas male_num correlates negatively with Sat scores
• Racial inequality in sat scores can be identified easily from above (white_per, hispanic_per, black_per, asian_per)

Plotting stage

Now that we have prepared our data, drawn some insights from it as well. Let’s now do some exploratory charts and maps for a better understanding of the problem at hand

# Converting Latitude and Longitude to float values
full['Latitude'] = pd.to_numeric(full['Latitude'],errors='coerce')
full['Longitude'] = pd.to_numeric(full['Longitude'],errors='coerce')

schools_map = folium.Map(location=[full['Latitude'].mean(), full['Longitude'].mean()], zoom_start=10)
marker_cluster = plugins.MarkerCluster().add_to(schools_map)
for name, row in full.loc[~full['Latitude'].isnull()].iterrows():
    folium.Marker([row["Latitude"], row["Longitude"]], popup="{0}: {1}".format(row["dbn"], row["school_name"])).add_to(marker_cluster)
schools_map.save('schools.html')
schools_map

Let’s try plotting a heatmap to better visualize school density across NY

schools_heatmap = folium.Map(location=[full['Latitude'].mean(), full['Longitude'].mean()], zoom_start=10)
schools_heatmap.add_children(plugins.HeatMap([[row["Latitude"], row["Longitude"]] for name, row in full.loc[~full['Latitude'].isnull()].iterrows()]))
schools_heatmap.save("heatmap.html")
schools_heatmap

Interesting, Let’s try assessing further if there is any difference in SAT scores between the school districs

# Aggregating data at district level
district_data = full.groupby("school_dist").agg(np.mean)
district_data.reset_index(inplace=True)
district_data['school_dist'] = district_data['school_dist'].apply(lambda x: str(int(x)))
districts_geojson = pygeoj.load(filepath='districts.geojson')

# Defining a function to overlay heatmap distribution of a metric over the district maps
def show_district_map(col,dist):
    districts = folium.Map(location=[full['Latitude'].mean(), full['Longitude'].mean()], zoom_start=10)
    districts.choropleth(
    geo_data=dist,
    data=district_data,
    columns=['school_dist', col],
    key_on='feature.properties.school_dist',
    fill_color='YlGn',
    fill_opacity=0.7,
    line_opacity=0.2,
    )
    return districts

show_district_map("sat_score",districts_geojson).save("districts_sat.html")
show_district_map("sat_score",districts_geojson)

This looks great! Now that we have developed a basic understanding of the dataset and SAT score variation across districts. Let’s look further into the analysis angles that we had proposed earlier.

%matplotlib inline
full.plot.scatter(x='total_enrollment', y='sat_score')

Hmm, There is not exactly a correlation here. Its just that there is a cluster of schools with low enrollment and low sat scores. Let’s try pulling up some names.

full.loc[(full['total_enrollment']<500)&(full['sat_score']<=1200),['name','sat_score']]

After spending some time on Google to understand what is common between these schools, I have realized that most of these are schools where the minority enrollment and % of economically disadvantaged students is high. So its not actually low enrollment causing low sat scores but high % of minority students

full.plot.scatter(x='frl_percent', y='sat_score')

As expected, there is a negative correlation between FRL percent (indicative of % of minority & economically disadvantaged class) and sat scores. Higher the % of FRL in the school, lower is the sat score

full.plot.scatter(x='ell_percent', y='sat_score')

It looks like there are a group of schools with a high ell_percentage that also have low average SAT scores. We can investigate this at the district level, by figuring out the percentage of English language learners in each district, and seeing it if matches our map of SAT scores by district:

show_district_map('ell_percent',districts_geojson)

The distinction here is not very clear, probably due to the color palette chosen. We can for sure see that there is small district right in the middle which has very high ell_percent but not high sat scores.

Checking the correlation between survey responses and SAT Scores

full.corr()["sat_score"][["rr_s", "rr_t", "rr_p", "N_s", "N_t", "N_p", "saf_tot_11", "com_tot_11", "aca_tot_11", "eng_tot_11"]].plot.bar()

The only factors having a significant correlation with sat scores here are number of parents, students, and teachers responding to the survey, and the % of students responding to the survey. As these also correlate highly with total enrollment, it makes sense to say that schools with low frl percent, also have high survey response rates (that is not our concern at the moment though)

Exploring relationship between race and SAT scores

full.corr()["sat_score"][["white_per", "asian_per", "black_per", "hispanic_per"]].plot.bar()

From above we can conclude that higher percentage of white and asian students lead to higher sat scores and the inverse holds true for black and hispanic students. It is likely that black and hispanic students are migrants who are also learning english language and are from economically disadvantaged community Let’s look at which schools have high white_per

full.plot.scatter(x='hispanic_per',y='sat_score')

full.loc[(full['hispanic_per']<40)&(full['sat_score']>1400),['school_name','sat_score']]

These schools rank high in charts and host admission test and screening before admission, which is probably why hispanic_per ther is lower than others

Gender differences in SAT scores

full.corr()["sat_score"][["male_per", "female_per"]].plot.bar()

AP test scores & SAT scores

One last angle that we would be exploring here is the Advance Placement test and how it affects SAT scores. We already saw that number of AP test takers correlates strongly with SAT scores

full['ap_per'] = full['ap_test_takers_']/full['total_enrollment']
full.plot.scatter(x='ap_per',y='sat_score')

Let’s try pulling some names from the cluster in top right (excluding outliers) where sat_scores are high

full.loc[(full['ap_per']>0.3)&(full['sat_score']>1400),['school_name','sat_score']]

Some Google searching reveals that these are mostly highly selective schools where you need to take a test to get in. It makes sense that these schools would have high proportions of AP test takers.

THE END!

This is all that we are going to cover as part of this project. I would like to thank DataQuest.io for guiding me through the project.