Core Data Manipulation Libraries (NumPy & pandas)

Introduction

NumPy and pandas are the two libraries almost every other data science tool in Python is built on top of. NumPy provides fast, memory-efficient numerical arrays; pandas builds labeled, spreadsheet-like tables on top of those arrays. Nearly every catalog entry later in this course either depends on one of them directly or shares their conventions.

NumPy: Fast Numerical Arrays

Use case: NumPy provides the ndarray, a fixed-type, contiguous-memory array that supports fast vectorized math — element-wise operations, linear algebra, and broadcasting — without writing explicit Python loops.

pip install numpy

NumPy in Action

import numpy as np

prices = np.array([19.99, 45.50, 12.00, 89.99, 5.25])
quantities = np.array([3, 1, 10, 2, 20])

# Vectorized element-wise multiplication -- no loop needed
totals = prices * quantities
revenue = totals.sum()

print("Line totals:", totals)
print("Total revenue:", round(revenue, 2))
print("Average price:", round(prices.mean(), 2))

pandas: Labeled Tabular Data

Use case: pandas provides the DataFrame, a table of rows and columns with labels, mixed data types per column, and built-in methods for filtering, grouping, merging, and reshaping — the standard tool for working with structured, spreadsheet-like data in Python.

pip install pandas

pandas in Action

import pandas as pd

df = pd.DataFrame({
    'region': ['North', 'South', 'North', 'East', 'South', 'East'],
    'product': ['Widget', 'Widget', 'Gadget', 'Widget', 'Gadget', 'Gadget'],
    'sales': [1200, 950, 1800, 600, 1100, 750],
})

# Filter: only Widget sales
widgets = df[df['product'] == 'Widget']

# Group + aggregate: total sales per region
by_region = df.groupby('region')['sales'].sum().sort_values(ascending=False)

print(widgets)
print()
print(by_region)

NumPy Arrays vs pandas DataFrames: When to Use Which

In practice, pandas DataFrames are built on top of NumPy arrays internally — when you call .values or .to_numpy() on a DataFrame, you get the underlying NumPy array back. Most projects use both together: pandas for loading and cleaning, NumPy for the numerical heavy lifting underneath.

Common Mistakes

Best Practices

Frequently Asked Questions

Key Takeaways

Summary

NumPy and pandas together form the foundation of the entire Python data science ecosystem. Nearly everything else in this course, from visualization to machine learning, either consumes their data structures directly or borrows their conventions.