Skip to content

Data Science & ML

6. Machine Learning & Data Visualization

This section covers how to process spreadsheets, clean data, split data sets, train statistical models, and save them for real-world predictions.

Tools & Libraries

  • numpy: The foundation for scientific computing in Python. It is optimized for high-speed mathematical calculations using grids and arrays.
  • pandas: Provides an interactive spreadsheet-like structure (a DataFrame) to manipulate, filter, and clean data files easily.
  • scikit-learn (sklearn): The industry standard for "classic" machine learning. It contains a massive toolbox of statistical and prediction algorithms.
  • joblib: A specialized tool used to "freeze" (serialize) your trained machine learning models and save them as files on your computer for later use.

Industry Upgrades (What Teams Use at Scale)

1. High-Performance Data: Polars

A lightning-fast alternative to Pandas written in Rust. It can process massive datasets (millions of rows) much faster by using all of your computer's processor cores at once.

  • How it looks in code:
import polars as pl

# Polars uses "LazyFrames" to plan the fastest way to scan a file before doing it
df = pl.scan_csv("huge_data.csv").filter(pl.col("sales") > 500).collect()

2. Deep Learning: PyTorch / TensorFlow

These are used to build Neural Networks—the tech behind AI like ChatGPT or self-driving cars. They are designed to run on powerful Graphics Cards (GPUs) rather than just standard computer processors.

Core Methods & Complete Code Example

  • pd.read_csv(): Converts raw spreadsheet files into interactive data tables called DataFrames.
  • df.dropna() / df.fillna(): Cleans up data by removing rows with missing values or filling in the blanks.
  • train_test_split(): Splices your data into a Training set (to learn from) and a Testing set (to check accuracy).
  • LinearRegression() / KNeighborsClassifier(): Imports standard prediction algorithms directly from the scikit-learn library catalog.
  • model.fit(): The "learning" phase where the model finds patterns in your training data.
  • joblib.dump(): Saves the "brain" of your trained model so you can use it in a web app without retraining it.
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
import joblib

# 1. Load data into a Pandas DataFrame
df = pd.read_csv('real_estate.csv')

# 2. Data Cleaning: Remove any rows with empty values
df = df.dropna()

# 3. Define Features (X) and Target (y)
X = df[['square_feet', 'rooms']]
y = df['price']

# 4. Split data: 80% for training, 20% for testing the model's accuracy
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# 5. Choose an algorithm and "Fit" it (train it) to the data
model = LinearRegression()
model.fit(X_train, y_train)

# 6. Make predictions on the 20% of data the model hasn't seen yet
predictions = model.predict(X_test)

# 7. Save the trained model to a file named 'house_model.pkl'
joblib.dump(model, 'house_model.pkl')