4.3Data science: NumPy, pandas, Matplotlib
The essentials in 30 seconds
- The trio: NumPy computes fast (vectorised arrays), pandas handles tables (DataFrame), Matplotlib shows (1 chart = 1 message).
- Nabil's method: inspect first (
head/info/describe/isna) then aggregate (groupby + sort= 80% of reports) — never the reverse. - Vectorise, never loop:
(df.montant * 1.2).mean()replaces loops, 100x faster on one million rows.
Running mini-case — Nabil answers in 5 lines
Management asks Nabil, in Algiers: “what are our 5 best cities this quarter?” Instead of a day sorting in a spreadsheet, Nabil loads the CSV with pandas, inspects (head, info, isna for gaps), aggregates by city with groupby and sort, then draws a titled bar chart. Answer delivered in one hour, reproducible next quarter by rerunning the script. Third milestone of the running thread: the analyst who answers fast, and proves what he claims.
Sequel to cases 4.1-4.2 — Nabil analyses the sales he exposes.
1. NumPy: compute fast, pandas: handle tables
NumPy replaces loops with vectorised arrays computed in C: (a * 1.2).mean() processes one million rows at once — 100x faster for ÷10 lines of code. pandas is the programmable spreadsheet: the DataFrame (rows/columns table) loads (read_csv), inspects (head, info, describe, isna), filters (df[df.ville == "Alger"]), aggregates (groupby(...).agg(...)) and merges (merge). Nabil ALWAYS inspects before analysing: types, missing values and orders of magnitude first — cleaning is the longest and most important step.
2. Matplotlib: show, and the Jupyter notebook
Matplotlib turns figures into a message: 1 chart = 1 message, with a title stating the conclusion (“Algiers drives 40% of revenue” rather than “Revenue by city”). df.plot() is enough to explore; a polished figure (title, labels, source) serves to communicate. The Jupyter notebook mixes code, results and narrative: ideal to explore and tell the analysis step by step. Nabil remembers the rule: pandas replaces the spreadsheet for large volumes and reproducibility; the spreadsheet stays perfect for quick small visual volumes.
Did you know?
The bear is only the data analysts' unofficial mascot.
🔎 Concrete example — 🔎 Concrete example — Nabil's top cities
Nabil's complete analysis in 5 lines:
df = pd.read_csv("ventes.csv")
print(df.info(), df.isna().sum()) # inspect first
top = df.groupby("ville")["montant"].sum().sort_values(ascending=False)
print(top.head()) # the figured answer
top.head().plot(kind="bar", title="Revenue by city")Reading: inspect → aggregate → show, in that order.
Lesson: groupby + sort = 80% of activity reports.
Section vocabulary
- NumPy
- Computing library on vectorised arrays (C operations): replaces Python loops, 100x faster on large volumes.
Simple example: Nabil applies a discount to one million amounts in one line.
What it is not: a for loop processing rows one by one.
Reflex: as soon as you loop over numbers, look for the vectorised operation.
- pandas / DataFrame
- Programmable spreadsheet: the DataFrame (rows/columns table) loads, filters, aggregates and merges data — read_csv, groupby, merge.
Simple example: Nabil aggregates his sales by city in two lines.
What it is not: manual sorts repeated in a spreadsheet.
Reflex: inspect (head/info/isna) before any analysis.
- groupby + aggregation
- Grouping rows by category then summarising (sum, mean): the move producing 80% of reports.
Simple example: Nabil's revenue by city: groupby city, sum of amounts, sort.
What it is not: totals recalculated by hand at each request.
Reflex: question by category? groupby first.
- Matplotlib
- Charting library: 1 chart = 1 message, with a title stating the conclusion.
Simple example: Nabil titles his bar chart with the conclusion, not just “Revenue by city”.
What it is not: ten decorative charts with no message.
Reflex: message-title, labels, source — otherwise it is decoration.
In practice — In practice — Analyse a CSV end to end
- Load a CSV with pd.read_csv and inspect: head, info, describe, isna().sum() — note types and gaps.
- Clean the vital minimum: date/amount types, missing values flagged or handled.
- Filter (df[condition]) then aggregate (groupby + agg + sort) to answer ONE precise question.
- Draw 1 chart with a message-title, then rerun the whole script at once to prove it is reproducible.
Ready-to-say script — presenting your analysis (30 s)
“This is Nabil: I loaded the sales, inspected gaps and types, then aggregated by city — Algiers drives most of the quarter. The script reruns identically next quarter, and the chart states the conclusion in its title.”
Say in a business review; show the chart, not the code.
Copy-paste template — pandas analysis (tested code)
import pandas as pd
import matplotlib.pyplot as plt
df = pd.read_csv("ventes.csv")
# 1. Inspect first: types, gaps, orders of magnitude
print(df.head())
print(df.info())
print(df.describe())
print(df.isna().sum())
# 2. Filter: Algiers sales this quarter
alger = df[df["ville"] == "Alger"]
# 3. Aggregate: revenue by city, sorted (80% of reports)
top = df.groupby("ville")["montant"].sum().sort_values(ascending=False)
print(top.head())
# 4. Show: 1 chart = 1 message
top.head().plot(kind="bar", title="Algiers drives most revenue")
plt.tight_layout()
plt.savefig("top-villes.png")
Small file
A few hundred rows? The spreadsheet stays faster — keep pandas for reproducibility.
Monthly report
Like Nabil: one script loading, inspecting, aggregating and exporting — rerun each month untouched.
Million rows
Vectorise everything (never for over rows): NumPy/pandas in C do in seconds what loops do in minutes.
Free exploration
Work in a Jupyter notebook: code, results and narrative side by side to search before concluding.
✓ Best practices
- Inspect (head/info/describe/isna) before analysing.
- Vectorise: never loop over millions of rows.
- One chart = one message, concluding title.
✗ Common pitfalls
- Looping with for over millions of rows.
- Analysing without inspecting types and gaps.
- Multiplying charts with no message.
Key takeaways
- TRIO: NUMPY (vectorised computing) / PANDAS (tables) / MATPLOTLIB (charts).
- DATAFRAME = programmable spreadsheet: read_csv, head, describe, groupby, merge.
- METHOD: inspect → aggregate → show; cleaning = longest step.
- JUPYTER = analysis narrative; 1 chart = 1 message with conclusion-title.
48h challenge — apply tomorrow
Take 1 real CSV from your work: inspect it, answer 1 question via groupby, and deliver 1 message-titled chart.
- I can name NumPy, pandas/DataFrame, Matplotlib.
- I can inspect then aggregate a CSV.
- I have 1 reproducible analysis with 1 message-chart.
Frequently asked questions
Does pandas replace Excel?
For large volumes and reproducibility: yes — a script redo in seconds what a spreadsheet redoes in hours, with no copy-paste error. For quick small visual volumes: no — the spreadsheet stays unbeatable. Nabil uses both: pandas for monthly reports, the spreadsheet for a glance.
Do I need to be strong in maths for data science?
Not at this stage: basic statistics (mean, median, distribution) suffice to inspect and aggregate with pandas. Advanced maths (algebra, probability) becomes useful in 4.4 to choose and tune machine learning models.
What is the core structure of pandas?