Making collect_data(), Part 5: Cleaning the Data 1

Python for Data Science Automation (Course 1)

Welcome to Python for Data Science Automation

🚀Python for Data Science Automation: Let's Do This! (2:18)
Course Prerequisites
Software Preview: Python, VSCode, & Conda Information
🌐 Private BSU Slack Community: How to Join
🏆 Course Certificate: How to Get It
Emoji Guide ⚠️💪😃
[Optional] Video Subtitles (Captions)
Would You Like To Become An Affiliate (And Earn 20% On Your Sales)?

Python Packages Used In This Course

Full Python Package List And Versions For the Course

Getting Help (Important!)

⚠️ [IMPORTANT] How to Get Help

Part 1 - Foundations of Data Analysis with Python

The Game Plan: Data Analysis Foundations (0:45)

Module 0: Business Project & Course Setup

📈 The Business Case: Building an Automated Forecast System (0:43)

0.1 Course Project Download

🔽 Course Project Zip [File Download] (0:43)

0.2 Key Resources

🗺️ Course Workflow: Tying Specific Actions to the Business Process (2:26)
📑 Ultimate Python Cheat Sheet: Python Ecosystem in 2 Pages (4:02)
🔽 The Transactional Database Model [PDF Download] (3:34)

0.3 Software Downloads

Anaconda Installation (2:36)
IDE (Integrated Development Environment) Options (2:18)
VSCode Installation (1:37)

0.4A CONDA ENVIRONMENT SETUP

3 Options For Recreating The Python Environments Used In The Course
[IMPORTANT] A quick word on Python Environments...
All Windows Users DO THIS!!!
Connect VSCode to Your Course Project Files (1:12)
Conda Env Create: Make the Python Course Environment (4:39)
Python Interpreter Selection (Requires VSCode Python Extension) (1:08)
Conda Env Update: Add Python Packages to Your Environment (1:43)
Conda Env Export: Review & Share Your Environment (1:11)
Conda Env List & Remove: List Available Environments & Remove Unnecessary Envs (1:21)
🆘 Mac: Troubleshooting Environment Guidance
🆘 Windows PowerShell: All Windows Users DO THIS!!!
🆘 Windows Users: Troubleshooting Guidance

0.4B VIRTUAL ENVIRONMENT SETUP (Alternative to Conda Environment Setup)

🔽 Virtual Environments: Creating And Troubleshooting (62:23)

04C: EASY INSTALL (Less Reproducible, but Will Get You Moving Forward)

Ok, so none of the environment setups are working for you... Here's what to do

0.5 VSCode Setup (for Data Science)

Getting to Know VSCode (1:14)
VSCode Theme Customization (2:06)
VSCode Icon Themes (0:43)
VSCode User & Workspace Settings (4:15)
VSCode Keyboard Shortcuts (1:16)
VSCode Python Extensions (3:22)
VSCode Jupyter Extension - Jupyter Notebook Support (2:04)
VSCode Jupyter Extension - Interactive Python (3:34)
[Optional VSCode Setting] Jupyter: Send Selection to Interactive Window (2:30)
VSCode Excel Viewer (1:00)
VSCode Markdown & PDF Extensions (2:42)
VSCode Path Intellisense (1:08)
VSCode SQLite Extension (0:40)
[Optional] VSCode Extensions for R Users (1:26)

0.6 Recap & Code Checkpoint

🔽 Python Environment Checkpoint [File Download] (3:52)

Module 1: Jumpstart - Sales Analysis (Time Series)

🔽 Getting Started [File Download] (4:07)
Using the Cheat Sheet (1:25)

1.1 Importing from Python Packages

Import: pandas, numpy, matplotlib.pyplot (3:33)
Importing From: plotnine, miziani (4:41)
Importing Functions and Submodules: os, rich (2:08)
Setting Up Python Interactive (2:44)
⚙️ [Reminder | Optional VSCode Setting] Jupyter: Send Selection to Interactive Window (2:30)

1.2 Data Import

Getting Help Documentation (2:46)
⚠️⚙️IMPORTANT VSCODE SETTING: File Paths | jupyter.notebookFileRoot (6:34)
Reading the Excel Files (6:45)
🔗 Checkpoint Link

1.3 Examining the Data

Reviewing the Data Model (5:09)
Exploratory 1: Top 5 Most Frequent Descriptions (3:54)
Exploratory 2: Plotting the Top 5 Bike Descriptions (6:22)
🔗 Checkpoint Link

1.4 Joining the Excel Tables

Preparing Orderlines for Merge: Drop Column (3:04)
Merging the Bikes DataFrame (3:33)
Merging the Bikeshops Data Frame (3:26)
🔗 Checkpoint Link

1.5 Data Wrangling (Transformation)

Datetime: Converting Order Date | Copy vs No Copy (4:50)
Splitting the Description: Category 1, Category 2, and Frame Material (7:26)
Splitting Location: City, State (3:03)
Create the Total Price Column (2:53)
Reorganizing the Columns (4:43)
Renaming Columns (4:05)
Reviewing the Data Transformations (1:11)
🔗 Checkpoint Link

1.6 Time Series Plotting

Save Your Work: Pickle it. (3:49)
🔗 Pandas Datetime Accessors (2:43)

1.6.1 Sales Analysis, Part 1 - Sales By Month

🔗Resampling: Working with Pandas Offsets (7:25)
Quick Plot: Plotting Single Time Series w/ Pandas Matplotlib Backend (1:40)
Plotnine Visualization: Sales By Month, Part 1 - Geometries (5:52)
Plotnine Visualization: Sales by Month, Part 2 - Scales & Themes (5:50)
🔗 Checkpoint Link

1.6.2 Sales Analysis, Part 2 - Sales by Week & Category 2

Resampling Groups: Combine groupby() and resample() (9:22)
Quick Plot: Plotting Multiple Time Series w/ Pandas Matplotlib Backend (7:23)
Plotnine Visualization, Part 1: Facetted Sales By Date & Category2 (Group) (8:57)
Plotnine Visualization, Part 2: Adding Themes & Scales (8:52)
🔗 Checkpoint Link

1.7 Save & Recap

Writing Files: Pickle, CSV, Excel (4:41)
🎉 Congrats. That was a fun whirlwind. Let's recap. (2:34)

1.8 Checkpoint: Module 1 - Jumpstart

🔽 Code Checkpoint Zip - Module 1 [File Download]

Module 2: SQL Databases & Python Packages

🔽 Getting Started [File Download] (1:21)

2.1 Importing Files

Pickle Files 🥒 (3:40)
CSV Files 💾 (3:58)
Excel Files 👔 (3:25)
🔗 Checkpoint Link

2.2 SQL Alchemy Databases

SQL Databases 🗂️ (1:46)
Pandas I/O & SQL Alchemy Overviews (3:01)
Make Database Directory (1:23)
🔗 Checkpoint Link

2.2.1 Creating a Database

Create the SQLite Database (4:19)
Read the Excel Files (3:03)
Create the Database Tables (7:11)
Close the Connection (0:53)
🔗 Checkpoint Link

2.2.2 Reconnecting to the Database

Connect to the Database (2:07)
Getting the Database Table Names (2:34)
Reading from the Tables with f-strings (1:47)
⚙️ [Bonus] VSCode SQLite Extension (3:04)
🔗 Checkpoint Link

2.3 Creating a Function: collect_data()

Making collect_data(), Part 1: Function Setup (6:39)
Making collect_data(), Part 2: Read Tables from the Database (8:38)
Making collect_data(), Part 3: Test the Database Import (1:14)
Making collect_data(), Part 4: Joining the Data (8:25)
Making collect_data(), Part 5: Cleaning the Data 1 (7:14)
Making collect_data(), Part 6: Cleaning the Data 2 (6:48)
⚙️ Making collect_data(), Part 7: VSCode Docstring Generator (3:58)
🔗 Checkpoint Link

2.4 Intro to Python Packages & Modules

📦 Making a Package (my_pandas_extensions): Adding the database module (4:41)
🥳Congrats! You're learning really powerful concepts. (1:06)

2.5 Checkpoint: Module 2 - SQL Databases

🔽 Code Checkpoint Zip - Module 2 [File Download]

3.0 Pandas Data Analysis Deep Dive

🔽 Getting Started [File Download] (2:24)
⚠️⚙️ [VSCode Setting] Jupyter: Send Selection to Interactive Window (1:12)

3.1 Data Structures for Analysis

Package & Function Imports (1:28)
My Pandas Extensions: Fix FutureWarning Message (regex) (1:28)

3.1.1 Python Objects (How Python Works)

How Python Works: Objects (5:29)

3.1.2 Data Structures for Analysis

Pandas DataFrame & Series (2:51)
Numpy Arrays (4:08)
🔗 Checkpoint Link

3.1.3 Python's Built-in Data Structures & Data Types

Python Builtin Data Structures: Dictionary, List, Tuple (5:53)
Python Builtin Data Types: Int, Float, Str, Bool, (3:41)
🔗 Checkpoint Link

3.1.4 Casting (Data Type Conversion)

Casting Basics: Numeric & String Conversions (4:09)
Casting Sequences: To List, Numpy Array, Pandas Series, & DataFrame (2:40)
Pandas Series Dtype Conversion (1:43)
🔗 Checkpoint Link

3.2 Pandas Core Functions: Deep-Dive

Pandas Data Wrangling Setup (2:08)

3.2.1 Column-Wise Data Frame Selection

Subsetting Columns by Name (2:16)
Subsetting by Column Index (Position): iloc[] (1:35)
💪 Subsetting Columns with Regex (Regular Expressions) (3:37)
Rearranging a Single Column (Column Subsetting) (2:16)
Rearranging Multiple Columns (Repetitive Way First) (1:43)
Rearranging Multiple Columns (List Comprehension) (2:50)
💪 Data Frame Rearrange: Select Dtypes, Concat, & Drop (6:32)
🔗 Checkpoint Link

3.2.2 Arranging Rows

💪 Sort Values (3:06)
🔗 Checkpoint Link

3.2.3 Rowwise Filtering (Slicing)

💪 Simple Filters with Boolean Series (3:54)
💪 Query Filters (3:47)
Filtering with isin() and ~ (3:40)
Index slicing with df.iloc[] (2:41)
Getting Distinct Values: Drop duplicates (1:43)
N-Largest and N-Smallest (2:14)
Random Samples (1:52)
🔗 Checkpoint Link

3.2.4 Calculated Columns (Mutating / Assigning)

DataFrame Column Assignment: Calculated Columns (2:24)
💪 Assign Basics: Lambda Functions (3:10)
Assign Cookbook: Making a Log Transformation (3:31)
Assign Cookbook: Searching Text (Boolean Flags) (5:26)
Assign Cookbook: Even-Width Binning with pd.cut() (3:45)
✨Visualizing Binning Strategies with a Pandas Heat Table (2:59)
Assign Cookbook: Quantile Binning with pd.qcut() (2:35)
🔗 Checkpoint Link

3.2.5 Grouping & Aggregate / Apply

➕ Aggregation Basics (Summarizations) (5:48)
Common Summary Functions (4:10)
💪 Groupby + Aggregate Basics (Summarizations) (5:26)
Groupby + Agg Cookbook (♻️Summary DF 1): Sum & Median Total Price By Category 1 & 2 (3:13)
Groupby + Agg Cookbook (♻️Summary DF 2): Sum Total Price & Quantity By Category 1 & 2 (3:23)
Groupby + Agg Details: Examining the Multilevel Column Index (2:00)
Groupby + Agg Cookbook (♻️Summary DF 3): Grouping Time Series with Groupby & Resample (4:11)
💪 Groupby + Apply Basics (Transformations) (3:41)
Groupby + Apply Cookbook: Transform All Columns by Group (2:33)
Groupby + Apply Cookbook: Filtering Slices by Group (3:24)
🔗 Checkpoint Link

3.2.6 Renaming Columns

Renaming Basics: Renaming All Columns with Lambda (4:27)
Renaming Basics: Targeting Specific Columns (1:20)
Advanced Renaming: Renaming Multi-Index Columns (5:56)
🔗 Checkpoint Link

3.2.7 Reshaping (Pivoting)

Set Up Summarized Data: Revenue by Category 1 (4:59)
💪 Pivot: To Wide Format (6:41)
✨Export a Stylized Pandas Table to Excel (Wide Data) (6:08)
💪 Melt: To Long Format (3:30)
✨Plotnine - Making a Faceted Horizontal Bar Chart (Tidy Long Data) (4:33)
Intro to Categorical Data: Sorting the Plotnine Plot (6:08)
💪 Pivot Table (An awesome function for BI Tables) (7:41)
Unstack: A programmatic version of pivot() (4:09)
Stack: A programmatic version of melt() (2:24)
🔗 Checkpoint Link

3.2.8 Joins

💪 Merge: Data Frame Joins (4:11)
💪 Concat: Binding DataFrames Rowwise & Columnwise (4:26)
🔗 Checkpoint Link

3.2.9 Splitting & Uniting Text Columns

🖖 Splitting Text Columns (3:07)
🤞 Combining Text Columns (1:05)
🔗 Checkpoint Link

3.2.10 The Apply Function (In-Depth)

Set Up Summarized Data: Sales by Category 2 Daily (3:01)
Apply: Lambda Aggregations vs Transformations (2:21)
Apply: Broadcasting Aggregations (1:52)
Grouped Apply: Broadcasting (2:23)
Grouped Transform: Alternative to Grouped Apply (Fixes Index Issue) (2:02)
🔗 Checkpoint Link

3.2.11 [Advanced] The Pipe: Method Chaining Helper

Making a "Data Frame" Function: add_columns() (6:06)
Pipe: Method chaining our custom function using the pipe (3:11)
🔗 Checkpoint Link

3.3 Code Checkpoint: Module 3 - Pandas Core

🔽 Code Checkpoint Zip - Module 3 [File Download]

⛰️ 3.4 Challenge #1: Test Your Data Wrangling Skills

🔽 Challenge #1: Data Wrangling with Pandas [File Download] (1:15)

💡 3.4.1 Introduction to JupyterLab Notebooks

Method 1: Jupyter VSCode Integration (2:24)
Method 2: Jupyter Notebooks (Legacy Method) (2:06)
🆒 Method 3: JupyterLab (Next Generation of Jupyter) (3:15)

3.4.2 Challenge #1 Assignment

Challenge Objectives (3:07)
⚠️ Getting Started: Syncing Your JupyterLab Current Working Directory (%cd and %pwd) (5:09)
Challenge Tasks (3:17)
🆘 Challenge Solution (8:39)
🎉 Congrats! You've finished your first challenge. (1:36)

Part 2: Time Series Forecasting Automation

Automating Time Series Forecasting (1:39)

4.0 Data Exploration & Time Series Fundamentals

🔽 Getting Started [File Download] (1:48)
⚠️⚙️ VSCode Extension: Browser Preview (1:39)

4.1 Pandas Profiling

Package Imports (1:39)
The ProfileReport() Class (1:09)

4.1.1 Profile Report Components

Section 1: Profile Overview (3:18)
Section 2A: Numeric & Date Variables (6:02)
Section 2B: Categorical (Text) Variables (5:01)
Sections 3-6: Interactions, Correlations, Missing Values, & Sample (2:51)
🔗 Checkpoint Link - Module 4

4.1.2 Profiling Workflow

Pandas Extension: df.profile_report() (3:08)
Exporting the Profile Report as HTML (1:35)
🔗 Checkpoint Link - Module 4

4.2 Time Series Fundamentals

Getting Started (0:49)

4.2.1 Pandas Datetime Basics

TimeStamp & Period Conversions (2:58)
Pandas Datetime Accessors (1:55)
Date Math: Offsetting Time with TimeDelta's (2:38)
Date Math: Getting Duration between Two TimeStamps (3:27)
Creating Date Sequences: pd.date_range() (3:08)
🔗 Checkpoint Link - Module 4

4.2.2 Periods & Time-Based Groupings (Resampling)

💪Periods (In-Depth) (7:57)
💪Resampling (In-Depth): ♻️ bike_sales_m_df (6:23)
💪Grouped Resampling (In-Depth): ♻️ bike_sales_cat2_m_wide_df (6:37)
Reorganizing: Adding Comments (1:30)
🔗 Checkpoint Link - Module 4

4.2.3 Measuring Change

Differencing with Lags (Single Time Series) (5:38)
Differencing with Lags (Multiple Time Series) (2:04)
Difference from First (Single Time Series) (1:42)
Difference From First (Multiple Time Series) (0:57)
🔗 Checkpoint Link - Module 4

4.2.4 Cumulative Calculations (Expanding Windows)

Cumulative Expanding Windows (Single Time Series) (3:20)
Cumulative Expanding Windows (Multiple Time Series) (1:37)
🔗 Checkpoint Link - Module 4

4.2.5 Rolling Window Calculations

Moving Average (Single Time Series) (8:14)
Moving Average (Multiple Time Series) (4:36)
🔗 Checkpoint Link - Module 4

4.4 Code Checkpoint: Module 4 - Time Series Fundamentals

🗺️ Next Steps (Where we are headed) (1:16)
🔽 Code Checkpoint Zip - Module 4 [File Download]

5.0 Functional Programming

🔽 Getting Started [File Download] (1:36)

5.1 Functional Programming with Pandas [Outlier Detection Function]

Setup: Python Imports & Data (0:45)

5.1.1 Examining Pandas Functions

Function Anatomy: pd.Series.max() (3:52)
Errors (Exceptions) (1:02)
Function Names (1:16)
Function Anatomy: **kwargs (5:11)
🔗 Checkpoint Link - Module 5

5.1.2 Detect Outliers Function

Detect Outliers: Function Setup (2:18)

5.1.2.1 Building an Outlier Detection Function

IQR Outlier Method, Part 1 (3:36)
IQR Method, Part 2 (4:06)
New Argument: IQR Multiplier (1:46)
New Argument: How? (Both, Upper, Lower) (2:35)
🔗 Checkpoint Link - Module 5

5.1.2.2 Adding Checks: Raising Informative Exceptions

Checking for Pandas Series Input (2:10)
Checking IQR Multiplier for Int or Float Type (2:53)
Checking that IQR Multiplier is a Positive Value (1:09)
Checking that How is a Valid Option: both, lower, upper (2:18)
🔗 Checkpoint Link - Module 5

5.1.2.3 Finalizing the Function

⚙️ Informative Help Documentation: Adding a Docstring (7:10)
Testing Our Function: Detecting Outliers within Groups (3:04)
Extending the Pandas Series Class (2:11)
🔗 Checkpoint Link - Module 5

5.2 Summarize By Time Function

Summarize By Time: A handy function for time series wrangling (3:59)

5.2.1 Beginning the Summarize by Time Function

Setting Up the "Summarize By Time" Function (4:55)
Handling the Date Column Input (1:30)
Handling Groups Input (2:02)
Handling the Time Series Resample (4:13)
Handling the Aggregation Function Input (3:15)
Handling the Value Column Input (1:39)
🔗 Checkpoint Link - Module 5

5.2.2 Debugging the Value Column 🐜

Forcing the Value Column Input to a List (to generate a data frame) (2:43)
🐜 Bug! Thinking through a solution (2:25)
Solution: Converting to a Function Dictionary with Zip + Dict (3:51)
🔗 Checkpoint Link - Module 5

5.2.3 Handling the Output Formatting (Wide Data)

Handling the Unstack (2:01)
Handling the Period Conversion (2:50)
Add Fill Missing Capability (2:24)
🔗 Checkpoint Link - Module 5

5.2.4 Finalizing the Function & Adding to Our Extensions Package

Review the Core Functionality (1:24)
Check Incoming Data: Raising a TypeError (1:49)
Adding the Docstring (7:27)
⚠️📦Pandas Flavor: Extending Pandas DataFrame Class (6:22)
🔗 Checkpoint Link - Module 5

5.3 Code Checkpoint: Module 5 - Functional Programming

🔽 Code Checkpoint Zip - Module 5 [File Download]

6.0 Forecasting with Sktime

🔽 Getting Started [File Download] (3:02)
📘 Sktime Documentation (4:35)
🌐 How to Google Search like a Pro (1:34)

6.1 Introduction to Sktime Forecasting

Set Up & Imports (2:40)

6.1.1 Data Summarizations (Always Needed Prior to Forecasting)

Summarizing to get Total Revenue by Month (4:59)
Summarizing to get Total Revenue by Category 2 & Month (2:40)
🔗 Checkpoint Link - Module 6

6.1.2 Single Time Series Forecasting: AutoARIMA()

💡 What is AutoARIMA? (4:58)
💪 AutoARIMA Applied: Forecaster, Fit, Predict (8:24)
Adding Confidence Intervals (Prediction Intervals) (2:40)
Tuple Unpacking (Predictions, Confidence Intervals) (2:38)
📈 Forecast Visualization (5:27)
Code Housekeeping 🧹 (0:23)
🔗 Checkpoint Link - Module 6

6.1.3 Multiple Time Series Forecasting (Modeling within For-Loops)

Multiple Time Series Forecasting: AutoARIMA() (3:09)
For Loop: Iterate Across the DataFrame Columns (2:19)
For Loop: Modeling AutoARIMA() (5:22)
For-Loop: Getting the Confidence Intervals (1:31)
For-Loop: Combine with DataFrame | Actual Values, Predictions, & CIs (4:11)
For-Loop: Storing the Results (as a Dictionary) (3:35)
Housekeeping: Appending Variable Types to Variable Names (1:52)
Visual Forecast Assessment (2:42)
TQDM: Progress Bars (3:40)
🔗 Checkpoint Link - Module 6

6.2 The ARIMA Forecast Automation

Setting up the ARIMA Automation Function (3:44)

6.2.1 Building the ARIMA Forecasting Automation

Making arima_forecast() | Function Definition (3:18)
Function Body | Setting Up the Iteration (4:40)
Training the AutoARIMA() Models (3:01)
Controlling Progress Bars: tqdm(min_interval) (1:11)
Making Predictions and Confidence Intervals (2:08)
Combine Results into a DataFrame (2:23)
Compose a Prediction Dictionary (1:49)
Return Results as a Single DataFrame | Rowwise Concatenation (2:36)
Setting the Column Names of the Output (9:14)
Drop remaining columns beginning with "level_" (2:50)
Testing the arima_forecast() function (2:04)
🔗 Checkpoint Link - Module 6

6.2.2 The forecasting.py Module

Creating the forecasting.py module (3:43)
Docstring: arima_forecast() (1:31)
Adding Checks: arima_forecast() (6:34)
Finally - Check Your Forecasts with Grouped Pandas Plotting (2:28)
Recap: You've just made an ARIMA Forecast Automation! (1:09)
🔗 Checkpoint Link - Module 6

6.3 Checkpoint

🔽 Code Checkpoint Zip - Module 6 [File Download]

Challenge 2: Make an AutoETS() Forecast Automation

Introduction to ETS Forecasting (Exponential Smoothing) (2:06)
🔽 Challenge 2 [File Download] (6:06)
Solution (5:17)

Part 3: Visualization & Report Automation

Part 3: Visualization & Reporting (1:23)

7.0 Visualization with Plotnine

🔽 Getting Started [File Download] (0:31)
📘 Plotnine Documentation (3:14)

7.1 Plotnine Basics

Plotnine Anatomy: Imports (2:56)

7.1.1 Data and Geometry Basics

Data Summarization: For Plotting Annual Bike Sales (2:53)
The Plot Canvas: Mapping Columns to Plot Components (7:12)
Plotnine Geometries (3:59)
Adding a Trend Line: geom_smooth() (2:59)
🔗 Checkpoint Link - Module 7

7.1.2 Formatting Basics

Formatting Plots (1:52)
Expand Limits (1:41)
Scales: Dollar Format for Y-Axis (3:51)
Scales: Date Format for X-Axis (2:14)
Labs and Themes (2:57)
Saving the ggplot (1:17)
Exploring the Plotnine Object (2:23)
🔗 Checkpoint Link - Module 7

7.2 Plotnine In-Depth

Setting Up (2:19)

7.2.1 Scatter Plot and Line Plot | geom_point() and geom_line()

Scatter Plot: Data Manipulation (2:51)
Scatter Plot: Visualization (3:16)
Line Plot: Data Manipulation (2:07)
Line Plot: Visualization (5:28)
🔗 Checkpoint Link - Module 7

7.2.2 Column Plot (Bar Plot) | geom_col()

Data Manipulation, Part 1: No Categorical Ordering (2:48)
Visualization, Part 1: Without Categorical Ordering (1:34)
Aside: Introduction to Plotting using Categorical Data Type (9:54)
Finalizing the Horizontal Bar Chart (1:19)
🔗 Checkpoint Link - Module 7

7.2.3 Histograms and Density Plots | geom_histogram() and geom_density()

Histogram: Data Manipulation (2:47)
Histogram: Visualization (2:14)
Histogram: Using Fill Aesthetic to Explore Differences by a Category (2:50)
Histogram: Using Facet Grids to Compare Distributions by Category (2:47)
Density Plots: Kernel Density Estimation (KDE) using geom_density() (2:54)
🔗 Checkpoint Link - Module 7

7.2.4 Box Plots and Violin Plots | geom_boxplot() and geom_violin()

Box Plot: Data Manipulation (2:19)
Box Plot: Visualization (7:23)
Violin Plot with Jitter: geom_violin() and geom_jitter() (3:31)
🔗 Checkpoint Link - Module 7

7.2.5 Text and Labels | geom_text() and geom_label()

Data Manipulation: Add a Total Price Text Column with USD Dollar Format (6:08)
Creating the Bar Plot: geom_col() and geom_smooth() (3:10)
Adding Text to a Bar Plot: geom_text() (5:34)
Highlighting an Outlier with a Label: geom_label() (5:57)
Finalizing the Plot with Scales and Themes (3:38)
🔗 Checkpoint Link - Module 7

7.2.6 Plot Customizations | Facets, Scales, & Themes

Sales by Month and Category 2: Data Manipulation (4:40)
Facets: Adding subplots "facets" with facet_wrap() (6:54)
Scales: Applying scales to alter x, y, and color mappings (4:33)
Themes: Theme Customization with Pre-Built Themes | theme_matplotlib() (3:52)
Theme Elements: Customization with theme() (5:32)
Plot Title and X/Y-Axis Labels: labs() (4:43)
🔗 Checkpoint Link - Module 7

Mini-Challenge: Convert our Cyberpunk 2077 Plot to a Business Science Theme

Challenge: Change the Dark Theme to Business Science Theme
Solution: Business Science Theme

7.3 Building the Plot Forecast Function

Getting Started (1:20)
Package Imports (2:09)
Our Forecasting Workflow Recap (4:59)

7.3.1 Forecast Plot

Data Preparation: Melting the Value and Prediction Columns (4:50)
Data Preparation: Fixing the FutureWarning (3:03)
Visualization: Setting up the canvas with ggplot() (3:34)
Visualization: Adding geoms and facets (5:42)
Visualization: Scales and Theme Minimal (5:10)
Visualization: Customizing the Theme Elements (4:21)
🔗 Checkpoint Link - Module 7

7.3.2 Forecast Plot Automation

Making the plot_forecast() Function Definition (3:22)
Data Wrangling: Implementing the Melt (4:37)
Handling the Time-Based Column: Converting to TimeStamp (8:51)
Visualization: Parameterizing the Plot (8:55)
Testing the Forecast Plot Function Parameters (7:09)
Testing the Automation Workflow (1:29)
Reordering the Subplots using Cat Tools (6:14)
Adding the plot_forecast() function to our forecasting module (3:35)
🔽 Docstring | Testing Our Imported plot_forecast() Function (3:29)
🔗 Checkpoint Link - Module 7

7.4 Code Checkpoint - Module 7

🔽 Code Checkpoint Zip - Module 7 [File Download]

8.0 Debugging: Building & Debugging the Database Read/Write Automation

🔽 Getting Started [File Download] (1:55)

8.1 Database Update: Developing Read/Write Functionality

Package Imports (2:32)
Reviewing Our Files (1:36)
Generating the Forecasting Workflow (5:25)
Generating the Forecast Visualization (1:37)
Overview of the Database I/O Process (1:26)
🔗 Checkpoint Link - Module 8

8.1.1 Preparation & Validation Functionality

Preparing the Forecast for Update (5:43)
Validating the Column Names (6:10)
Testing the Prep Forecast for Database Function (1:14)
🔗 Checkpoint Link - Module 8

8.1.2 Write Forecast to Database Function

Setting Up the Write Forecast to Database Function (5:27)
Modularizing the Data Preparation Step (1:18)
Specifying SQL Data Types (6:46)
Write to Database (6:49)
Close Connection (0:54)
Testing Our Function (4:28)
🔗 Checkpoint Link - Module 8

8.1.3 Read Forecast from Database Function

Creating our Read Forecast Function (5:59)
🔗 Checkpoint Link - Module 8

8.1.4 Add Functions to Database Module

Adding Functions to Database Module (4:18)
📄 Docstrings (3:06)
🔗 Checkpoint Link - Module 8

8.2 Debugging the Automation Workflow

Automation Workflow with Database I/O (2:43)

8.2.1 Testing the Summarizing & Forecasting Automation Workflow

Forecasting 1: Total Revenue (4:42)
Fix #1: Reorder Columns in Prep Data Function (4:29)
Plotting Total Revenue Forecast (1:24)
Forecasting 2: Revenue by Category 1 (5:32)
Forecasting 3: Revenue by Category 2 (4:36)
Forecasting 4: Forecast Quarterly Revenue by Customer (5:33)
Fix #2: Prep Data | Add timestamp conversion (1:32)
Rerun Our Workflow: Success! (3:19)
🔗 Checkpoint Link - Module 8

8.2.2 Testing the Write Database Workflow

Writing to the Database (3:14)
💡 Pro-Tip: Saving Intermediate Data (1:45)
Utility Function: Convert to Datetime (6:58)
Rerun the Forecast Workflow (3:45)
Read Forecast from Database (2:00)
Recap: Debugging is a Skill (3:41)
🔗 Checkpoint Link - Module 8

8.3 Code Checkpoint - Module 8

🔽 Code Checkpoint Zip - Module 8 [File Download]

Module 9 | Jupyter Reporting Automation (Part 1): Automating Jupyter Notebooks

Jupyter Automated Reporting (1:15)
🔽 Getting Started [File Download] (2:53)

9.0 Automatic Forecasting... The Update Database Script

The Updated Database Script: Automatically Run Forecasts (5:48)
python update_database.py (3:27)
SQLite Explorer (1:22)

9.1 Jupyter Sales Analysis

Setting Up the Working Directory (6:13)
Importing Data and Parameterizing a Header with Markdown (6:11)
Parameterizing a Paragraph with Markdown (3:55)
Performance Summary: Pivot Table, Part 1 (5:34)
Performance Summary: Pivot Table, Part 2 (2:13)
Plotting the Forecast: plot_forecast() (2:03)
🔗 Checkpoint Link - Module 9

9.2 Papermill

Papermill Setup (1:15)
Package Imports (2:11)
Papermill Documentation (3:18)
🔗 Checkpoint Link - Module 9

9.2.1 Making Parameters

Developing Parameters: Game Plan (2:10)
Making ID Sets, Part 1 (3:20)
Making ID Sets, Part 2 (4:31)
🔗 Checkpoint Link - Module 9

9.2.2 Setting Up the Directory (Programmatically)

Part 1: Intro to Pathlib and OS (3:01)
Part 2: Detecting Directories Exist & Making New Directories (4:41)
🔗 Checkpoint Link - Module 9

9.2.3 Creating the Jupyter Template

Jupyter Template Setup (2:56)
Parameterizing the Jupyter Template (3:48)
Finishing the Juyter Template Parameterization (3:46)
🔗 Checkpoint Link - Module 9

9.2.4 Automating Jupyter Notebooks with Papermill

The pm.exectute_notebook() function (4:05)
Setting Up Key Parameters (6:01)
Iterating without a For-Loop (6:08)
Iterating with a For-Loop (6:00)
🔗 Checkpoint Link - Module 9

9.3 Run Reports Function: Version 1

Getting Started (1:07)
Setting Up the Report Parameters (3:32)
🔗 Checkpoint Link - Module 9

9.3.1 Handling Paths

Creating a Resource Path (1:28)
String Transformation: Make File Names from Report Titles (6:31)
🔗 Checkpoint Link - Module 9

9.3.2 Building the Reporting Function

Setting Up run_reports() (2:45)
Make the Report Directory (4:47)
Setting Up the For-Loop Parameters (6:22)
Setting Up Jupyter Notebook Execution (Inside of For-Loop) (4:01)
🔗 Checkpoint Link - Module 9

9.3.3 Adding Run Reports to the Python Module

Package Resources: Setting Up the Template Path (6:03)
Integrating the Run Reports Function into Our Package (4:34)
🔗 Checkpoint Link - Module 9

9.4 Checkpoint: Module 9 - Jupyter Automation, Part 1

🔽 Code Checkpoint Zip - Module 9 [File Download]

Module 10: Jupyter Automation (Part 2): Automating HTML & PDF Reports

🔽 Getting Started [File Download] (3:31)

10.0 NB Convert: HTML & PDF Conversion Setup

NB Convert Documentation & Installation Requirements (1:49)
Step 1: Pandoc Installation (1:05)
Step 2: Tex Installation (MikTex Windows Shown | Mac Use MacTex) (1:05)
HTML Report Conversion (1:33)
PDF Report Conversion (1:03)

10.1 NB Convert Automation

Setup & Imports (4:21)
Making the Config() (4:22)
Locating Files with Glob (3:19)
Exporting an HTML Report Programmatically (6:38)
HTML Automation: Using a For-Loop to Convert All 4 Reports (5:50)
PDF Automation: Using a For-Loop to Convert All 4 Reports (5:59)
🔗 Checkpoint Link - Module 10

10.2 Run Reports Function, Version 2: run_reports()

Getting Set Up (2:41)
Integrating glob: Pulling the Jupyter Notebook File Paths (2:46)
🏗️ Integrate "Convert to HTML" Report Automation (4:11)
🧪 Test "Convert to HTML" Report Automation (2:31)
🏗️ Integrate "Convert to PDF" Report Automation (1:46)
🧪 Test "Convert to PDF" Report Automation (3:39)
🔗 Checkpoint Link - Module 10

10.3 Upgrading our My Pandas Extensions & Making run_forecast_reports.py

My Pandas Extensions: Upgrade reporting.py with HTML & PDF Reports, Part 1 (6:31)
My Pandas Extensions: Upgrade reporting.py with HTML & PDF Reports, Part 2 (4:00)
Resources for Improving PDF Reports
Run Forecast Reports Py: Part 1 - The main() function (5:07)
Run Forecast Reports Py: Part 2 - Adding Timestamps to Folders (6:01)
Run Forecast Reports Py: Part 3 - Running Reports (2:56)
Run Forecast Reports Py: Part 4 - Adjusting Folder Automation (3:13)
🔗 Checkpoint Link - Module 10

10.4 Checkpoint: Module 10 - Jupyter Automation, Part 2

🔽 Checkpoint - Module 10 [File Download]

✨✨✨ BONUSES: Scheduling Python Scripts

Scheduling Python Scripts Bonus!!! ✨ (0:27)

BONUS #1: Windows Task Scheduler

Making the Batch File (.bat) to run our Python Script (2:25)
Setting up Automated Tasks with Windows Task Scheduler (2:13)
Debugging Windows Task Scheduler Tasks with Pause (0:39)
Fixing the SQL Alchemy Connection (1:52)
Removing the Automation: Disable & Delete (0:23)

BONUS #2: Mac Automator & Calendar App Automation

Python Script Setup | SQL Database Absolute Path (2:30)
The Mac Automator (3:22)
Scheduling the Automator App with Calendar (2:00)

CONGRATULATIONS!!! You Did It! 🥳🥳🥳

Congratulations!!! (1:10)

SPECIAL BONUS CONTENT 😲

About the Special Bonus Content
Forecasting 100 Time Series in Python with Sktime (93:10)

Making collect_data(), Part 5: Cleaning the Data 1

Lesson content locked

If you're already enrolled, you'll need to login.

Enroll in Course to Unlock