Transforming and Saving Data
Data science is an iterative lifecycle: you load raw records, transform them to extract insights or isolate subsets, and write the formatted data back to disk. In this chapter, we will learn how to extract unique values from column tables, use Boolean indexing for logical filtering, and save results using to_csv().
15.1 Finding Unique Elements (unique())
Real-world datasets often contain millions of rows with repeating labels. To identify the distinct, non-repeating entries in a column, Pandas provides the .unique() method. It filters out all duplicates and returns the unique elements as an array.
# Find all unique release years of albums
unique_years = df['Released'].unique()
print(unique_years)Visualizing Uniqueness: Think of a stack of 13 colorful blocks containing only 3 distinct colors. Applying .unique() returns just those 3 individual colors, disregarding how many times they repeat in the stack.
Click the action button to apply unique() to our stack of 13 colored blocks. Notice how duplicate values fade out, isolating the unique entries.
15.2 Boolean Indexing & Data Filtering
Filtering rows based on logic or inequality thresholds is one of the most common tasks in data preprocessing. To filter data in Pandas, we use conditional expressions to generate a **Boolean Series** (True/False values), then pass that series back inside brackets to filter the DataFrame.
# Step 1: Create a Boolean condition series
condition = df['Released'] >= 1980
# Step 2: Use the condition series to filter rows
df1 = df[condition]Single-Line Shorthand: You can combine these steps into a single line. In squared brackets, you place the inequality condition directly: df1 = df[df['Released'] >= 1980].
Evaluate row values against our condition (Released >= 1980), inspect the generated Boolean Series, and apply the slice to filter out False rows.
| Artist | Album | Released | |
|---|---|---|---|
| 0 | Michael Jackson | Thriller | 1982 |
| 1 | AC/DC | Back in Black | 1980 |
| 2 | Pink Floyd | Dark Side of the Moon | 1973 |
| 3 | Whitney Houston | The Bodyguard | 1992 |
| 4 | Fleetwood Mac | Rumours | 1977 |
15.3 Saving Data: to_csv() and Formats
After transforming your dataset, you must write it back to physical storage. The most common method is to_csv(), which writes dataframes into flat CSV files. To prevent Pandas from adding an unnecessary row index column, always pass index=False.
# Save DataFrame as a CSV file, skipping row index columns
df1.to_csv("filtered_data.csv", index=False).csv for CSV tables or .xlsx for Excel sheets) so operating systems parse them correctly.
Enter a custom filename, toggle parameters, and execute the export function to watch disk operations run.
Practice Quiz
Validate your understanding of unique value filtering, Boolean indexing, and exporting files in Pandas.