The world runs on data. It has contributed to some of the most significant innovations in the last decade, with AI and machine learning at the forefront. From fears of users developing feelings for ChatGPT to luxury retailers like Michael Kors deploying AI virtual assistants, data is the lifeblood of the digital AI-oriented world we live in.
If you're looking to harness the potential for AI and machine learning in your business, you must first ensure that your data is in the correct format. This is where data normalization comes in. Loosely defined, it is the process of transforming numerical and text data into a standard format that can help improve the accuracy of a machine-learning model.
With this in mind, today's article will explore:
- Data normalization and why it's important
- Common data normalization techniques
- How to apply these techniques in Python
Keep reading below for more insights.
What Is Data Normalization?
Data normalization is one of the most important steps in data preprocessing. Without it, getting insights from any information becomes difficult, especially when working with varied datasets.
For example, if you're comparing the performance of students in a class across different subjects and these subjects have varying grading scales, you may use data normalization techniques to convert the grades into a similar scale for easier comparison.
Take an example of math, science, English, and history with the following grading scales:
- Math: 0 to 100
- English: 0 to 50
- Science: 0 to 80
- History: 0 to 30
A student scores the following:
- Math: 80
- English: 35
- Science: 50
- History: 20
At first glance, it might seem that math is way more critical than history because these subjects use different grading systems. Comparing a student's performance in the current data format can be quite complicated. However, data normalization can make it easier by rescaling these scores to a standard range, typically from 0 to 1 or -1 to 1.
First, let's prepare this data for normalization by getting the percentage score for each subject:
The maximum percentage is 0.80 (math), and the minimum is 0.625 (science).
One of the simplest ways to normalize data is through simple feature scaling, also known as Min-Max Scaling. This technique divides each value by the maximum value for that feature. We'll use it to manually normalize the performance data above. Here is the formula:
Normalized Value = (X - X_min) / (X_max - X_min)
Where:
- X is the original value.
- X_min is the minimum value of the feature in the dataset.
- X_max is the maximum value of the feature in the dataset.
We can normalize this data as follows:
- Math:
Normalized Value = (0.80- 0.625) / (0.80- 0.625) = 1.000
- English:
Normalized Value = (0.70- 0.625) / (0.80- 0.625) = 0.429
- Science:
Normalized Value = (0.625- 0,625) / (0.80 - 0.625) = 0.000
- History:
Normalized Value = (0.67- 0.625) / (0.80 - 0.625) = 0.257
Now you have:
- Math: 1.00
- English: 0.429
- Science: 0.000
- History: 0.257
The performance is now at a standard scale. Drawing insights from this data, such as the subject that was best and least performed, is easier. And this is what data normalization seeks to achieve.
Why Data Normalization Is Important
Data normalization is important when preprocessing data for algorithms and machine learning models. To understand why, let's look at our previous school performance example. Bringing up the values:
- Math: 80
- English: 35
- Science: 50
- History: 20
Suppose we want to use this data to train a machine learning algorithm. Feeding these data points into the system in their current format can cause various problems down the road. To explain why, we will offer a broad, general explanation without delving too much into the intricate details.
Think of the machine learning process as a system of different-sized hills; the lowest point possible represents the lowest error, while the highest peak represents the highest error. Naturally, the algorithm aims to get to the lowest point possible.
Most modern AI models, especially those that utilize neural networks, function by iteratively adjusting the model parameters. At each iteration, the algorithm adjusts the parameters in a direction that takes it to the lowest point possible.
Some hills, like math, with a score of 80, are big hills, while some, like history, with a score of 20, are small hills. They are all equally important, but the algorithm doesn't know this since it processes data in its raw form. As a result, it may focus more on the big hills, neglecting the small hills, which can lead to biased and inaccurate predictions.
This is why data normalization is important. It levels the hills, ensuring none is seemingly more important than the other. In this way, the model can take balanced and accurate steps toward the lowest point possible, otherwise known as convergence.
In short, normalization data analysis methods ensure AI models don't focus on any one feature just because it has a bigger range. By normalizing your data on a standard scale, the algorithm will consider each data point equally, helping it reach convergence faster and more accurately.
Understanding Data Normalization in Python
We're now at the crux of our discussion, where we'll show you how to normalize data in Python. But before we get to that, why the Python programming language? There are many languages; what makes this one suitable for normalizing data? Keep reading below to find out.
Robust Data Normalization Libraries
One of the primary reasons to use Python to normalize your data is the rich ecosystem of libraries this platform offers. They include
- Pandas
Use this library for its wide array of data analysis and structuring tools.
- Scikit-learn
This is a comprehensive library that includes prebuilt functions for normalization, such as MinMaxScaler and StandardScaler.
- NumPy
This library supports working with arrays and matrices, making it useful for custom normalization tasks.
Easy-to-Use Syntax
Python, as a programming language, has a fluid syntax that makes it easy to read and understand code. With strong community support, code maintenance, and debugging are also relatively straightforward, with many free resources available online.
Integration With Machine Learning Frameworks
Python seamlessly integrates with popular machine learning frameworks such as Tensorflow and PyTorch. This naturally makes it the language of choice for normalizing input data for machine learning models.
Now that you understand the versatility of this programming language, you may be wondering how to normalize data in Python. Don’t worry—we’ve got you covered. Keep reading to find out more:
How to Normalize Data in Python
Below is a rundown of the different techniques for normalizing data in Python:
Min-Max Scaling
Remember the formula we used to normalize our student score results? It's known as Min-Max Scaling because it takes the minimum and maximum values of a given dataset and uses them to compute the normalized value. Going back to our previous example:
- Math: 80 out of 100
- English: 35 out of 50
- Science: 50 out of 80
- History: 20 out of 30
Let us normalize this data using Min-Max Scaling in Python. Run the following code in your editor of choice with all the prerequisites installed:
from sklearn.preprocessing import MinMaxScaler
import pandas as pd
# Original data
data = {'Subject': ['Math', 'English', 'Science', 'History'],
'Score': [80, 35, 50, 20],
'Max_Score': [100, 50, 80, 30]}
# Convert to DataFrame
df = pd.DataFrame(data)
# Calculate the percentage score
df['Percentage'] = df['Score'] / df['Max_Score']
# Initialize the MinMaxScaler
scaler = MinMaxScaler()
# Fit and transform the percentage scores
df['Normalized'] = scaler.fit_transform(df[['Percentage']])
# Display the normalized data
print(df[['Subject', 'Normalized']])
This is the response you'll get:
Subject Normalized
0 Math 1.000000
1 English 0.428571
2 Science 0.000000
3 History 0.238095
Notice that the results are similar to the normalized values we calculated manually. This process is much faster and can save you a lot of time when working with large datasets.
Z-Score Scaling
While the min-max method uses the minimum and maximum values in a given data set for normalization, the Z-Score scaling method centers normalized data around the standard deviation, ensuring that each feature contributes equally to the model. Here is the formula:
Z = (X - μ) / σ
Where:
- Z is the Z-score (standardized value).
- X is the original value.
- μ (mu) is the mean of the feature.
- σ (sigma) is the standard deviation of the feature.
Instead of spending time calculating manually, especially for large datasets, you can use Python to normalize data using Z-score scaling. Applying our previous example, here is the code you should use:
from sklearn.preprocessing import StandardScaler
import pandas as pd
# Original data
data = {'Subject': ['Math', 'English', 'Science', 'History'],
'Score': [80, 35, 50, 20]}
# Convert to DataFrame
df = pd.DataFrame(data)
# Initialize the StandardScaler
scaler = StandardScaler()
# Fit and transform the data
df['Z-Score'] = scaler.fit_transform(df[['Score']])
# Display the standardized data
print(df[['Subject', 'Z-Score']])
This is the response you should get:
Subject Z-Score
0 Math 1.095445
1 English -0.297076
2 Science 0.000000
3 History -0.798369
Now, if you're wondering why the results differ, it's because we used a different approach. The Z-score measures the number of standard deviations a value is from the mean.
For example, Science, which had the lowest score, is 0 standard deviations from the mean, while Math, the highest-scoring subject, is 1.095 standard deviations from the mean. As you can see, the interpretation of the data remains consistent across the two techniques discussed.
MaxAbs Scaling
This data normalization technique is useful for datasets with positive and negative values. It helps normalize the data within a specific range, typically from -1 to 1, by dividing each feature by its maximum absolute value using this formula:
X_scaled = X / max(|X|)
Where:
- X_scaled is the scaled value of the feature.
- X is the original value of the feature.
- max(|X|) is the maximum absolute value of the feature.
We're going to use a different example to show you how this method works. Consider a dataset with the following values:
We'll use the following Python code to get the normalized values using the MaxAbsScaler for feature A:
from sklearn.preprocessing import MaxAbsScaler
import pandas as pd
# Original data
data = {'Feature': [10, -20, 15, -5]}
df = pd.DataFrame(data)
# Initialize the MaxAbsScaler
scaler = MaxAbsScaler()
# Fit and transform the data
df['Scaled'] = scaler.fit_transform(df[['Feature']])
# Display the scaled data
print(df[['Feature', 'Scaled']])
This is the result you should get:
Feature Scaled
0 10 0.50
1 -20 -1.00
2 15 0.75
3 -5 -0.25
We now have a standard range from -1 to 0.5, which is much easier for an algorithm to interpret.
Decimal Scaling
This technique is suitable for datasets with varied decimal points. It normalizes data by moving the decimal point depending on the absolute value of the feature. Here is the formula:
X_scaled = X / 10^j
Where:
- X_scaled is the scaled value of the feature.
- X is the original value of the feature.
- 10^j is 10 raised to the power of j, which is determined based on the maximum value of the feature.
Let's normalize the following dataset using decimal scaling in Python:
- Values: 0.345, -1.789, 2.456, -0.678
Here is the code:
import pandas as pd
# Original data with decimal points
data = {'Feature': [0.345, -1.789, 2.456, -0.678]}
df = pd.DataFrame(data)
# Find the maximum absolute value in the dataset
max_abs_value = df['Feature'].abs().max()
# Determine the scaling factor
# Find the smallest power of 10 greater than the maximum absolute value
import math
scaling_factor = 10 ** math.ceil(math.log10(max_abs_value))
# Apply Decimal Scaling
df['Scaled'] = df['Feature'] / scaling_factor
# Display the original and scaled data
print(df)
This is the result:
Feature Scaled
0 0.345 0.0345
1 -1.789 -0.1789
2 2.456 0.2456
3 -0.678 -0.0678
How to Normalize Text Data in Python
As you perform data analysis and preparation for your machine learning model, you may also have to normalize text data. Unlike numerical values, normalizing text data typically includes lowercasing, removing punctuation, and tokenization. For example, you may have to split individual text into units called tokens, which can be phrases or symbols. Say we wanted to split this text:
"Tokenization splits texts into words."
Use the following code to normalize this text data:
import nltk
from nltk.tokenize import word_tokenize
# Download the necessary NLTK resource
nltk.download('punkt')
# Sample text
text = "Tokenization splits text into words."
# Tokenize the text
tokens = word_tokenize(text)
# Display the tokens
print(tokens)
The tokenized text should be as follows:
['Tokenization', 'splits', 'text', 'into', 'words', '.']
Conclusion
We have come to the end of our discussion on data normalization methods. This process is essential for preparing input data for machine learning models. While you can manually normalize data, it's impractical for large datasets.
Python is preferred because it has comprehensive libraries, a clean fluid syntax, and it's open source. The methods we've discussed include Min-Max, Decimal, and Z-Score Scaling. Now, you can train your machine learning models with greater accuracy!
FAQ
What is the difference between scaling and normalization?
While both are common data preprocessing techniques, there's a subtle difference between them. Normalization attempts to adjust the range of values by fitting them into a specific range, whereas scaling attempts to adjust a range of feature values without necessarily fitting them into a specific range.
What is the difference between standardization and normalization?
Standardization transforms data to have a mean of zero and a standard deviation of one without binding it to a specific range. Normalization rescales data to fit within a specific range, say [0,1] or [-1,1].
What is the difference between normalized data and denormalized data?
Normalized data has been transformed to fit within a specific range as part of machine learning data preprocessing. On the other hand, denormalized data has been reverted back to its original form.
