Data Cleaning and Manipulation in Pandas:

 Data Cleaning and Manipulation in Pandas:

  1. Handling missing values:

In real-world datasets, missing values are quite common. Pandas provides various functions to handle missing values, such as isna(), fillna(), and dropna().

For example:

python
import pandas as pd 
data = {'Name': ['John', 'Alice', 'Bob', 'Mary'], 'Age': [25, 30, None, 35]} 
df = pd.DataFrame(data) 
df.fillna(0, inplace=True) 
print(df)

This will fill all the missing values in the DataFrame with 0.

  1. Data filtering:

Data filtering is the process of selecting rows or columns based on some condition. Pandas provides various ways to filter data, such as using boolean indexing, query() method, and loc[] and iloc[] methods.

For example:

bash
import pandas as pd 
data = {'Name': ['John', 'Alice', 'Bob', 'Mary'], 'Age': [25, 30, 35, 40]} 
df = pd.DataFrame(data) 
filtered_data = df[df['Age'] > 30] 
print(filtered_data)

This will filter out all the rows where the age is less than or equal to 30.

  1. Data transformation:

Data transformation is the process of converting data from one form to another. Pandas provides various functions to transform data, such as apply(), map(), and replace().

For example:

scss
import pandas as pd 
data = {'Name': ['John', 'Alice', 'Bob', 'Mary'], 'Age': [25, 30, 35, 40]} 
df = pd.DataFrame(data) 
df['Gender'] = ['Male', 'Female', 'Male', 'Female'] 
df['Age'] = df['Age'].apply(lambda x: x + 10) 
print(df)

This will add a new column 'Gender' to the DataFrame and increase the age of all individuals by 10.

  1. Data merging and joining:

Data merging and joining are the processes of combining data from different sources into a single DataFrame. Pandas provides various functions to merge and join data, such as merge(), concat(), and join().

For example:

python
import pandas as pd 
data1 = {'Name': ['John', 'Alice', 'Bob'], 'Age': [25, 30, 35]}
data2 = {'Name': ['Alice', 'Bob', 'Mary'], 'Salary': [50000, 60000, 70000]} 
df1 = pd.DataFrame(data1) 
df2 = pd.DataFrame(data2) 
merged_data = pd.merge(df1, df2, on='Name') 
print(merged_data)

This will merge the two DataFrames based on the 'Name' column and create a new DataFrame that contains the columns 'Name', 'Age', and 'Salary'.

Comments

Post a Comment

Popular posts from this blog

How to use the statsmodels library in Python to calculate Exponential Smoothing

Image
Exponential smoothing is a widely used smoothening technique in business analytics that assigns exponentially decreasing weights to past observations. It is particularly useful for forecasting future values based on historical data. There are three main types of exponential smoothing methods:  simple exponential smoothing, double exponential smoothing, and triple exponential smoothing  (also known as Holt-Winters method). In pandas, you can utilize the  statsmodels  library in Python for exponential smoothing calculations. Here's an example of how to perform exponential smoothing using  statsmodels : The Code: # Import the required libraries import pandas as pd import statsmodels.api as sm # Create a DataFrame with a time series data data = {'Month': ['Jan', 'Feb', 'Mar', 'Apr'], 'Sales': [100, 120, 110, 130]} df = pd.DataFrame(data) # Set the 'Month' column as the index df.set_index('Month', inplace=True) ...
Read more

K-means Clustering 3D Plot Swiss roll Dataset

Image
  K-means is a widely used clustering algorithm in machine learning and data mining. It is an unsupervised learning algorithm that aims to partition a given dataset into distinct groups or clusters based on similarity of data points. The algorithm is called “K-means” because it divides the data into K clusters, where K is a user-specified parameter. K-means aims to minimize the within-cluster sum of squares. The objective is to have the data points within each cluster as similar as possible, while keeping the clusters as distinct as possible. However, it’s important to note that K-means is sensitive to the initial random selection of cluster centers. Therefore, it’s often recommended to run the algorithm multiple times with different initializations and choose the clustering with the lowest inertia. Here’s an example of K-means clustering using Python and machine learning: import numpy as np import matplotlib. pyplot as plt from sklearn. cluster import KMeans from sklearn....
Read more

How to detect Credit Card Fraud Using Python Pandas

Image
  Detecting fraud in credit card transactions is an important application of Machine Learning. Given below is a step-by-step guide on how to approach fraud detection using Python (Pandas and Scikit-Learn) with the Credit Card Fraud Detection Dataset from Kaggle: Data source: Credit Card Fraud Detection Dataset  https://www.kaggle.com/mlg-ulb/creditcardfraud Step 1: Data Preprocessing Start by importing the necessary libraries and loading the dataset into a Pandas DataFrame. import pandas as pd # Load the dataset data = pd.read_csv('creditcard.csv') #replace with the downloaded file path # Explore the dataset print(data.head()) Output: Step 2: Data Exploration Understand the dataset by checking its structure, summary statistics, and class distribution (fraudulent vs. non-fraudulent transactions). # Check the dataset shape print (data.shape) # Check summary statistics #print(data.describe()) # Check class distribution print (data[ 'Class' ].value_counts()) Output: ( 284...
Read more