CSV Data Analyzer

A comprehensive skill for loading, exploring, cleaning, and analyzing CSV datasets within research workflows. Designed for researchers who need to quickly understand the structure, quality, and statistical properties of tabular data before conducting deeper analysis.

Overview

Research datasets commonly arrive as CSV files from instrument exports, survey platforms, government repositories, and collaborator handoffs. This skill provides a structured approach to the entire CSV analysis pipeline: ingestion, profiling, quality assessment, cleaning, transformation, and summary statistics. It emphasizes reproducibility by generating audit logs of every transformation applied to the raw data.

The skill supports datasets of varying complexity, from single-table survey results to multi-file longitudinal study exports with hundreds of columns. It works with standard Python data science libraries (pandas, numpy, scipy) and produces outputs suitable for inclusion in methods sections and supplementary materials.

Data Loading and Initial Profiling

Loading Strategies

import pandas as pd
import numpy as np

def load_and_profile_csv(filepath: str, encoding: str = 'utf-8') -> dict:
    """
    Load a CSV file and generate an initial data profile.
    Handles common encoding issues and delimiter detection.
    """
    # Try multiple encodings if default fails
    encodings = [encoding, 'latin-1', 'utf-8-sig', 'cp1252']
    df = None
    for enc in encodings:
        try:
            df = pd.read_csv(filepath, encoding=enc, low_memory=False)
            break
        except (UnicodeDecodeError, pd.errors.ParserError):
            continue

    if df is None:
        raise ValueError(f"Could not parse {filepath} with any supported encoding")

    profile = {
        'rows': len(df),
        'columns': len(df.columns),
        'memory_mb': df.memory_usage(deep=True).sum() / 1e6,
        'dtypes': df.dtypes.value_counts().to_dict(),
        'missing_pct': (df.isnull().sum() / len(df) * 100).to_dict(),
        'duplicates': df.duplicated().sum(),
        'column_names': df.columns.tolist()
    }
    return df, profile

Column Type Inference

def infer_semantic_types(df: pd.DataFrame) -> dict:
    """
    Infer semantic column types beyond pandas dtypes.
    Detects dates, identifiers, categorical, continuous, and text columns.
    """
    semantic_types = {}
    for col in df.columns:
        nunique = df[col].nunique()
        ratio = nunique / len(df) if len(df) > 0 else 0

        if ratio > 0.95 and df[col].dtype == 'object':
            semantic_types[col] = 'identifier'
        elif nunique <= 20 and df[col].dtype in ['object', 'int64']:
            semantic_types[col] = 'categorical'
        elif df[col].dtype in ['float64', 'int64']:
            semantic_types[col] = 'continuous'
        elif pd.to_datetime(df[col], errors='coerce').notna().mean() > 0.8:
            semantic_types[col] = 'datetime'
        else:
            semantic_types[col] = 'text'
    return semantic_types