Data Handling Module

The pydis_nn.data module provides functionality for loading, validating, and preprocessing 5D datasets.

Data handling module for loading and preprocessing 5D datasets.

This module provides functionality to load datasets from pickle files, validate dimensions, handle missing values, split into train/val/test sets, and standardize features.

pydis_nn.data.load_raw_dataset(filepath)

Load raw dataset from .pkl file without preprocessing.

Parameters:

filepath (str) – Path to the .pkl file

Return type:

Dict[str, ndarray]

Returns:

Dictionary with keys ‘X’ and ‘y’ containing numpy arrays

Raises:

• FileNotFoundError – If the file doesn’t exist

• ValueError – If the data format is invalid

pydis_nn.data.load_dataset(filepath)

Load a 5D dataset from a pickle file.

Expected format: dictionary with ‘X’ and ‘y’ keys. - ‘X’: numpy array with shape (n_samples, 5) containing features - ‘y’: numpy array with shape (n_samples,) containing targets

Parameters:

filepath (str) – Path to the .pkl file

Return type:

Dict[str, ndarray]

Returns:

Dictionary with keys ‘X’ and ‘y’ containing numpy arrays

Raises:

• FileNotFoundError – If the file doesn’t exist

• ValueError – If the data format is invalid or dimensions are wrong

pydis_nn.data.split_data(X, y, train_size=0.7, val_size=0.15, test_size=0.15, random_state=None)

Split data into train, validation, and test sets.

Parameters:

• X (ndarray) – Feature array (n_samples, n_features)

• y (ndarray) – Target array (n_samples,)

• train_size (float) – Proportion for training set (default: 0.7)

• val_size (float) – Proportion for validation set (default: 0.15)

• test_size (float) – Proportion for test set (default: 0.15)

• random_state (Optional[int]) – Random seed for reproducibility

Return type:

Dict[str, ndarray]

Returns:

Dictionary with keys ‘X_train’, ‘X_val’, ‘X_test’, ‘y_train’, ‘y_val’, ‘y_test’

Raises:

ValueError – If proportions don’t sum to approximately 1.0

pydis_nn.data.standardize_features(X_train, X_val=None, X_test=None)

Standardize features using training set statistics.

Fits scaler on training data, then transforms train/val/test sets.

Parameters:

• X_train (ndarray) – Training features (n_samples, n_features)

• X_val (Optional[ndarray]) – Validation features (optional)

• X_test (Optional[ndarray]) – Test features (optional)

Return type:

Tuple[ndarray, Optional[ndarray], Optional[ndarray], StandardScaler]

Returns:

Tuple of (X_train_scaled, X_val_scaled, X_test_scaled, scaler) If X_val or X_test are None, corresponding output will be None

pydis_nn.data.load_and_preprocess(filepath, train_size=0.7, val_size=0.15, test_size=0.15, standardize=True, random_state=None)

Complete pipeline: load dataset, split, and optionally standardize.

Convenience function that combines all data handling steps.

Parameters:

• filepath (str) – Path to the .pkl file

• train_size (float) – Proportion for training set

• val_size (float) – Proportion for validation set

• test_size (float) – Proportion for test set

• standardize (bool) – Whether to standardize features (default: True)

• random_state (Optional[int]) – Random seed for reproducibility

Returns:

• ‘X_train’, ‘X_val’, ‘X_test’, ‘y_train’, ‘y_val’, ‘y_test’

• ’scaler’ (if standardize=True)

• ’X_train’ etc. will be standardized if standardize=True

Return type:

Dictionary with all data arrays and optionally the scaler

pydis_nn.data.calculate_dataset_statistics(X_raw, y_raw, X_processed, y_processed)

Calculate comprehensive statistics for a dataset.

Computes statistics from both raw (unprocessed) and processed data: - Raw stats: missing values count, feature ranges (pre-standardization) - Processed stats: duplicate rows, memory usage, feature statistics

Parameters:

• X_raw (ndarray) – Raw feature array before preprocessing (n_samples, 5)

• y_raw (ndarray) – Raw target array before preprocessing (n_samples,)

• X_processed (ndarray) – Processed feature array after handling missing values (n_samples, 5)

• y_processed (ndarray) – Processed target array after handling missing values (n_samples,)

Returns:

• ‘missing_values’: int - Count of missing/invalid values in raw data

• ’feature_ranges’: List[Dict] - List of {‘min’: float, ‘max’: float} for each feature (raw)

• ’duplicate_rows’: int - Number of duplicate rows in processed data

• ’memory_usage_mb’: float - Memory usage in MB for processed data

• ’feature_stats’: Dict with min_avg, max_avg, target_mean, target_std, target_min, target_max

Return type:

Dictionary containing

Examples

Loading a dataset:

from pydis_nn.data import load_dataset

data = load_dataset('my_dataset.pkl')
X, y = data['X'], data['y']

Complete preprocessing pipeline:

from pydis_nn.data import load_and_preprocess

data = load_and_preprocess(
    'my_dataset.pkl',
    train_size=0.7,
    val_size=0.15,
    test_size=0.15,
    standardize=True,
    random_state=42
)

X_train = data['X_train']
y_train = data['y_train']