Results Manager

PermutationManager

calculate_p_values(true_results, perms) staticmethod

Calculate p-values based on true results and permutation results.

:param true_results: DataFrame with the true results. :param perms: DataFrame with the permutation results. :return: DataFrame with the calculated p-values.

calculate_p_values_edges_fdr(true_stability, permutation_stability) staticmethod

Calculate FDR-corrected p-values for each edge in a connectivity matrix using permutation-based empirical p-values and the Benjamini–Yekutieli procedure.

For each edge, an empirical p-value is calculated by comparing the true stability score to the distribution of permuted scores at the same edge. The Benjamini–Yekutieli (BY) method is then applied to correct for multiple comparisons, controlling the false discovery rate (FDR).

Parameters:

Name	Type	Description	Default
true_stability	ndarray of shape (n_regions, n_regions)	Symmetric matrix containing the observed stability scores for each edge.	required
permutation_stability	ndarray of shape (n_permutations, n_regions, n_regions)	Array containing stability scores from each permutation run. Each entry is a symmetric matrix of the same shape as true_stability.	required

Returns:

Name	Type	Description
sig_stability	ndarray of shape (n_regions, n_regions)	Symmetric matrix of FDR-corrected p-values for each edge, calculated by first computing empirical p-values and then applying the Benjamini–Yekutieli correction to control the expected false discovery rate across all edges.

calculate_p_values_edges_max_value(true_stability, permutation_stability) staticmethod

Calculate empirical p-values for each edge in a connectivity matrix using the max-value method from permutation testing.

For each permutation, the maximum value across all edges is taken to construct a max-null distribution. Each true edge value is then compared to this distribution to compute a p-value, which controls the family-wise error rate (FWER).

Parameters:

Name	Type	Description	Default
true_stability	ndarray of shape (n_regions, n_regions)	Symmetric matrix containing the observed stability scores for each edge.	required
permutation_stability	ndarray of shape (n_permutations, n_regions, n_regions)	Array containing stability scores from each permutation run. Each entry is a symmetric matrix of the same shape as true_stability.	required

Returns:

Name	Type	Description
sig_stability	ndarray of shape (n_regions, n_regions)	Symmetric matrix of empirical p-values for each edge, calculated by comparing the true stability values to the max null distribution. The p-values reflect the probability of observing a value as extreme or more extreme under the null. Family-wise error is controlled via the max-statistic method.

calculate_permutation_results(results_directory, logger) staticmethod

Calculate and save the permutation test results.

:param results_directory: Directory where the results are saved.

ResultsManager

A class to handle the aggregation, formatting, and saving of results.

Parameters:

Name	Type	Description	Default
output_dir	str	Directory where results will be saved.	required

calculate_edge_stability(write=True, best_param_id=None)

Calculate and save edge stability and overlap.

:param cv_edges: Cross-validation edges. :param results_directory: Directory to save the results.

calculate_final_cv_results()

Calculate mean and standard deviation of cross-validation results and save to CSV.

:param cv_results: DataFrame with cross-validation results. :param results_directory: Directory to save the results. :return: Updated cross-validation results DataFrame.

calculate_model_increments()

Calculate model increments comparing full model to a baseline.

:param cv_results: Cross-validation results. :param metrics: List of metrics to calculate. :return: Cross-validation results with increments.

initialize_edges(n_folds, n_features, n_params=None) staticmethod

Initialize a dictionary to store edges for cross-validation.

:param n_folds: Number of outer folds. :param n_features: Number of features in the data. :return: Dictionary to store edges.

load_cv_results(folder) staticmethod

Load cross-validation results from a CSV file.

:param folder: Directory containing the results file. :return: DataFrame with the loaded results.

save_network_strengths()

Save network strengths to CSV.

save_predictions()

Save predictions to CSV.

store_metrics(metrics, params, fold, param_id)

Update metrics DataFrame with new metrics and parameters.

:param metrics: Dictionary with computed metrics. :param params: Best hyperparameters from inner cross-validation. :param fold: Current fold number. :return: Updated metrics DataFrame.

store_predictions(y_pred, y_true, params, fold, param_id, test_indices)

Update predictions DataFrame with new predictions and parameters.

:param y_pred: Predicted values. :param y_true: True values. :param params: Best hyperparameters from inner cross-validation. :param fold: Current fold number. :return: Updated predictions DataFrame.

update_results_directory(output_dir)

Determine the directory to save results.

:param output_dir: :return: Results directory path.