Results Manager
PermutationManager
calculate_p_values(true_results, perms)
staticmethod
Calculate p-values based on true results and permutation results.
:param true_results: DataFrame with the true results. :param perms: DataFrame with the permutation results. :return: DataFrame with the calculated p-values.
calculate_p_values_edges_nbs(true_stability, permutation_stability, threshold=0.5, component_stat='extent', alpha=0.05, return_diagnostics=False)
staticmethod
Network-Based Statistic (Zalesky et al., 2010) for edge-stability significance.
Edges whose stability meets threshold form a graph; its connected
components are the candidate subnetworks. A permutation null of the
largest component statistic controls the family-wise error rate, so
an observed component is significant if it is larger/stronger than the
biggest component seen in (almost) any permutation. Each component's
p-value is broadcast onto all of its member edges; every other edge is
assigned p = 1.
Inference is at the subnetwork level: a significant result licenses "this connected subnetwork is selected more consistently than chance", not per-edge claims.
Note on discreteness: stability over K outer folds takes only the
values {0, 1/K, ..., 1}, so threshold=0.5 keeps edges selected
in a majority of folds and the effective thresholding is coarse for few
folds. A continuous edge statistic (deferred) would sharpen this.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
true_stability
|
ndarray of shape (n_nodes, n_nodes, 2, 1)
|
Observed edge stability; dim 2 is the positive/negative network. |
required |
permutation_stability
|
ndarray of shape (n_nodes, n_nodes, 2, n_perms)
|
Edge stability from each permutation run. |
required |
threshold
|
float
|
Stability threshold ( |
0.5
|
component_stat
|
(extent, intensity)
|
|
'extent'
|
alpha
|
float
|
Significance level recorded in the diagnostics. |
0.05
|
return_diagnostics
|
bool
|
If |
False
|
Returns:
| Name | Type | Description |
|---|---|---|
sig_stability |
ndarray of shape (n_nodes, n_nodes, 2)
|
Per-edge p-values (member edges carry their component's p-value). |
diagnostics |
(dict, optional)
|
Returned only when |
calculate_p_values_edges_tfce(true_stability, permutation_stability, E=0.5, H=2.0, dh=0.1, alpha=0.05, return_diagnostics=False)
staticmethod
Threshold-Free Cluster Enhancement (Smith & Nichols, 2009) adapted to networks, for per-edge stability significance without an arbitrary primary threshold.
Each edge's TFCE score integrates the support of the components it belongs to across a sweep of stability thresholds. A permutation max-TFCE null across edges controls the family-wise error rate, so this yields genuine per-edge FWER-corrected p-values (unlike NBS, which is subnetwork-level).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
true_stability
|
ndarray of shape (n_nodes, n_nodes, 2, 1)
|
Observed edge stability; dim 2 is the positive/negative network. |
required |
permutation_stability
|
ndarray of shape (n_nodes, n_nodes, 2, n_perms)
|
Edge stability from each permutation run. |
required |
E
|
float
|
TFCE extent/height exponents (field-standard defaults 0.5 / 2.0). |
0.5
|
H
|
float
|
TFCE extent/height exponents (field-standard defaults 0.5 / 2.0). |
0.5
|
dh
|
float
|
Step of the stability-threshold sweep over |
0.1
|
alpha
|
float
|
Significance level recorded in the diagnostics. |
0.05
|
return_diagnostics
|
bool
|
If |
False
|
Returns:
| Name | Type | Description |
|---|---|---|
sig_stability |
ndarray of shape (n_nodes, n_nodes, 2)
|
Per-edge FWER-corrected p-values. |
diagnostics |
(dict, optional)
|
Returned only when |
calculate_permutation_results(results_directory, logger, method='nbs', nbs_threshold=0.5, nbs_component_stat='extent')
staticmethod
Calculate and save the permutation test results.
Model-level metric p-values are always computed. Edge-stability
significance is established at the subnetwork level via a
Network-Based Statistic (method='nbs') or, threshold-free, via
network TFCE (method='tfce'); both control the family-wise error
rate through a permutation max-statistic and write a per-edge p-value
matrix (edges belonging to a significant subnetwork carry that
subnetwork's p-value) to stability_edges_significance.npy.
:param results_directory: Directory where the results are saved.
:param logger: Logger for progress messages.
:param method: Edge-significance method, 'nbs' (default) or 'tfce'.
:param nbs_threshold: Stability threshold for NBS component forming.
:param nbs_component_stat: NBS component statistic, 'extent' or 'intensity'.
ResultsManager
A class to handle the aggregation, formatting, and saving of results.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output_dir
|
str
|
Directory where results will be saved. |
required |
aggregate_inner_folds()
Calculates increments, aggregates across folds, and saves results.
calculate_edge_stability(write=True, best_param_id=None)
Calculate and save edge stability and overlap.
:param cv_edges: Cross-validation edges. :param results_directory: Directory to save the results.
load_cv_results(folder)
staticmethod
Load cross-validation results from a CSV file.
:param folder: Directory containing the results file. :return: DataFrame with the loaded results.
save_network_strengths()
Save network strengths to CSV.
save_predictions()
Save predictions to CSV.
store_edges(param_idx, fold_idx, edges_tensor)
Stores the edge masks for Positive and Negative networks.
Args: param_idx: Index of current parameter. fold_idx: Index of current fold. edges_tensor: Boolean Tensor of shape [Features, 2, Runs]. Dimension 1 must correspond to [Positive, Negative].
store_metrics(param_idx, fold_idx, metrics_tensor)
Stores a batch of metrics returned by FastCPMMetrics.
Args: param_idx: Index of the current parameter configuration. fold_idx: Index of the current CV fold. metrics_tensor: 4D Tensor [Metrics, Models, Networks, Runs]