Graph Embeding LEM Example¶
This example will show you how to use the LEM graph embedding with a PHOTONAI Graph Pipeline
At first we have to import the requirements
In [1]:
Copied!
from photonai.base import Hyperpipe, PipelineElement
from photonai_graph.GraphUtilities import get_random_connectivity_data, get_random_labels
from sklearn.model_selection import KFold
from photonai.base import Hyperpipe, PipelineElement
from photonai_graph.GraphUtilities import get_random_connectivity_data, get_random_labels
from sklearn.model_selection import KFold
DGL backend not selected or invalid. Assuming PyTorch for now.
Setting the default backend to "pytorch". You can change it in the ~/.dgl/config.json file or export the DGLBACKEND environment variable. Valid options are: pytorch, mxnet, tensorflow (all lowercase)
Now we are able to generate some sample data for a classification
In [2]:
Copied!
# make random matrices to simulate connectivity matrices
X = get_random_connectivity_data(number_of_nodes=50, number_of_individuals=100)
y = get_random_labels(l_type="classification", number_of_labels=100)
# make random matrices to simulate connectivity matrices
X = get_random_connectivity_data(number_of_nodes=50, number_of_individuals=100)
y = get_random_labels(l_type="classification", number_of_labels=100)
The next step is the definition of our PHOTONAI pipeline
In [3]:
Copied!
my_pipe = Hyperpipe('basic_gembedding_pipe',
inner_cv=KFold(n_splits=5),
outer_cv=KFold(n_splits=5),
optimizer='sk_opt',
optimizer_params={'n_configurations': 25},
metrics=['accuracy', 'balanced_accuracy', 'recall', 'precision'],
best_config_metric='accuracy')
my_pipe.add(PipelineElement('GraphConstructorThreshold', threshold=0.95))
my_pipe.add(PipelineElement('GraphEmbeddingLocallyLinearEmbedding'))
my_pipe.add(PipelineElement('SVC'))
my_pipe = Hyperpipe('basic_gembedding_pipe',
inner_cv=KFold(n_splits=5),
outer_cv=KFold(n_splits=5),
optimizer='sk_opt',
optimizer_params={'n_configurations': 25},
metrics=['accuracy', 'balanced_accuracy', 'recall', 'precision'],
best_config_metric='accuracy')
my_pipe.add(PipelineElement('GraphConstructorThreshold', threshold=0.95))
my_pipe.add(PipelineElement('GraphEmbeddingLocallyLinearEmbedding'))
my_pipe.add(PipelineElement('SVC'))
Finally we can fit the pipeline
In [4]:
Copied!
my_pipe.fit(X, y)
my_pipe.fit(X, y)
===================================================================================================== PHOTONAI ANALYSIS: basic_gembedding_pipe ===================================================================================================== Found 2 target classes: [0. 1.] ***************************************************************************************************** Outer Cross validation Fold 1 ***************************************************************************************************** Did not find any hyperparameter to convert into skopt space.
/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/photonai/optimization/scikit_optimize/sk_opt.py:169: UserWarning: Did not find any hyperparameter to convert into skopt space. warnings.warn(msg)
-----------------------------------------------------------------------------------------------------
BEST_CONFIG
-----------------------------------------------------------------------------------------------------
{}
-----------------------------------------------------------------------------------------------------
VALIDATION PERFORMANCE
-----------------------------------------------------------------------------------------------------
+-------------------+-------------------+------------------+
| METRIC | PERFORMANCE TRAIN | PERFORMANCE TEST |
+-------------------+-------------------+------------------+
| accuracy | 0.9062 | 0.5250 |
| balanced_accuracy | 0.9030 | 0.5285 |
| recall | 0.9320 | 0.7572 |
| precision | 0.8917 | 0.5347 |
+-------------------+-------------------+------------------+
-----------------------------------------------------------------------------------------------------
TEST PERFORMANCE
-----------------------------------------------------------------------------------------------------
+-------------------+-------------------+------------------+
| METRIC | PERFORMANCE TRAIN | PERFORMANCE TEST |
+-------------------+-------------------+------------------+
| accuracy | 0.9250 | 0.6000 |
| balanced_accuracy | 0.9248 | 0.7143 |
| recall | 0.9286 | 1.0000 |
| precision | 0.9286 | 0.4286 |
+-------------------+-------------------+------------------+
*****************************************************************************************************
Outer Cross validation Fold 2
*****************************************************************************************************
Did not find any hyperparameter to convert into skopt space.
/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/photonai/optimization/scikit_optimize/sk_opt.py:169: UserWarning: Did not find any hyperparameter to convert into skopt space. warnings.warn(msg) /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/metrics/_classification.py:1334: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior. _warn_prf(average, modifier, msg_start, len(result))
-----------------------------------------------------------------------------------------------------
BEST_CONFIG
-----------------------------------------------------------------------------------------------------
{}
-----------------------------------------------------------------------------------------------------
VALIDATION PERFORMANCE
-----------------------------------------------------------------------------------------------------
+-------------------+-------------------+------------------+
| METRIC | PERFORMANCE TRAIN | PERFORMANCE TEST |
+-------------------+-------------------+------------------+
| accuracy | 0.8969 | 0.4750 |
| balanced_accuracy | 0.8932 | 0.5226 |
| recall | 0.8639 | 0.3867 |
| precision | 0.9280 | 0.3822 |
+-------------------+-------------------+------------------+
-----------------------------------------------------------------------------------------------------
TEST PERFORMANCE
-----------------------------------------------------------------------------------------------------
+-------------------+-------------------+------------------+
| METRIC | PERFORMANCE TRAIN | PERFORMANCE TEST |
+-------------------+-------------------+------------------+
| accuracy | 0.8500 | 0.4500 |
| balanced_accuracy | 0.8493 | 0.4293 |
| recall | 0.8205 | 0.2222 |
| precision | 0.8649 | 0.3333 |
+-------------------+-------------------+------------------+
*****************************************************************************************************
Outer Cross validation Fold 3
*****************************************************************************************************
Did not find any hyperparameter to convert into skopt space.
/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/photonai/optimization/scikit_optimize/sk_opt.py:169: UserWarning: Did not find any hyperparameter to convert into skopt space. warnings.warn(msg)
-----------------------------------------------------------------------------------------------------
BEST_CONFIG
-----------------------------------------------------------------------------------------------------
{}
-----------------------------------------------------------------------------------------------------
VALIDATION PERFORMANCE
-----------------------------------------------------------------------------------------------------
+-------------------+-------------------+------------------+
| METRIC | PERFORMANCE TRAIN | PERFORMANCE TEST |
+-------------------+-------------------+------------------+
| accuracy | 0.8812 | 0.5125 |
| balanced_accuracy | 0.8806 | 0.5497 |
| recall | 0.8742 | 0.3897 |
| precision | 0.8749 | 0.6048 |
+-------------------+-------------------+------------------+
-----------------------------------------------------------------------------------------------------
TEST PERFORMANCE
-----------------------------------------------------------------------------------------------------
+-------------------+-------------------+------------------+
| METRIC | PERFORMANCE TRAIN | PERFORMANCE TEST |
+-------------------+-------------------+------------------+
| accuracy | 0.7625 | 0.3000 |
| balanced_accuracy | 0.7613 | 0.3000 |
| recall | 0.7368 | 0.1000 |
| precision | 0.7568 | 0.1667 |
+-------------------+-------------------+------------------+
*****************************************************************************************************
Outer Cross validation Fold 4
*****************************************************************************************************
Did not find any hyperparameter to convert into skopt space.
/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/photonai/optimization/scikit_optimize/sk_opt.py:169: UserWarning: Did not find any hyperparameter to convert into skopt space. warnings.warn(msg)
-----------------------------------------------------------------------------------------------------
BEST_CONFIG
-----------------------------------------------------------------------------------------------------
{}
-----------------------------------------------------------------------------------------------------
VALIDATION PERFORMANCE
-----------------------------------------------------------------------------------------------------
+-------------------+-------------------+------------------+
| METRIC | PERFORMANCE TRAIN | PERFORMANCE TEST |
+-------------------+-------------------+------------------+
| accuracy | 0.9281 | 0.4625 |
| balanced_accuracy | 0.9248 | 0.4622 |
| recall | 0.9097 | 0.1308 |
| precision | 0.9359 | 0.3571 |
+-------------------+-------------------+------------------+
-----------------------------------------------------------------------------------------------------
TEST PERFORMANCE
-----------------------------------------------------------------------------------------------------
+-------------------+-------------------+------------------+
| METRIC | PERFORMANCE TRAIN | PERFORMANCE TEST |
+-------------------+-------------------+------------------+
| accuracy | 0.8625 | 0.5000 |
| balanced_accuracy | 0.8570 | 0.5354 |
| recall | 0.7838 | 0.1818 |
| precision | 0.9062 | 0.6667 |
+-------------------+-------------------+------------------+
*****************************************************************************************************
Outer Cross validation Fold 5
*****************************************************************************************************
Did not find any hyperparameter to convert into skopt space.
/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/photonai/optimization/scikit_optimize/sk_opt.py:169: UserWarning: Did not find any hyperparameter to convert into skopt space. warnings.warn(msg) /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/metrics/_classification.py:1334: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior. _warn_prf(average, modifier, msg_start, len(result))
-----------------------------------------------------------------------------------------------------
BEST_CONFIG
-----------------------------------------------------------------------------------------------------
{}
-----------------------------------------------------------------------------------------------------
VALIDATION PERFORMANCE
-----------------------------------------------------------------------------------------------------
+-------------------+-------------------+------------------+
| METRIC | PERFORMANCE TRAIN | PERFORMANCE TEST |
+-------------------+-------------------+------------------+
| accuracy | 0.8812 | 0.5375 |
| balanced_accuracy | 0.8712 | 0.5576 |
| recall | 0.8012 | 0.3457 |
| precision | 0.9307 | 0.3057 |
+-------------------+-------------------+------------------+
-----------------------------------------------------------------------------------------------------
TEST PERFORMANCE
-----------------------------------------------------------------------------------------------------
+-------------------+-------------------+------------------+
| METRIC | PERFORMANCE TRAIN | PERFORMANCE TEST |
+-------------------+-------------------+------------------+
| accuracy | 0.8750 | 0.5000 |
| balanced_accuracy | 0.8636 | 0.5625 |
| recall | 0.7500 | 0.2500 |
| precision | 0.9643 | 0.7500 |
+-------------------+-------------------+------------------+
*****************************************************************************************************
Finished all outer fold computations.
*****************************************************************************************************
ANALYSIS INFORMATION ================================================================================
Project Folder: /home/runner/work/photonai_graph/photonai_graph/documentation/docs/examples/basic_gembedding_pipe_results_2023-02-17_15-45-36,
Computation Time: 2023-02-17 15:45:36.042304 - 2023-02-17 15:46:46.351014
Duration: 0:01:10.308710
Optimized for: accuracy
Hyperparameter Optimizer: sk_opt
DUMMY RESULTS =======================================================================================
+-------------------+--+
| PERFORMANCE DUMMY | |
+-------------------+--+
+-------------------+--+
AVERAGE PERFORMANCE ACROSS OUTER FOLDS ==============================================================
+-------------------+---------------+--------------+-----------+----------+
| Metric Name | Training Mean | Training Std | Test Mean | Test Std |
+-------------------+---------------+--------------+-----------+----------+
| accuracy | 0.855 | 0.052797 | 0.47 | 0.09798 |
| balanced_accuracy | 0.851203 | 0.052339 | 0.508286 | 0.13837 |
| recall | 0.803942 | 0.068739 | 0.350808 | 0.328515 |
| precision | 0.884146 | 0.071392 | 0.469048 | 0.214233 |
+-------------------+---------------+--------------+-----------+----------+
BEST HYPERPARAMETER CONFIGURATION ===================================================================
{}
+--------+----------+-------------------+--------+-----------+----------------------------+
| fold # | accuracy | balanced_accuracy | recall | precision | Best Hyperparameter Config |
+--------+----------+-------------------+--------+-----------+----------------------------+
| 1* | 0.6000 | 0.7143 | 1.0000 | 0.4286 | {} |
| 2 | 0.4500 | 0.4293 | 0.2222 | 0.3333 | {} |
| 3 | 0.3000 | 0.3000 | 0.1000 | 0.1667 | {} |
| 4 | 0.5000 | 0.5354 | 0.1818 | 0.6667 | {} |
| 5 | 0.5000 | 0.5625 | 0.2500 | 0.7500 | {} |
+--------+----------+-------------------+--------+-----------+----------------------------+
PHOTONAI 2.3.0 ======================================================================================
Your results are stored in /home/runner/work/photonai_graph/photonai_graph/documentation/docs/examples/basic_gembedding_pipe_results_2023-02-17_15-45-36
Go to https://explorer.photon-ai.com and upload your photonai_results.json for convenient result visualization!
For more info and documentation visit https://www.photon-ai.com
--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) File /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/IPython/core/formatters.py:972, in MimeBundleFormatter.__call__(self, obj, include, exclude) 969 method = get_real_method(obj, self.print_method) 971 if method is not None: --> 972 return method(include=include, exclude=exclude) 973 return None 974 else: File /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/base.py:631, in BaseEstimator._repr_mimebundle_(self, **kwargs) 629 output = {"text/plain": repr(self)} 630 if get_config()["display"] == "diagram": --> 631 output["text/html"] = estimator_html_repr(self) 632 return output File /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/utils/_estimator_html_repr.py:417, in estimator_html_repr(estimator) 403 fallback_msg = ( 404 "In a Jupyter environment, please rerun this cell to show the HTML" 405 " representation or trust the notebook. <br />On GitHub, the" 406 " HTML representation is unable to render, please try loading this page" 407 " with nbviewer.org." 408 ) 409 out.write( 410 f"<style>{style_with_id}</style>" 411 f'<div id="{container_id}" class="sk-top-container">' (...) 415 '<div class="sk-container" hidden>' 416 ) --> 417 _write_estimator_html( 418 out, 419 estimator, 420 estimator.__class__.__name__, 421 estimator_str, 422 first_call=True, 423 ) 424 out.write("</div></div>") 426 html_output = out.getvalue() File /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/utils/_estimator_html_repr.py:151, in _write_estimator_html(out, estimator, estimator_label, estimator_label_details, first_call) 149 """Write estimator to html in serial, parallel, or by itself (single).""" 150 if first_call: --> 151 est_block = _get_visual_block(estimator) 152 else: 153 with config_context(print_changed_only=True): File /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/utils/_estimator_html_repr.py:127, in _get_visual_block(estimator) 123 # check if estimator looks like a meta estimator wraps estimators 124 if hasattr(estimator, "get_params"): 125 estimators = [ 126 (key, est) --> 127 for key, est in estimator.get_params(deep=False).items() 128 if hasattr(est, "get_params") and hasattr(est, "fit") 129 ] 130 if estimators: 131 return _VisualBlock( 132 "parallel", 133 [est for _, est in estimators], 134 names=[f"{key}: {est.__class__.__name__}" for key, est in estimators], 135 name_details=[str(est) for _, est in estimators], 136 ) File /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/base.py:211, in BaseEstimator.get_params(self, deep) 209 out = dict() 210 for key in self._get_param_names(): --> 211 value = getattr(self, key) 212 if deep and hasattr(value, "get_params") and not isinstance(value, type): 213 deep_items = value.get_params().items() AttributeError: 'Hyperpipe' object has no attribute 'best_config_metric'
--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) File /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/IPython/core/formatters.py:342, in BaseFormatter.__call__(self, obj) 340 method = get_real_method(obj, self.print_method) 341 if method is not None: --> 342 return method() 343 return None 344 else: File /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/base.py:625, in BaseEstimator._repr_html_inner(self) 620 def _repr_html_inner(self): 621 """This function is returned by the @property `_repr_html_` to make 622 `hasattr(estimator, "_repr_html_") return `True` or `False` depending 623 on `get_config()["display"]`. 624 """ --> 625 return estimator_html_repr(self) File /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/utils/_estimator_html_repr.py:417, in estimator_html_repr(estimator) 403 fallback_msg = ( 404 "In a Jupyter environment, please rerun this cell to show the HTML" 405 " representation or trust the notebook. <br />On GitHub, the" 406 " HTML representation is unable to render, please try loading this page" 407 " with nbviewer.org." 408 ) 409 out.write( 410 f"<style>{style_with_id}</style>" 411 f'<div id="{container_id}" class="sk-top-container">' (...) 415 '<div class="sk-container" hidden>' 416 ) --> 417 _write_estimator_html( 418 out, 419 estimator, 420 estimator.__class__.__name__, 421 estimator_str, 422 first_call=True, 423 ) 424 out.write("</div></div>") 426 html_output = out.getvalue() File /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/utils/_estimator_html_repr.py:151, in _write_estimator_html(out, estimator, estimator_label, estimator_label_details, first_call) 149 """Write estimator to html in serial, parallel, or by itself (single).""" 150 if first_call: --> 151 est_block = _get_visual_block(estimator) 152 else: 153 with config_context(print_changed_only=True): File /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/utils/_estimator_html_repr.py:127, in _get_visual_block(estimator) 123 # check if estimator looks like a meta estimator wraps estimators 124 if hasattr(estimator, "get_params"): 125 estimators = [ 126 (key, est) --> 127 for key, est in estimator.get_params(deep=False).items() 128 if hasattr(est, "get_params") and hasattr(est, "fit") 129 ] 130 if estimators: 131 return _VisualBlock( 132 "parallel", 133 [est for _, est in estimators], 134 names=[f"{key}: {est.__class__.__name__}" for key, est in estimators], 135 name_details=[str(est) for _, est in estimators], 136 ) File /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/sklearn/base.py:211, in BaseEstimator.get_params(self, deep) 209 out = dict() 210 for key in self._get_param_names(): --> 211 value = getattr(self, key) 212 if deep and hasattr(value, "get_params") and not isinstance(value, type): 213 deep_items = value.get_params().items() AttributeError: 'Hyperpipe' object has no attribute 'best_config_metric'
Out[4]:
Hyperpipe(name='basic_gembedding_pipe')