Problem Statement¶
Business Context¶
Renewable energy sources play an increasingly important role in the global energy mix, as the effort to reduce the environmental impact of energy production increases.
Out of all the renewable energy alternatives, wind energy is one of the most developed technologies worldwide. The U.S Department of Energy has put together a guide to achieving operational efficiency using predictive maintenance practices.
Predictive maintenance uses sensor information and analysis methods to measure and predict degradation and future component capability. The idea behind predictive maintenance is that failure patterns are predictable and if component failure can be predicted accurately and the component is replaced before it fails, the costs of operation and maintenance will be much lower.
The sensors fitted across different machines involved in the process of energy generation collect data related to various environmental factors (temperature, humidity, wind speed, etc.) and additional features related to various parts of the wind turbine (gearbox, tower, blades, break, etc.).
Objective¶
“ReneWind” is a company working on improving the machinery/processes involved in the production of wind energy using machine learning and has collected data of generator failure of wind turbines using sensors. They have shared a ciphered version of the data, as the data collected through sensors is confidential (the type of data collected varies with companies). Data has 40 predictors, 20000 observations in the training set and 5000 in the test set.
The objective is to build various classification models, tune them, and find the best one that will help identify failures so that the generators could be repaired before failing/breaking to reduce the overall maintenance cost. The nature of predictions made by the classification model will translate as follows:
- True positives (TP) are failures correctly predicted by the model. These will result in repairing costs.
- False negatives (FN) are real failures where there is no detection by the model. These will result in replacement costs.
- False positives (FP) are detections where there is no failure. These will result in inspection costs.
It is given that the cost of repairing a generator is much less than the cost of replacing it, and the cost of inspection is less than the cost of repair.
“1” in the target variables should be considered as “failure” and “0” represents “No failure”.
Data Description¶
The data provided is a transformed version of the original data which was collected using sensors.
- Train.csv - To be used for training and tuning of models.
- Test.csv - To be used only for testing the performance of the final best model.
Both the datasets consist of 40 predictor variables and 1 target variable.
Install, import necessary libraries and set seed¶
Installing necessary libraries with required versions¶
Note:
- After running the above cell, kindly restart the runtime (for Google Colab) or notebook kernel (for Jupyter Notebook), and run all cells sequentially from the next cell.
- On executing the above line of code, you might see a warning regarding package dependencies. This error message can be ignored as the above code ensures that all necessary libraries and their dependencies are maintained to successfully execute the code in this notebook.
Importing necessary libraries with required versions¶
Setting seeds.¶
Loading the given Datasets¶
Loading Train dataset¶
Reading Train dataset into renewind_train_df DataFrame¶
Displaying first 5 records from the renewind_train_df DataFrame¶
Out[5]:
| V1 | V2 | V3 | V4 | V5 | V6 | V7 | V8 | V9 | V10 | ... | V32 | V33 | V34 | V35 | V36 | V37 | V38 | V39 | V40 | Target | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | -4.464606 | -4.679129 | 3.101546 | 0.506130 | -0.221083 | -2.032511 | -2.910870 | 0.050714 | -1.522351 | 3.761892 | ... | 3.059700 | -1.690440 | 2.846296 | 2.235198 | 6.667486 | 0.443809 | -2.369169 | 2.950578 | -3.480324 | 0 |
| 1 | 3.365912 | 3.653381 | 0.909671 | -1.367528 | 0.332016 | 2.358938 | 0.732600 | -4.332135 | 0.565695 | -0.101080 | ... | -1.795474 | 3.032780 | -2.467514 | 1.894599 | -2.297780 | -1.731048 | 5.908837 | -0.386345 | 0.616242 | 0 |
| 2 | -3.831843 | -5.824444 | 0.634031 | -2.418815 | -1.773827 | 1.016824 | -2.098941 | -3.173204 | -2.081860 | 5.392621 | ... | -0.257101 | 0.803550 | 4.086219 | 2.292138 | 5.360850 | 0.351993 | 2.940021 | 3.839160 | -4.309402 | 0 |
| 3 | 1.618098 | 1.888342 | 7.046143 | -1.147285 | 0.083080 | -1.529780 | 0.207309 | -2.493629 | 0.344926 | 2.118578 | ... | -3.584425 | -2.577474 | 1.363769 | 0.622714 | 5.550100 | -1.526796 | 0.138853 | 3.101430 | -1.277378 | 0 |
| 4 | -0.111440 | 3.872488 | -3.758361 | -2.982897 | 3.792714 | 0.544960 | 0.205433 | 4.848994 | -1.854920 | -6.220023 | ... | 8.265896 | 6.629213 | -10.068689 | 1.222987 | -3.229763 | 1.686909 | -2.163896 | -3.644622 | 6.510338 | 0 |
5 rows × 41 columns
Out[6]:
| V1 | V2 | V3 | V4 | V5 | V6 | V7 | V8 | V9 | V10 | ... | V32 | V33 | V34 | V35 | V36 | V37 | V38 | V39 | V40 | Target | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 19995 | -2.071318 | -1.088279 | -0.796174 | -3.011720 | -2.287540 | 2.807310 | 0.481428 | 0.105171 | -0.586599 | -2.899398 | ... | -8.273996 | 5.745013 | 0.589014 | -0.649988 | -3.043174 | 2.216461 | 0.608723 | 0.178193 | 2.927755 | 1 |
| 19996 | 2.890264 | 2.483069 | 5.643919 | 0.937053 | -1.380870 | 0.412051 | -1.593386 | -5.762498 | 2.150096 | 0.272302 | ... | -4.159092 | 1.181466 | -0.742412 | 5.368979 | -0.693028 | -1.668971 | 3.659954 | 0.819863 | -1.987265 | 0 |
| 19997 | -3.896979 | -3.942407 | -0.351364 | -2.417462 | 1.107546 | -1.527623 | -3.519882 | 2.054792 | -0.233996 | -0.357687 | ... | 7.112162 | 1.476080 | -3.953710 | 1.855555 | 5.029209 | 2.082588 | -6.409304 | 1.477138 | -0.874148 | 0 |
| 19998 | -3.187322 | -10.051662 | 5.695955 | -4.370053 | -5.354758 | -1.873044 | -3.947210 | 0.679420 | -2.389254 | 5.456756 | ... | 0.402812 | 3.163661 | 3.752095 | 8.529894 | 8.450626 | 0.203958 | -7.129918 | 4.249394 | -6.112267 | 0 |
| 19999 | -2.686903 | 1.961187 | 6.137088 | 2.600133 | 2.657241 | -4.290882 | -2.344267 | 0.974004 | -1.027462 | 0.497421 | ... | 6.620811 | -1.988786 | -1.348901 | 3.951801 | 5.449706 | -0.455411 | -2.202056 | 1.678229 | -1.974413 | 0 |
5 rows × 41 columns
Observation(s):
- The training dataset has been loaded successfully and the population of column values looks good as well.
Loading Test dataset¶
Reading Test dataset into renewind_test_df DataFrame¶
Displaying first 5 records from the renewind_test_df DataFrame¶
Out[8]:
| V1 | V2 | V3 | V4 | V5 | V6 | V7 | V8 | V9 | V10 | ... | V32 | V33 | V34 | V35 | V36 | V37 | V38 | V39 | V40 | Target | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | -0.613489 | -3.819640 | 2.202302 | 1.300420 | -1.184929 | -4.495964 | -1.835817 | 4.722989 | 1.206140 | -0.341909 | ... | 2.291204 | -5.411388 | 0.870073 | 0.574479 | 4.157191 | 1.428093 | -10.511342 | 0.454664 | -1.448363 | 0 |
| 1 | 0.389608 | -0.512341 | 0.527053 | -2.576776 | -1.016766 | 2.235112 | -0.441301 | -4.405744 | -0.332869 | 1.966794 | ... | -2.474936 | 2.493582 | 0.315165 | 2.059288 | 0.683859 | -0.485452 | 5.128350 | 1.720744 | -1.488235 | 0 |
| 2 | -0.874861 | -0.640632 | 4.084202 | -1.590454 | 0.525855 | -1.957592 | -0.695367 | 1.347309 | -1.732348 | 0.466500 | ... | -1.318888 | -2.997464 | 0.459664 | 0.619774 | 5.631504 | 1.323512 | -1.752154 | 1.808302 | 1.675748 | 0 |
| 3 | 0.238384 | 1.458607 | 4.014528 | 2.534478 | 1.196987 | -3.117330 | -0.924035 | 0.269493 | 1.322436 | 0.702345 | ... | 3.517918 | -3.074085 | -0.284220 | 0.954576 | 3.029331 | -1.367198 | -3.412140 | 0.906000 | -2.450889 | 0 |
| 4 | 5.828225 | 2.768260 | -1.234530 | 2.809264 | -1.641648 | -1.406698 | 0.568643 | 0.965043 | 1.918379 | -2.774855 | ... | 1.773841 | -1.501573 | -2.226702 | 4.776830 | -6.559698 | -0.805551 | -0.276007 | -3.858207 | -0.537694 | 0 |
5 rows × 41 columns
Out[9]:
| V1 | V2 | V3 | V4 | V5 | V6 | V7 | V8 | V9 | V10 | ... | V32 | V33 | V34 | V35 | V36 | V37 | V38 | V39 | V40 | Target | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 4995 | -5.120451 | 1.634804 | 1.251259 | 4.035944 | 3.291204 | -2.932230 | -1.328662 | 1.754066 | -2.984586 | 1.248633 | ... | 9.979118 | 0.063438 | 0.217281 | 3.036388 | 2.109323 | -0.557433 | 1.938718 | 0.512674 | -2.694194 | 0 |
| 4996 | -5.172498 | 1.171653 | 1.579105 | 1.219922 | 2.529627 | -0.668648 | -2.618321 | -2.000545 | 0.633791 | -0.578938 | ... | 4.423900 | 2.603811 | -2.152170 | 0.917401 | 2.156586 | 0.466963 | 0.470120 | 2.196756 | -2.376515 | 0 |
| 4997 | -1.114136 | -0.403576 | -1.764875 | -5.879475 | 3.571558 | 3.710802 | -2.482952 | -0.307614 | -0.921945 | -2.999141 | ... | 3.791778 | 7.481506 | -10.061396 | -0.387166 | 1.848509 | 1.818248 | -1.245633 | -1.260876 | 7.474682 | 0 |
| 4998 | -1.703241 | 0.614650 | 6.220503 | -0.104132 | 0.955916 | -3.278706 | -1.633855 | -0.103936 | 1.388152 | -1.065622 | ... | -4.100352 | -5.949325 | 0.550372 | -1.573640 | 6.823936 | 2.139307 | -4.036164 | 3.436051 | 0.579249 | 0 |
| 4999 | -0.603701 | 0.959550 | -0.720995 | 8.229574 | -1.815610 | -2.275547 | -2.574524 | -1.041479 | 4.129645 | -2.731288 | ... | 2.369776 | -1.062408 | 0.790772 | 4.951955 | -7.440825 | -0.069506 | -0.918083 | -2.291154 | -5.362891 | 0 |
5 rows × 41 columns
Observation(s):
- The test dataset has been loaded successfully and the population of column values looks good as well.
Data Overview¶
Overview of Train Dataset¶
Checking dataset shape¶
Out[10]:
(20000, 41)
Observation(s):
- The training dataset has 20,000 rows (records) and 41 columns.
Checking if any duplicate records exist¶
Out[11]:
0
Observation(s):
- The training dataset has no duplicate records.
Checking dataset structure/information¶
<class 'pandas.core.frame.DataFrame'> RangeIndex: 20000 entries, 0 to 19999 Data columns (total 41 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 V1 19982 non-null float64 1 V2 19982 non-null float64 2 V3 20000 non-null float64 3 V4 20000 non-null float64 4 V5 20000 non-null float64 5 V6 20000 non-null float64 6 V7 20000 non-null float64 7 V8 20000 non-null float64 8 V9 20000 non-null float64 9 V10 20000 non-null float64 10 V11 20000 non-null float64 11 V12 20000 non-null float64 12 V13 20000 non-null float64 13 V14 20000 non-null float64 14 V15 20000 non-null float64 15 V16 20000 non-null float64 16 V17 20000 non-null float64 17 V18 20000 non-null float64 18 V19 20000 non-null float64 19 V20 20000 non-null float64 20 V21 20000 non-null float64 21 V22 20000 non-null float64 22 V23 20000 non-null float64 23 V24 20000 non-null float64 24 V25 20000 non-null float64 25 V26 20000 non-null float64 26 V27 20000 non-null float64 27 V28 20000 non-null float64 28 V29 20000 non-null float64 29 V30 20000 non-null float64 30 V31 20000 non-null float64 31 V32 20000 non-null float64 32 V33 20000 non-null float64 33 V34 20000 non-null float64 34 V35 20000 non-null float64 35 V36 20000 non-null float64 36 V37 20000 non-null float64 37 V38 20000 non-null float64 38 V39 20000 non-null float64 39 V40 20000 non-null float64 40 Target 20000 non-null int64 dtypes: float64(40), int64(1) memory usage: 6.3 MB
Observation(s):
- The training dataset has 41 columns - 40 independent variables (V0 through V40) and 1 target variable (Target).
- Independent variables V1 and V2 each have about 18 missing or null values and will be imputed during data preprocessing.
Viewing Statistical summary¶
Out[13]:
| count | mean | std | min | 25% | 50% | 75% | max | |
|---|---|---|---|---|---|---|---|---|
| V1 | 19982.0 | -0.271996 | 3.441625 | -11.876451 | -2.737146 | -0.747917 | 1.840112 | 15.493002 |
| V2 | 19982.0 | 0.440430 | 3.150784 | -12.319951 | -1.640674 | 0.471536 | 2.543967 | 13.089269 |
| V3 | 20000.0 | 2.484699 | 3.388963 | -10.708139 | 0.206860 | 2.255786 | 4.566165 | 17.090919 |
| V4 | 20000.0 | -0.083152 | 3.431595 | -15.082052 | -2.347660 | -0.135241 | 2.130615 | 13.236381 |
| V5 | 20000.0 | -0.053752 | 2.104801 | -8.603361 | -1.535607 | -0.101952 | 1.340480 | 8.133797 |
| V6 | 20000.0 | -0.995443 | 2.040970 | -10.227147 | -2.347238 | -1.000515 | 0.380330 | 6.975847 |
| V7 | 20000.0 | -0.879325 | 1.761626 | -7.949681 | -2.030926 | -0.917179 | 0.223695 | 8.006091 |
| V8 | 20000.0 | -0.548195 | 3.295756 | -15.657561 | -2.642665 | -0.389085 | 1.722965 | 11.679495 |
| V9 | 20000.0 | -0.016808 | 2.160568 | -8.596313 | -1.494973 | -0.067597 | 1.409203 | 8.137580 |
| V10 | 20000.0 | -0.012998 | 2.193201 | -9.853957 | -1.411212 | 0.100973 | 1.477045 | 8.108472 |
| V11 | 20000.0 | -1.895393 | 3.124322 | -14.832058 | -3.922404 | -1.921237 | 0.118906 | 11.826433 |
| V12 | 20000.0 | 1.604825 | 2.930454 | -12.948007 | -0.396514 | 1.507841 | 3.571454 | 15.080698 |
| V13 | 20000.0 | 1.580486 | 2.874658 | -13.228247 | -0.223545 | 1.637185 | 3.459886 | 15.419616 |
| V14 | 20000.0 | -0.950632 | 1.789651 | -7.738593 | -2.170741 | -0.957163 | 0.270677 | 5.670664 |
| V15 | 20000.0 | -2.414993 | 3.354974 | -16.416606 | -4.415322 | -2.382617 | -0.359052 | 12.246455 |
| V16 | 20000.0 | -2.925225 | 4.221717 | -20.374158 | -5.634240 | -2.682705 | -0.095046 | 13.583212 |
| V17 | 20000.0 | -0.134261 | 3.345462 | -14.091184 | -2.215611 | -0.014580 | 2.068751 | 16.756432 |
| V18 | 20000.0 | 1.189347 | 2.592276 | -11.643994 | -0.403917 | 0.883398 | 2.571770 | 13.179863 |
| V19 | 20000.0 | 1.181808 | 3.396925 | -13.491784 | -1.050168 | 1.279061 | 3.493299 | 13.237742 |
| V20 | 20000.0 | 0.023608 | 3.669477 | -13.922659 | -2.432953 | 0.033415 | 2.512372 | 16.052339 |
| V21 | 20000.0 | -3.611252 | 3.567690 | -17.956231 | -5.930360 | -3.532888 | -1.265884 | 13.840473 |
| V22 | 20000.0 | 0.951835 | 1.651547 | -10.122095 | -0.118127 | 0.974687 | 2.025594 | 7.409856 |
| V23 | 20000.0 | -0.366116 | 4.031860 | -14.866128 | -3.098756 | -0.262093 | 2.451750 | 14.458734 |
| V24 | 20000.0 | 1.134389 | 3.912069 | -16.387147 | -1.468062 | 0.969048 | 3.545975 | 17.163291 |
| V25 | 20000.0 | -0.002186 | 2.016740 | -8.228266 | -1.365178 | 0.025050 | 1.397112 | 8.223389 |
| V26 | 20000.0 | 1.873785 | 3.435137 | -11.834271 | -0.337863 | 1.950531 | 4.130037 | 16.836410 |
| V27 | 20000.0 | -0.612413 | 4.368847 | -14.904939 | -3.652323 | -0.884894 | 2.189177 | 17.560404 |
| V28 | 20000.0 | -0.883218 | 1.917713 | -9.269489 | -2.171218 | -0.891073 | 0.375884 | 6.527643 |
| V29 | 20000.0 | -0.985625 | 2.684365 | -12.579469 | -2.787443 | -1.176181 | 0.629773 | 10.722055 |
| V30 | 20000.0 | -0.015534 | 3.005258 | -14.796047 | -1.867114 | 0.184346 | 2.036229 | 12.505812 |
| V31 | 20000.0 | 0.486842 | 3.461384 | -13.722760 | -1.817772 | 0.490304 | 2.730688 | 17.255090 |
| V32 | 20000.0 | 0.303799 | 5.500400 | -19.876502 | -3.420469 | 0.052073 | 3.761722 | 23.633187 |
| V33 | 20000.0 | 0.049825 | 3.575285 | -16.898353 | -2.242857 | -0.066249 | 2.255134 | 16.692486 |
| V34 | 20000.0 | -0.462702 | 3.183841 | -17.985094 | -2.136984 | -0.255008 | 1.436935 | 14.358213 |
| V35 | 20000.0 | 2.229620 | 2.937102 | -15.349803 | 0.336191 | 2.098633 | 4.064358 | 15.291065 |
| V36 | 20000.0 | 1.514809 | 3.800860 | -14.833178 | -0.943809 | 1.566526 | 3.983939 | 19.329576 |
| V37 | 20000.0 | 0.011316 | 1.788165 | -5.478350 | -1.255819 | -0.128435 | 1.175533 | 7.467006 |
| V38 | 20000.0 | -0.344025 | 3.948147 | -17.375002 | -2.987638 | -0.316849 | 2.279399 | 15.289923 |
| V39 | 20000.0 | 0.890653 | 1.753054 | -6.438880 | -0.272250 | 0.919261 | 2.057540 | 7.759877 |
| V40 | 20000.0 | -0.875630 | 3.012155 | -11.023935 | -2.940193 | -0.920806 | 1.119897 | 10.654265 |
| Target | 20000.0 | 0.055500 | 0.228959 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 1.000000 |
Observation(s):
- As noted earlier:
- 20,000 records exist in the training dataset.
- Missing or null values will need to be imputed for independent variables V1 and V2
- By looking at the statistical summary (especially mean and standard deviation), it's obvious that the dataset has already been standardized; further scaling or standardization is not required.
- The target variable has a binary value of either 0 or 1. No other values were found in the training dataset.
Checking proportion value of "Target" variable¶
Out[14]:
| proportion | |
|---|---|
| Target | |
| 0 | 94.45 |
| 1 | 5.55 |
Observation(s):
- The training dataset contains about 94.5% of records from properly working sensors, which have no faultiness.
- The remaining 5.5% of records are from faulty sensors that do not provide accurate data as per calibration.
Looking at unique value of features¶
Out[15]:
| 0 | |
|---|---|
| V1 | 19982 |
| V2 | 19982 |
| V3 | 20000 |
| V4 | 20000 |
| V5 | 20000 |
| V6 | 20000 |
| V7 | 20000 |
| V8 | 20000 |
| V9 | 20000 |
| V10 | 20000 |
| V11 | 20000 |
| V12 | 20000 |
| V13 | 20000 |
| V14 | 20000 |
| V15 | 20000 |
| V16 | 20000 |
| V17 | 20000 |
| V18 | 20000 |
| V19 | 20000 |
| V20 | 20000 |
| V21 | 20000 |
| V22 | 20000 |
| V23 | 20000 |
| V24 | 20000 |
| V25 | 20000 |
| V26 | 20000 |
| V27 | 20000 |
| V28 | 20000 |
| V29 | 20000 |
| V30 | 20000 |
| V31 | 20000 |
| V32 | 20000 |
| V33 | 20000 |
| V34 | 20000 |
| V35 | 20000 |
| V36 | 20000 |
| V37 | 20000 |
| V38 | 20000 |
| V39 | 20000 |
| V40 | 20000 |
| Target | 2 |
Observation(s):
- The independent variables in the training dataset are unique identifiers for each record.
- As noted earlier, independent variables V1 and V2 have 18 missing values each.
- The target variable has two unique values i.e., 0 or 1.
Checking for missing or null values.¶
Out[16]:
| V1 | V2 | V3 | V4 | V5 | V6 | V7 | V8 | V9 | V10 | ... | V32 | V33 | V34 | V35 | V36 | V37 | V38 | V39 | V40 | Target | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 89 | NaN | -3.961403 | 2.787804 | -4.712526 | -3.007329 | -1.541245 | -0.881148 | 1.476656 | 0.574700 | -1.100884 | ... | -8.326069 | -5.140552 | 1.121314 | -0.305907 | 5.315007 | 3.750044 | -5.631174 | 2.372485 | 2.195956 | 0 |
| 5941 | NaN | 1.008391 | 1.227702 | 5.397082 | 0.064230 | -2.706919 | -2.028368 | 0.534046 | 3.006797 | -2.362238 | ... | 1.869502 | -3.115298 | -0.550197 | 1.713781 | -2.256960 | 0.410992 | -3.434400 | -1.299388 | -1.768734 | 0 |
| 6317 | NaN | -5.205346 | 1.997652 | -3.707913 | -1.042200 | -1.593126 | -2.653309 | 0.852280 | -1.310489 | 2.406924 | ... | 3.074149 | -0.067649 | -0.277521 | 3.196840 | 7.016205 | 1.302334 | -4.580096 | 2.956254 | -2.363150 | 0 |
| 6464 | NaN | 2.146202 | 5.004415 | 4.192063 | 1.427887 | -6.438263 | -0.931339 | 3.794120 | -0.683032 | -0.738941 | ... | 5.231472 | -5.113312 | 1.745687 | 2.587189 | 3.990777 | 0.610716 | -4.273457 | 1.864568 | -3.599079 | 0 |
| 7073 | NaN | 2.534010 | 2.762821 | -1.673718 | -1.942214 | -0.029961 | 0.911323 | -3.199743 | 2.948610 | -0.413229 | ... | -4.887077 | -2.611526 | -1.500807 | 2.036186 | -0.828979 | -1.369591 | 0.572366 | -0.132183 | -0.322007 | 0 |
| 8431 | NaN | -1.398710 | -2.008106 | -1.750341 | 0.932279 | -1.290327 | -0.270476 | 4.458834 | -2.776270 | -1.211766 | ... | 4.560244 | -0.420834 | -2.037313 | 1.109793 | 1.520594 | 2.113872 | -2.252571 | -0.939249 | 2.542411 | 0 |
| 8439 | NaN | -3.840585 | 0.197220 | 4.147789 | 1.151400 | -0.993298 | -4.732363 | 0.558966 | -0.926683 | 0.457914 | ... | 6.818725 | 3.451213 | 0.241818 | 3.215765 | 1.203210 | 1.274857 | -1.921229 | 0.578890 | -2.837521 | 0 |
| 11156 | NaN | -0.666978 | 3.715829 | 4.934000 | 1.667596 | -4.356097 | -2.823137 | 0.373175 | -0.709951 | 2.177428 | ... | 6.663446 | -2.897697 | 3.068461 | 2.486862 | 4.808548 | 0.069305 | -1.215784 | 3.013674 | -5.972586 | 0 |
| 11287 | NaN | -2.561519 | -0.180836 | -7.194814 | -1.043832 | 1.384845 | 1.306093 | 1.559192 | -2.992173 | 1.274543 | ... | -2.531655 | 0.560392 | -1.153884 | -0.019205 | 4.065248 | 0.978880 | -0.571288 | 0.630374 | 3.919467 | 0 |
| 11456 | NaN | 1.299595 | 4.382858 | 1.583219 | -0.076564 | 0.658770 | -1.638530 | -4.814763 | -0.914819 | 2.811808 | ... | 1.772287 | 5.755242 | 1.203739 | 5.663939 | 0.413630 | -2.643934 | 5.529745 | 2.104536 | -4.945350 | 0 |
| 12221 | NaN | -2.326319 | -0.051978 | 0.615063 | -0.895755 | -2.437003 | 0.349826 | 2.092611 | -2.933523 | 2.291272 | ... | 0.134995 | -5.183424 | 5.251667 | 0.716371 | 3.210930 | 1.641985 | 1.543559 | 1.805163 | -2.039510 | 0 |
| 12447 | NaN | 0.752613 | -0.271099 | 1.301204 | 2.038697 | -1.485203 | -0.411939 | 0.980629 | 0.810336 | -0.065120 | ... | 4.410397 | -2.208567 | -1.358706 | -1.725697 | 1.679060 | -0.208564 | -2.335547 | 0.112248 | -0.542931 | 0 |
| 13086 | NaN | 2.056243 | 3.330642 | 2.741497 | 2.783166 | -0.444191 | -2.015376 | -0.887154 | -1.110920 | 0.025289 | ... | 5.112126 | 4.675408 | -1.709632 | 2.429762 | 0.996644 | -1.190509 | 1.207054 | 0.511023 | -0.884200 | 0 |
| 13411 | NaN | 2.704511 | 4.587169 | 1.867930 | 2.050133 | -0.925076 | -1.669496 | -1.653803 | -0.243383 | -0.317316 | ... | 2.527207 | 3.625279 | -1.200200 | 2.328028 | 1.666937 | -0.943228 | 0.946846 | 1.655145 | -1.665439 | 0 |
| 14202 | NaN | 7.038653 | 2.144536 | -3.201788 | 4.112972 | 3.375972 | -1.337179 | -4.546371 | 1.941427 | -5.466593 | ... | 0.157778 | 9.768106 | -10.258190 | 0.513864 | -1.974958 | -0.029436 | 3.127486 | 0.009482 | 4.538125 | 0 |
| 15520 | NaN | 1.382556 | 3.236896 | -3.818363 | -1.917264 | 0.437686 | 1.347540 | -2.036067 | 1.155712 | 0.306502 | ... | -5.414599 | -0.896510 | -1.057864 | 1.417365 | 1.161990 | -1.147123 | -0.048258 | 0.604532 | 0.814557 | 0 |
| 16576 | NaN | 3.933815 | -0.761930 | 2.651889 | 1.753614 | -0.554092 | 1.829107 | -0.105409 | -3.737081 | 1.036776 | ... | 3.486408 | 1.028094 | 2.845747 | 1.744060 | -1.999615 | -0.783041 | 8.698449 | 0.352489 | -2.005397 | 0 |
| 18104 | NaN | 1.492173 | 2.659206 | 0.222784 | -0.303648 | -1.347322 | 0.044309 | -0.159095 | 1.108116 | -0.572670 | ... | -1.007343 | -2.229579 | -0.870845 | 1.299595 | 0.667952 | -0.503349 | -1.485419 | -0.153722 | 0.156501 | 0 |
18 rows × 41 columns
Observation(s):
- The 18 missing values (i.e. NaN) shown above will be imputed during data preprocessing.
Overview of Test Dataset¶
Checking dataset shape¶
Out[17]:
(5000, 41)
Observation(s):
- The test dataset has 5,000 rows (records) and 41 columns.
Checking if any duplicate records exist¶
Out[18]:
0
Observation(s):
- The test dataset has no duplicate records.
Checking dataset structure/information¶
<class 'pandas.core.frame.DataFrame'> RangeIndex: 5000 entries, 0 to 4999 Data columns (total 41 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 V1 4995 non-null float64 1 V2 4994 non-null float64 2 V3 5000 non-null float64 3 V4 5000 non-null float64 4 V5 5000 non-null float64 5 V6 5000 non-null float64 6 V7 5000 non-null float64 7 V8 5000 non-null float64 8 V9 5000 non-null float64 9 V10 5000 non-null float64 10 V11 5000 non-null float64 11 V12 5000 non-null float64 12 V13 5000 non-null float64 13 V14 5000 non-null float64 14 V15 5000 non-null float64 15 V16 5000 non-null float64 16 V17 5000 non-null float64 17 V18 5000 non-null float64 18 V19 5000 non-null float64 19 V20 5000 non-null float64 20 V21 5000 non-null float64 21 V22 5000 non-null float64 22 V23 5000 non-null float64 23 V24 5000 non-null float64 24 V25 5000 non-null float64 25 V26 5000 non-null float64 26 V27 5000 non-null float64 27 V28 5000 non-null float64 28 V29 5000 non-null float64 29 V30 5000 non-null float64 30 V31 5000 non-null float64 31 V32 5000 non-null float64 32 V33 5000 non-null float64 33 V34 5000 non-null float64 34 V35 5000 non-null float64 35 V36 5000 non-null float64 36 V37 5000 non-null float64 37 V38 5000 non-null float64 38 V39 5000 non-null float64 39 V40 5000 non-null float64 40 Target 5000 non-null int64 dtypes: float64(40), int64(1) memory usage: 1.6 MB
Observation(s):
- The test dataset has 41 columns - 40 independent variables (V0 through V40) and 1 target variable (Target).
- Independent variables V1 and V2 each have about 5 and 6 missing or null values, respectively; missing values will be imputed during data preprocessing.
Viewing Statistical summary of dataset¶
Out[20]:
| count | mean | std | min | 25% | 50% | 75% | max | |
|---|---|---|---|---|---|---|---|---|
| V1 | 4995.0 | -0.277622 | 3.466280 | -12.381696 | -2.743691 | -0.764767 | 1.831313 | 13.504352 |
| V2 | 4994.0 | 0.397928 | 3.139562 | -10.716179 | -1.649211 | 0.427369 | 2.444486 | 14.079073 |
| V3 | 5000.0 | 2.551787 | 3.326607 | -9.237940 | 0.314931 | 2.260428 | 4.587000 | 15.314503 |
| V4 | 5000.0 | -0.048943 | 3.413937 | -14.682446 | -2.292694 | -0.145753 | 2.166468 | 12.140157 |
| V5 | 5000.0 | -0.080120 | 2.110870 | -7.711569 | -1.615238 | -0.131890 | 1.341197 | 7.672835 |
| V6 | 5000.0 | -1.042138 | 2.005444 | -8.924196 | -2.368853 | -1.048571 | 0.307555 | 5.067685 |
| V7 | 5000.0 | -0.907922 | 1.769017 | -8.124230 | -2.054259 | -0.939695 | 0.212228 | 7.616182 |
| V8 | 5000.0 | -0.574592 | 3.331911 | -12.252731 | -2.642088 | -0.357943 | 1.712896 | 10.414722 |
| V9 | 5000.0 | 0.030121 | 2.174139 | -6.785495 | -1.455712 | -0.079891 | 1.449548 | 8.850720 |
| V10 | 5000.0 | 0.018524 | 2.145437 | -8.170956 | -1.353320 | 0.166292 | 1.511248 | 6.598728 |
| V11 | 5000.0 | -2.008615 | 3.112220 | -13.151753 | -4.050432 | -2.043122 | 0.044069 | 9.956400 |
| V12 | 5000.0 | 1.576413 | 2.907401 | -8.164048 | -0.449674 | 1.488253 | 3.562626 | 12.983644 |
| V13 | 5000.0 | 1.622456 | 2.882892 | -11.548209 | -0.126012 | 1.718649 | 3.464604 | 12.620041 |
| V14 | 5000.0 | -0.921097 | 1.803470 | -7.813929 | -2.110952 | -0.896011 | 0.272324 | 5.734112 |
| V15 | 5000.0 | -2.452174 | 3.387041 | -15.285768 | -4.479072 | -2.417131 | -0.432943 | 11.673420 |
| V16 | 5000.0 | -3.018503 | 4.264407 | -20.985779 | -5.648343 | -2.773763 | -0.178105 | 13.975843 |
| V17 | 5000.0 | -0.103721 | 3.336513 | -13.418281 | -2.227683 | 0.047462 | 2.111907 | 19.776592 |
| V18 | 5000.0 | 1.195606 | 2.586403 | -12.214016 | -0.408850 | 0.881395 | 2.604014 | 13.642235 |
| V19 | 5000.0 | 1.210490 | 3.384662 | -14.169635 | -1.026394 | 1.295864 | 3.526278 | 12.427997 |
| V20 | 5000.0 | 0.138429 | 3.657171 | -13.719620 | -2.325454 | 0.193386 | 2.539550 | 13.870565 |
| V21 | 5000.0 | -3.664398 | 3.577841 | -16.340707 | -5.944369 | -3.662870 | -1.329645 | 11.046925 |
| V22 | 5000.0 | 0.961960 | 1.640414 | -6.740239 | -0.047728 | 0.986020 | 2.029321 | 7.505291 |
| V23 | 5000.0 | -0.422182 | 4.056714 | -14.422274 | -3.162690 | -0.279222 | 2.425911 | 13.180887 |
| V24 | 5000.0 | 1.088841 | 3.968207 | -12.315545 | -1.623203 | 0.912815 | 3.537195 | 17.806035 |
| V25 | 5000.0 | 0.061235 | 2.010227 | -6.770139 | -1.298377 | 0.076703 | 1.428491 | 6.556937 |
| V26 | 5000.0 | 1.847261 | 3.400330 | -11.414019 | -0.242470 | 1.917032 | 4.156106 | 17.528193 |
| V27 | 5000.0 | -0.552397 | 4.402947 | -13.177038 | -3.662591 | -0.871982 | 2.247257 | 17.290161 |
| V28 | 5000.0 | -0.867678 | 1.926181 | -7.933388 | -2.159811 | -0.930695 | 0.420587 | 7.415659 |
| V29 | 5000.0 | -1.095805 | 2.655454 | -9.987800 | -2.861373 | -1.340547 | 0.521843 | 14.039466 |
| V30 | 5000.0 | -0.118699 | 3.023292 | -12.438434 | -1.996743 | 0.112463 | 1.946450 | 10.314976 |
| V31 | 5000.0 | 0.468810 | 3.446324 | -11.263271 | -1.822421 | 0.485742 | 2.779008 | 12.558928 |
| V32 | 5000.0 | 0.232567 | 5.585628 | -17.244168 | -3.556267 | -0.076694 | 3.751857 | 26.539391 |
| V33 | 5000.0 | -0.080115 | 3.538624 | -14.903781 | -2.348121 | -0.159713 | 2.099160 | 13.323517 |
| V34 | 5000.0 | -0.392663 | 3.166101 | -14.699725 | -2.009604 | -0.171745 | 1.465402 | 12.146302 |
| V35 | 5000.0 | 2.211205 | 2.948426 | -12.260591 | 0.321818 | 2.111750 | 4.031639 | 13.489237 |
| V36 | 5000.0 | 1.594845 | 3.774970 | -12.735567 | -0.866066 | 1.702964 | 4.104409 | 17.116122 |
| V37 | 5000.0 | 0.022931 | 1.785320 | -5.079070 | -1.240526 | -0.110415 | 1.237522 | 6.809938 |
| V38 | 5000.0 | -0.405659 | 3.968936 | -15.334533 | -2.984480 | -0.381162 | 2.287998 | 13.064950 |
| V39 | 5000.0 | 0.938800 | 1.716502 | -5.451050 | -0.208024 | 0.959152 | 2.130769 | 7.182237 |
| V40 | 5000.0 | -0.932406 | 2.978193 | -10.076234 | -2.986587 | -1.002764 | 1.079738 | 8.698460 |
| Target | 5000.0 | 0.056400 | 0.230716 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 1.000000 |
Observation(s):
- As noted earlier:
- 5,000 records exist in the training dataset.
- Missing or null values will need to be imputed for independent variables V1 and V2
- By looking at the statistical summary (especially mean and standard deviation), it's obvious that the dataset has already been standardized; further scaling or standardization is not required.
- The target variable has a binary value of either 0 or 1. No other values were found in the training dataset.
Checking proportion value of "Target" variable¶
Out[21]:
| proportion | |
|---|---|
| Target | |
| 0 | 94.36 |
| 1 | 5.64 |
Observation(s):
- The test dataset contains about 94.3% of records from properly working sensors, which have no faultiness.
- The remaining 5.6% of records are from faulty sensors that do not provide accurate data as per calibration.
Looking at unique value of features¶
Out[22]:
| 0 | |
|---|---|
| V1 | 4995 |
| V2 | 4994 |
| V3 | 5000 |
| V4 | 5000 |
| V5 | 5000 |
| V6 | 5000 |
| V7 | 5000 |
| V8 | 5000 |
| V9 | 5000 |
| V10 | 5000 |
| V11 | 5000 |
| V12 | 5000 |
| V13 | 5000 |
| V14 | 5000 |
| V15 | 5000 |
| V16 | 5000 |
| V17 | 5000 |
| V18 | 5000 |
| V19 | 5000 |
| V20 | 5000 |
| V21 | 5000 |
| V22 | 5000 |
| V23 | 5000 |
| V24 | 5000 |
| V25 | 5000 |
| V26 | 5000 |
| V27 | 5000 |
| V28 | 5000 |
| V29 | 5000 |
| V30 | 5000 |
| V31 | 5000 |
| V32 | 5000 |
| V33 | 5000 |
| V34 | 5000 |
| V35 | 5000 |
| V36 | 5000 |
| V37 | 5000 |
| V38 | 5000 |
| V39 | 5000 |
| V40 | 5000 |
| Target | 2 |
Observation(s):
- The independent variables in the training dataset are unique identifiers for each record.
- As noted earlier, independent variables V1 and V2 have 5 and 6 missing values respectively.
- The target variable has two unique values i.e., 0 or 1.
Checking for missing or null values.¶
Out[23]:
| V1 | V2 | V3 | V4 | V5 | V6 | V7 | V8 | V9 | V10 | ... | V32 | V33 | V34 | V35 | V36 | V37 | V38 | V39 | V40 | Target | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 709 | 3.171300 | NaN | -0.899604 | -7.687193 | -1.844379 | 2.229502 | 0.649609 | 0.680742 | -0.079613 | -3.926869 | ... | -8.569958 | 1.198974 | -3.747194 | -0.834087 | 0.364598 | 3.687177 | -1.450631 | -0.012682 | 6.569833 | 0 |
| 1777 | 1.255877 | NaN | 1.123121 | 0.347719 | -0.199314 | 0.542522 | -0.904536 | -2.398356 | 0.228689 | 0.245489 | ... | 0.851358 | 1.657839 | -1.410919 | 3.587088 | -1.116910 | -0.865736 | 2.766820 | -0.368560 | -0.864084 | 0 |
| 1869 | -1.272832 | NaN | 4.426359 | -3.013970 | -1.294693 | -0.883173 | -1.731633 | 0.098774 | -0.991360 | 2.495756 | ... | -0.396983 | 1.190134 | 0.629071 | 2.411258 | 6.166668 | -0.140616 | -4.208798 | 2.623088 | -1.368893 | 0 |
| 2741 | -2.938927 | NaN | 2.913242 | 1.431121 | 4.003345 | -4.743048 | -2.450111 | 3.795883 | -0.339877 | -2.897700 | ... | 12.077656 | 0.671770 | -6.354040 | 3.887011 | 3.420416 | 0.506994 | -5.913055 | 0.214129 | -0.931294 | 0 |
| 3266 | 5.896134 | NaN | 7.342806 | -1.052112 | -1.393952 | -0.410402 | 0.392391 | -6.141263 | 2.100145 | 1.897655 | ... | -5.652135 | -1.973205 | 0.275744 | 3.894656 | 2.108591 | -2.803778 | 3.971349 | 2.233942 | -2.542753 | 0 |
| 4186 | 5.034513 | NaN | 4.450708 | -6.077425 | 0.445417 | 2.491588 | 1.958447 | -5.311945 | -1.397204 | 2.755499 | ... | -5.133693 | 1.139969 | -1.421471 | 0.822456 | 4.099736 | -2.152178 | 7.063377 | 2.377923 | 1.906096 | 0 |
6 rows × 41 columns
Observation(s):
- The 6 missing values (i.e. NaN), for independent variable V2, shown above will be imputed during data preprocessing.
Standardization or scaling of training and test datasets¶
- As noted in the statistical summary sections for both the training and test datasets, it's obvious that the dataset has already been standardized.
- Therefore, normalization, scaling, or standardization is not required for the provided training and test datasets.
Exploratory Data Analysis¶
Univariate analysis¶
Functions needed to visualize data using univariate analysis¶
Visualize independent variables using histogram and boxplot - Training Dataset¶
Observation(s):
- Majority of the independent variables appear to be normally or symmetrically distributed..
- Majority of the independent variables have outliers; however, outlier treatment will not be performed for the following reasons:
- No data dictionary was provided, so it's hard to interpret what the values and their units of measure represent.
- No SME with business domain expertise available to provide support.
- If outlier treatment is performed despite the points noted above, it could potentially impact the integrity of the data.
Visualize target variable using barplot - Training Dataset¶
Observation(s):
- The bar chart shows that the distribution of target variable classes is imbalanced.
- Only about 5.5% of sensors are not working as expected and transmitting incorrect measures or data points.
Visualize independent variables using histogram and boxplot - Test Dataset¶
Observation(s):
- Majority of the independent variables appear to be normally or symmetrically distributed.
- Majority of the independent variables have outliers; however, outlier treatment will not be performed for the following reasons:
- No data dictionary was provided, so it's hard to interpret what the values and their units of measure represent.
- No SME with business domain expertise available to provide support.
- If outlier treatment is performed despite the points noted above, it could potentially impact the integrity of the data.
Visualize target variable using barplot - Test Dataset¶
Observation(s):
- The bar chart shows that the distribution of target variable classes is imbalanced.
- Only about 5.6% of sensors are not working as expected and transmitting incorrect measures or data points.
Bivariate Analysis¶
Heatmap for the training dataset¶
Observation(s):
- The independent variables have a mixed set of correlations to each other (both positive and negative correlations) - Independent variables V7 and V15 have a high positive correlation of about 0.87, followed by 1) V24 and V32 (0.83), 2) V8 and V16 (0.80).
- Independent variables V2 and V14 have a high negative correlation of about -0.85, followed by 1) V3 and V23 (-0.79), 2) V27 and V32 (-0.77).
Heatmap for the test dataset¶
Observation(s):
- The independent variables have a mixed set of correlations to each other (both positive and negative correlations)
- Independent variables V7 and V15 have a high positive correlation of about 0.87, followed by V16 and V21 (0.84).
- Independent variables V2 and V14 have a high negative correlation of about -0.85, followed by V3 and V23 (-0.78).
Data Preprocessing¶
Scaling datasets¶
- Considering that the train and test datasets contain ciphered data and no data dictionary was provided, I assumed that the data had already been normalized or standardized or scaled. Therefore, I skipped the standardization or scaling of the datasets.
- Moreover, the datasets cannot be standardized or scaled without understanding the data points and their units of measure. Otherwise, the performance of models will be impacted adversely.
Splitting Datasets¶
Split training dataset into two different datasets - 1) Training and 2) Validation¶
Create X_train and y_train datasets¶
Create X_train, y_train, X_val and y_val datasets from X and y¶
Validate if the split was successful by displaying first 5 records from X_train, X_val, y_train, y_val datasets¶
Out[34]:
| V1 | V2 | V3 | V4 | V5 | V6 | V7 | V8 | V9 | V10 | ... | V31 | V32 | V33 | V34 | V35 | V36 | V37 | V38 | V39 | V40 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 5949 | -3.867777 | 5.218493 | 1.169385 | 1.370053 | 4.773402 | 1.445327 | -1.113678 | -0.806844 | -2.819800 | -0.786280 | ... | 5.940367 | 11.807331 | 12.043824 | -6.638613 | 4.957755 | -2.344377 | -3.171544 | 4.308970 | -1.547411 | 0.087674 |
| 18977 | -0.242338 | -1.327562 | 4.228952 | -0.008461 | -1.846820 | -2.213711 | -0.585124 | -0.196052 | -0.265413 | 1.965249 | ... | 0.921772 | -1.381113 | -1.783854 | 2.959768 | 2.886630 | 3.381226 | -0.450530 | -1.730357 | 2.248676 | -3.376100 |
| 7985 | -5.432375 | -2.721531 | 0.214081 | -0.169307 | 1.678438 | -1.355190 | -3.224389 | -0.324686 | -1.128019 | 2.068739 | ... | 0.592385 | 8.357520 | 1.641170 | -1.231252 | 2.615718 | 4.808997 | 0.557285 | -1.231271 | 2.470466 | -3.865032 |
| 18818 | 3.623791 | 0.895273 | 4.102492 | -0.808108 | -2.182981 | 0.164355 | -0.833773 | -2.186793 | 0.137834 | 0.174275 | ... | 0.512026 | -1.438225 | 3.145337 | -1.529566 | 6.948283 | -1.199801 | -1.645405 | 1.101975 | -0.607037 | -0.916648 |
| 48 | 1.178000 | -0.015717 | 2.038847 | -0.324694 | 0.264110 | -1.824574 | -0.377089 | 1.826273 | -1.567831 | 0.644669 | ... | 2.920648 | 3.537616 | -0.927180 | -1.078343 | 3.266084 | 2.158036 | -0.426700 | -1.499870 | -0.394165 | 0.171686 |
5 rows × 40 columns
Out[35]:
| Target | |
|---|---|
| 5949 | 0 |
| 18977 | 0 |
| 7985 | 0 |
| 18818 | 0 |
| 48 | 0 |
Out[36]:
| V1 | V2 | V3 | V4 | V5 | V6 | V7 | V8 | V9 | V10 | ... | V31 | V32 | V33 | V34 | V35 | V36 | V37 | V38 | V39 | V40 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 10019 | 7.000994 | 2.600869 | 11.855825 | -3.121053 | -2.374331 | -1.826619 | 0.243395 | -4.320870 | 1.871625 | 1.733411 | ... | 5.933289 | -10.062595 | -4.133742 | 0.789327 | 3.559830 | 6.423058 | -2.084511 | -1.046006 | 3.275849 | -0.231192 |
| 12223 | 4.472676 | 2.176988 | 1.646611 | 3.472448 | -2.427811 | -0.441140 | -1.431770 | -2.993525 | 4.114965 | -1.858772 | ... | -3.687935 | 0.209139 | 1.886537 | -2.238449 | 6.375328 | -6.174799 | -2.062803 | -0.460730 | -2.259602 | -3.535283 |
| 1621 | -3.739724 | -0.985003 | -1.718680 | -2.337082 | 1.423353 | 2.021703 | -1.293042 | -2.348012 | -1.573117 | 1.092688 | ... | -1.256778 | 2.551527 | 3.891239 | -1.395520 | 0.402924 | 1.695628 | 0.789087 | 4.086245 | 1.723682 | -0.501791 |
| 17851 | -4.319057 | 1.619180 | 7.336608 | 4.607435 | 1.073154 | -3.782849 | -2.289456 | -1.747669 | 0.434157 | 1.506115 | ... | 1.779307 | 1.310150 | -2.387195 | 2.995175 | 2.719836 | 4.460380 | -0.823088 | -0.263092 | 3.335333 | -4.878334 |
| 9803 | 0.639677 | -1.732999 | -1.620421 | -0.330020 | -0.479976 | -1.149986 | 0.859776 | 4.005725 | 0.030204 | -1.344766 | ... | 0.102588 | -1.598347 | -3.790657 | 0.800650 | -3.205477 | 0.491296 | 1.722653 | -4.749710 | -0.978993 | 2.597085 |
5 rows × 40 columns
Out[37]:
| Target | |
|---|---|
| 10019 | 0 |
| 12223 | 0 |
| 1621 | 0 |
| 17851 | 0 |
| 9803 | 1 |
Observation(s):
- The original training dataset (renewind_train_df) has been split into training and validation datasets successfully.
- The split must be performed before imputing missing or null values to avoid data leakage.
Note
This step is necessary because the two datasets provided to solve the business problems were the training and test datasets.
The test dataset (also known as the held-out dataset) is used to evaluate the performance of the final model, which is chosen after building and tuning a different set of models.
Split independent and dependent variables from Test Dataset¶
Create X_test and y_test datasets¶
Check the shape of X_test and y_test datasets.¶
Out[39]:
(5000, 40)
Out[40]:
(5000,)
Validate if the split was successful by displaying first 5 records from X_test and y_test datasets.¶
Out[41]:
| V1 | V2 | V3 | V4 | V5 | V6 | V7 | V8 | V9 | V10 | ... | V31 | V32 | V33 | V34 | V35 | V36 | V37 | V38 | V39 | V40 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | -0.613489 | -3.819640 | 2.202302 | 1.300420 | -1.184929 | -4.495964 | -1.835817 | 4.722989 | 1.206140 | -0.341909 | ... | 1.586765 | 2.291204 | -5.411388 | 0.870073 | 0.574479 | 4.157191 | 1.428093 | -10.511342 | 0.454664 | -1.448363 |
| 1 | 0.389608 | -0.512341 | 0.527053 | -2.576776 | -1.016766 | 2.235112 | -0.441301 | -4.405744 | -0.332869 | 1.966794 | ... | -1.782158 | -2.474936 | 2.493582 | 0.315165 | 2.059288 | 0.683859 | -0.485452 | 5.128350 | 1.720744 | -1.488235 |
| 2 | -0.874861 | -0.640632 | 4.084202 | -1.590454 | 0.525855 | -1.957592 | -0.695367 | 1.347309 | -1.732348 | 0.466500 | ... | 2.907291 | -1.318888 | -2.997464 | 0.459664 | 0.619774 | 5.631504 | 1.323512 | -1.752154 | 1.808302 | 1.675748 |
| 3 | 0.238384 | 1.458607 | 4.014528 | 2.534478 | 1.196987 | -3.117330 | -0.924035 | 0.269493 | 1.322436 | 0.702345 | ... | 3.808667 | 3.517918 | -3.074085 | -0.284220 | 0.954576 | 3.029331 | -1.367198 | -3.412140 | 0.906000 | -2.450889 |
| 4 | 5.828225 | 2.768260 | -1.234530 | 2.809264 | -1.641648 | -1.406698 | 0.568643 | 0.965043 | 1.918379 | -2.774855 | ... | -3.591421 | 1.773841 | -1.501573 | -2.226702 | 4.776830 | -6.559698 | -0.805551 | -0.276007 | -3.858207 | -0.537694 |
5 rows × 40 columns
Out[42]:
| Target | |
|---|---|
| 0 | 0 |
| 1 | 0 |
| 2 | 0 |
| 3 | 0 |
| 4 | 0 |
Observation(s):
- The provided test dataset (renewind_test_df) has been split into X_test and y_test successfully.
Missing value treatment¶
Note:
Considering that the data appear to be symmetrical in nature and no dictionary was provided to better understand the data, Median value is used to impute missing or null values.
Impute Training dataset (X_train)¶
Display records with missing or null values¶
Out[43]:
| V1 | V2 | V3 | V4 | V5 | V6 | V7 | V8 | V9 | V10 | ... | V31 | V32 | V33 | V34 | V35 | V36 | V37 | V38 | V39 | V40 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 5941 | NaN | 1.008391 | 1.227702 | 5.397082 | 0.064230 | -2.706919 | -2.028368 | 0.534046 | 3.006797 | -2.362238 | ... | -2.748630 | 1.869502 | -3.115298 | -0.550197 | 1.713781 | -2.256960 | 0.410992 | -3.434400 | -1.299388 | -1.768734 |
| 8431 | NaN | -1.398710 | -2.008106 | -1.750341 | 0.932279 | -1.290327 | -0.270476 | 4.458834 | -2.776270 | -1.211766 | ... | 0.415482 | 4.560244 | -0.420834 | -2.037313 | 1.109793 | 1.520594 | 2.113872 | -2.252571 | -0.939249 | 2.542411 |
| 12339 | -1.663687 | NaN | -0.712286 | -4.346935 | 1.391670 | -0.093951 | -2.163175 | -0.380573 | 0.031191 | -0.658845 | ... | -1.121747 | 0.306588 | -2.690990 | -3.111879 | -1.596402 | 5.821108 | 3.462033 | -1.736752 | 2.291092 | 2.240769 |
| 15520 | NaN | 1.382556 | 3.236896 | -3.818363 | -1.917264 | 0.437686 | 1.347540 | -2.036067 | 1.155712 | 0.306502 | ... | 2.066104 | -5.414599 | -0.896510 | -1.057864 | 1.417365 | 1.161990 | -1.147123 | -0.048258 | 0.604532 | 0.814557 |
| 11456 | NaN | 1.299595 | 4.382858 | 1.583219 | -0.076564 | 0.658770 | -1.638530 | -4.814763 | -0.914819 | 2.811808 | ... | 0.724201 | 1.772287 | 5.755242 | 1.203739 | 5.663939 | 0.413630 | -2.643934 | 5.529745 | 2.104536 | -4.945350 |
| 6464 | NaN | 2.146202 | 5.004415 | 4.192063 | 1.427887 | -6.438263 | -0.931339 | 3.794120 | -0.683032 | -0.738941 | ... | 1.854951 | 5.231472 | -5.113312 | 1.745687 | 2.587189 | 3.990777 | 0.610716 | -4.273457 | 1.864568 | -3.599079 |
| 9848 | -1.763501 | NaN | 2.845012 | -2.753083 | -0.811848 | -0.101166 | -1.382141 | -1.105042 | -0.054339 | 0.159742 | ... | 0.264272 | -2.158880 | 1.859682 | -0.337278 | 1.509300 | 3.408411 | 0.922594 | -1.502959 | 2.514666 | -0.793574 |
| 8483 | -4.484232 | NaN | 1.200644 | -2.042064 | 2.779443 | -0.801748 | -5.403548 | -1.225314 | 1.485831 | -0.974382 | ... | 1.661795 | 9.467401 | 4.281421 | -7.588117 | 3.266825 | 5.232311 | 1.278590 | -5.370513 | 1.984130 | -1.643391 |
| 18343 | -2.377369 | NaN | -0.009173 | -1.471979 | 1.295482 | 0.724894 | -1.122797 | -3.190475 | 3.250575 | -4.861648 | ... | -5.729277 | -5.093149 | 0.439355 | -3.167241 | -2.713266 | -0.592845 | 3.229219 | 1.315635 | 2.282838 | 1.151589 |
| 11637 | -2.270541 | NaN | 1.710061 | 1.157522 | -0.355177 | -5.449480 | -0.786321 | 3.936176 | -1.576138 | 0.800881 | ... | -1.545940 | 2.651480 | -8.429332 | 3.511387 | 1.500102 | 5.552380 | 2.588580 | -3.453418 | 2.324339 | -2.760081 |
| 9362 | 2.662045 | NaN | 2.980068 | 4.430762 | -0.237769 | 0.671919 | 0.380068 | -7.646684 | 4.434754 | -0.746393 | ... | -4.606549 | -5.493590 | -1.104656 | 1.224987 | 0.975596 | -4.794411 | -2.269039 | 7.670648 | 0.824983 | -3.929104 |
| 7073 | NaN | 2.534010 | 2.762821 | -1.673718 | -1.942214 | -0.029961 | 0.911323 | -3.199743 | 2.948610 | -0.413229 | ... | -0.201166 | -4.887077 | -2.611526 | -1.500807 | 2.036186 | -0.828979 | -1.369591 | 0.572366 | -0.132183 | -0.322007 |
| 613 | -2.048681 | NaN | -1.623885 | -3.324224 | 0.152256 | 0.600157 | -1.812802 | 0.852194 | -1.522600 | 0.211071 | ... | -0.471473 | 3.264218 | 2.379064 | -2.457084 | 1.719365 | 2.537010 | 1.701780 | -1.434535 | 0.597365 | 0.739238 |
| 4653 | 5.465769 | NaN | 4.540947 | -2.916550 | 0.399752 | 2.798925 | 0.029477 | -7.334071 | 1.122874 | 1.695269 | ... | 3.652232 | -1.535753 | 4.596212 | -4.103525 | 4.295524 | 0.152672 | -3.726700 | 6.562692 | 0.706452 | -0.461696 |
| 13086 | NaN | 2.056243 | 3.330642 | 2.741497 | 2.783166 | -0.444191 | -2.015376 | -0.887154 | -1.110920 | 0.025289 | ... | 3.295126 | 5.112126 | 4.675408 | -1.709632 | 2.429762 | 0.996644 | -1.190509 | 1.207054 | 0.511023 | -0.884200 |
| 6810 | -2.631454 | NaN | 2.330188 | 1.090080 | 0.603973 | -1.139383 | -0.690121 | -1.358935 | 0.355568 | -1.189176 | ... | -1.960479 | -0.950215 | 0.209717 | 0.448728 | 1.046063 | 0.536937 | 0.763187 | 1.728621 | 1.885821 | -1.701774 |
| 8439 | NaN | -3.840585 | 0.197220 | 4.147789 | 1.151400 | -0.993298 | -4.732363 | 0.558966 | -0.926683 | 0.457914 | ... | -2.012722 | 6.818725 | 3.451213 | 0.241818 | 3.215765 | 1.203210 | 1.274857 | -1.921229 | 0.578890 | -2.837521 |
| 12447 | NaN | 0.752613 | -0.271099 | 1.301204 | 2.038697 | -1.485203 | -0.411939 | 0.980629 | 0.810336 | -0.065120 | ... | 2.091916 | 4.410397 | -2.208567 | -1.358706 | -1.725697 | 1.679060 | -0.208564 | -2.335547 | 0.112248 | -0.542931 |
| 18342 | -0.928572 | NaN | 2.375506 | -1.236914 | 3.228744 | -2.100088 | -2.189908 | 0.588644 | 1.955973 | -5.008491 | ... | 0.332569 | 1.613181 | -1.820569 | -6.664808 | -0.455080 | 3.054891 | 2.935276 | -3.791135 | 0.863011 | 3.335753 |
| 11156 | NaN | -0.666978 | 3.715829 | 4.934000 | 1.667596 | -4.356097 | -2.823137 | 0.373175 | -0.709951 | 2.177428 | ... | 0.464499 | 6.663446 | -2.897697 | 3.068461 | 2.486862 | 4.808548 | 0.069305 | -1.215784 | 3.013674 | -5.972586 |
| 2508 | -1.430888 | NaN | 0.659576 | -2.876402 | 1.150137 | -0.785760 | -1.560174 | 2.898635 | -2.346989 | -0.217607 | ... | 3.915422 | 6.279266 | 3.323914 | -4.047760 | 3.119220 | 3.336260 | 0.603524 | -3.781725 | -0.157478 | 1.503298 |
| 8894 | 3.263555 | NaN | 8.446574 | -3.253218 | -3.417978 | -2.995838 | -0.669271 | -0.161283 | -0.666870 | 3.133527 | ... | 4.554619 | -4.242730 | -3.122680 | 2.522415 | 5.283805 | 7.291310 | -0.867555 | -4.315230 | 3.124488 | -2.393239 |
| 7788 | -4.203459 | NaN | 2.953868 | 0.584466 | 4.103940 | -0.639211 | -2.810799 | -0.112492 | -1.362768 | -0.800101 | ... | 6.185166 | 12.522374 | 9.502488 | -7.152953 | 5.668769 | 1.249833 | -2.158520 | -0.954461 | -0.002385 | -1.546808 |
| 9425 | -2.354134 | NaN | 2.053893 | 0.811660 | 2.540366 | -0.924875 | -0.208380 | -0.562864 | -0.140210 | -2.146916 | ... | -0.041593 | -0.621103 | -0.896509 | -1.181480 | -1.236617 | 1.237120 | 1.228277 | 2.073727 | 1.223874 | 1.472175 |
| 11287 | NaN | -2.561519 | -0.180836 | -7.194814 | -1.043832 | 1.384845 | 1.306093 | 1.559192 | -2.992173 | 1.274543 | ... | 3.188166 | -2.531655 | 0.560392 | -1.153884 | -0.019205 | 4.065248 | 0.978880 | -0.571288 | 0.630374 | 3.919467 |
| 18907 | -0.119181 | NaN | 3.657612 | -1.231802 | 1.946873 | -0.119089 | 0.652414 | -1.490208 | -0.033631 | -2.556604 | ... | 0.960273 | -4.670353 | -0.593916 | -1.650592 | -1.405071 | 1.531267 | 1.079147 | 2.832949 | 1.450781 | 3.232659 |
| 12221 | NaN | -2.326319 | -0.051978 | 0.615063 | -0.895755 | -2.437003 | 0.349826 | 2.092611 | -2.933523 | 2.291272 | ... | -2.262264 | 0.134995 | -5.183424 | 5.251667 | 0.716371 | 3.210930 | 1.641985 | 1.543559 | 1.805163 | -2.039510 |
| 14202 | NaN | 7.038653 | 2.144536 | -3.201788 | 4.112972 | 3.375972 | -1.337179 | -4.546371 | 1.941427 | -5.466593 | ... | 1.920413 | 0.157778 | 9.768106 | -10.258190 | 0.513864 | -1.974958 | -0.029436 | 3.127486 | 0.009482 | 4.538125 |
| 89 | NaN | -3.961403 | 2.787804 | -4.712526 | -3.007329 | -1.541245 | -0.881148 | 1.476656 | 0.574700 | -1.100884 | ... | -1.875548 | -8.326069 | -5.140552 | 1.121314 | -0.305907 | 5.315007 | 3.750044 | -5.631174 | 2.372485 | 2.195956 |
| 18104 | NaN | 1.492173 | 2.659206 | 0.222784 | -0.303648 | -1.347322 | 0.044309 | -0.159095 | 1.108116 | -0.572670 | ... | 1.338870 | -1.007343 | -2.229579 | -0.870845 | 1.299595 | 0.667952 | -0.503349 | -1.485419 | -0.153722 | 0.156501 |
30 rows × 40 columns
Calculate median value of each independent variable in X_train¶
Validate if the missing or null values have been imputed¶
Out[45]:
| V1 | V2 | V3 | V4 | V5 | V6 | V7 | V8 | V9 | V10 | ... | V31 | V32 | V33 | V34 | V35 | V36 | V37 | V38 | V39 | V40 |
|---|
0 rows × 40 columns
Observation(s):
- All missing or null values from X_train dataset have been imputed using median value.
- No missing or null values exist in X_train.
Impute Validation dataset (X_val)¶
Display records with missing or null values¶
Out[46]:
| V1 | V2 | V3 | V4 | V5 | V6 | V7 | V8 | V9 | V10 | ... | V31 | V32 | V33 | V34 | V35 | V36 | V37 | V38 | V39 | V40 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 15913 | 0.768122 | NaN | 5.296110 | 0.043018 | -1.173729 | -2.248575 | 0.956395 | -0.089941 | -0.241678 | -1.061413 | ... | -0.803008 | -7.720265 | -4.518617 | 3.182253 | 0.453452 | 2.175494 | 1.261707 | 0.892630 | 2.026732 | 0.632903 |
| 6317 | NaN | -5.205346 | 1.997652 | -3.707913 | -1.042200 | -1.593126 | -2.653309 | 0.852280 | -1.310489 | 2.406924 | ... | 1.492476 | 3.074149 | -0.067649 | -0.277521 | 3.196840 | 7.016205 | 1.302334 | -4.580096 | 2.956254 | -2.363150 |
| 16576 | NaN | 3.933815 | -0.761930 | 2.651889 | 1.753614 | -0.554092 | 1.829107 | -0.105409 | -3.737081 | 1.036776 | ... | -1.477069 | 3.486408 | 1.028094 | 2.845747 | 1.744060 | -1.999615 | -0.783041 | 8.698449 | 0.352489 | -2.005397 |
| 13411 | NaN | 2.704511 | 4.587169 | 1.867930 | 2.050133 | -0.925076 | -1.669496 | -1.653803 | -0.243383 | -0.317316 | ... | 2.580065 | 2.527207 | 3.625279 | -1.200200 | 2.328028 | 1.666937 | -0.943228 | 0.946846 | 1.655145 | -1.665439 |
| 2236 | -3.760658 | NaN | 0.194954 | -1.637958 | 1.261479 | -1.573947 | -3.685700 | 1.575651 | -0.309823 | -0.137656 | ... | 1.162112 | 7.620821 | 1.695061 | -3.956354 | 2.707644 | 4.657387 | 1.619307 | -5.537285 | 1.246650 | -1.162793 |
| 8947 | -3.793170 | NaN | 0.719610 | 2.306296 | 0.934728 | -0.984321 | 0.504867 | -0.441008 | -2.767177 | 1.734671 | ... | -1.292989 | 1.527720 | -0.496910 | 3.789736 | 1.130689 | 0.618278 | -0.111146 | 5.708912 | 1.542366 | -2.481019 |
6 rows × 40 columns
Calculate median value of each independent variable in X_val¶
Validate if the missing or null values have been imputed¶
Out[48]:
| V1 | V2 | V3 | V4 | V5 | V6 | V7 | V8 | V9 | V10 | ... | V31 | V32 | V33 | V34 | V35 | V36 | V37 | V38 | V39 | V40 |
|---|
0 rows × 40 columns
Observation(s):
- All missing or null values from X_val dataset have been imputed using median value.
- No missing or null values exist in X_val.
Impute Test dataset (X_test)¶
Display records with missing or null values¶
Out[49]:
| V1 | V2 | V3 | V4 | V5 | V6 | V7 | V8 | V9 | V10 | ... | V31 | V32 | V33 | V34 | V35 | V36 | V37 | V38 | V39 | V40 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 709 | 3.171300 | NaN | -0.899604 | -7.687193 | -1.844379 | 2.229502 | 0.649609 | 0.680742 | -0.079613 | -3.926869 | ... | -2.574088 | -8.569958 | 1.198974 | -3.747194 | -0.834087 | 0.364598 | 3.687177 | -1.450631 | -0.012682 | 6.569833 |
| 859 | NaN | 1.481190 | 2.208128 | -2.550029 | 1.526045 | -0.964918 | 0.559579 | 3.004337 | -3.937734 | 0.105693 | ... | 5.891019 | 5.093845 | 2.920339 | -3.080601 | 3.750633 | 2.422388 | -0.692277 | -0.182557 | -0.709241 | 2.498946 |
| 1070 | NaN | 1.222743 | 7.023517 | -1.227970 | -3.385548 | -1.500321 | -0.375947 | -2.898488 | 3.016750 | 0.097183 | ... | 1.327931 | -6.401373 | -2.539566 | -0.427894 | 4.971210 | 1.229448 | -1.620576 | -2.472413 | 0.692331 | -1.427785 |
| 1639 | NaN | -5.280584 | 1.695313 | -0.787160 | -1.872912 | -0.469312 | -1.970327 | -2.099606 | -1.573940 | 4.521625 | ... | -2.193425 | -1.939828 | -2.575771 | 5.279322 | 1.557176 | 5.542348 | 1.058056 | 1.696663 | 3.691808 | -3.802066 |
| 1777 | 1.255877 | NaN | 1.123121 | 0.347719 | -0.199314 | 0.542522 | -0.904536 | -2.398356 | 0.228689 | 0.245489 | ... | -0.854021 | 0.851358 | 1.657839 | -1.410919 | 3.587088 | -1.116910 | -0.865736 | 2.766820 | -0.368560 | -0.864084 |
| 1832 | NaN | -0.558554 | 5.315575 | 1.517019 | -2.304035 | -1.410233 | -1.974341 | -3.081827 | 1.762233 | 1.134216 | ... | -1.274813 | -3.017435 | -0.475546 | 1.987185 | 4.541473 | 1.335494 | -0.812582 | -0.545365 | 1.922588 | -4.117640 |
| 1869 | -1.272832 | NaN | 4.426359 | -3.013970 | -1.294693 | -0.883173 | -1.731633 | 0.098774 | -0.991360 | 2.495756 | ... | 4.026130 | -0.396983 | 1.190134 | 0.629071 | 2.411258 | 6.166668 | -0.140616 | -4.208798 | 2.623088 | -1.368893 |
| 2741 | -2.938927 | NaN | 2.913242 | 1.431121 | 4.003345 | -4.743048 | -2.450111 | 3.795883 | -0.339877 | -2.897700 | ... | 4.347062 | 12.077656 | 0.671770 | -6.354040 | 3.887011 | 3.420416 | 0.506994 | -5.913055 | 0.214129 | -0.931294 |
| 3266 | 5.896134 | NaN | 7.342806 | -1.052112 | -1.393952 | -0.410402 | 0.392391 | -6.141263 | 2.100145 | 1.897655 | ... | 2.074129 | -5.652135 | -1.973205 | 0.275744 | 3.894656 | 2.108591 | -2.803778 | 3.971349 | 2.233942 | -2.542753 |
| 4051 | NaN | 3.983783 | 0.524783 | -4.776552 | 2.590121 | 1.040410 | 3.097642 | -1.744755 | -0.269377 | 0.410768 | ... | 4.250514 | -4.134022 | -5.444258 | -1.925177 | -5.736453 | 4.155637 | 0.047600 | 3.864513 | 1.224684 | 4.916014 |
| 4186 | 5.034513 | NaN | 4.450708 | -6.077425 | 0.445417 | 2.491588 | 1.958447 | -5.311945 | -1.397204 | 2.755499 | ... | 5.075327 | -5.133693 | 1.139969 | -1.421471 | 0.822456 | 4.099736 | -2.152178 | 7.063377 | 2.377923 | 1.906096 |
11 rows × 40 columns
Calculate median value of each independent variable in X_test¶
Validate if the missing or null values have been imputed¶
Out[51]:
| V1 | V2 | V3 | V4 | V5 | V6 | V7 | V8 | V9 | V10 | ... | V31 | V32 | V33 | V34 | V35 | V36 | V37 | V38 | V39 | V40 |
|---|
0 rows × 40 columns
Observation(s):
- All missing or null values from X_test dataset have been imputed using median value.
- No missing or null values exist in X_test.
Model Building¶
Different kinds of models built for this project:¶
- Model_0 with one hidden layer and 40 neurons.
- Model_1 with two hidden layers of 40, 20 neurons respectively and default/simple SGD optimizer.
- Model_2 with two hidden layers of 40 and 20 neurons, default/simple SGD optimizer and "class_weight" parameter.
- Model_3 with two hidden layers (40 and 20 neurons, respectively), SGD with momentum=0.5 and class_weight parameter.
- Model_4 with two hidden layers (40 and 20 neurons, respectively), and Adam optimizer.
- Model_5 with two hidden layers (40 and 20 neurons, respectively), Adam optimizer, and dropout.
- Model_6 with two hidden layers (40 and 20 neurons, respectively), Adam optimizer, and BatchNomalization.
- Model_7 with two hidden layers (40 and 20 neurons, respectively), Adam optimizer, BatchNomalization and Dropout.
- Model_8 with two hidden layers (40 and 20 neurons, respectively), Adam optimizer, he_normalizer weight initialization and Dropout.
- Model_9 with two hidden layers (40 and 20 neurons, respectively), Adam optimizer using oversampled training dataset (SMOTE).
Note: I observed that the performance of the model with two hidden layers (40 and 20 neurons, respectively) was excellent; hence, I built 9 out of the 10 models using 2 hidden layers, but used different hyperparameters to tune the model and validated the performance.
Model Evaluation Criterion¶
Rationale for selecting Recall and F1-Score to validate model performance:
The cost of repairing a generator is much less than the cost of replacing it, and the cost of inspection is less than the cost of repair. In the context of this project, repairing a generator is considered a True Positive, replacing it is a False Negative, and an inspection is a False Positive.
The objective of this project is to develop a model that predicts actual generator failures. This will prevent costly failures, as replacing a failed generator is much more expensive than proactive repair. Therefore, the Recall score must be maximized so the model identifies as many failures as possible to avoid expensive replacements.
If the model predicts more failures than actually occur, leading to a high number of False Positives, the cost of inspection will increase.
In order to minimize both the cost of replacement and inspection, the Recall and F1-Score will be used to validate the model's performance. This approach ensures the model can identify most failures while managing false positives.
Functions required to validate model performance¶
Defining a DataFrame to store metrics so it is easy to compare the performance of models and choose the model with best performance to move forward with.¶
Out[54]:
| Model Name | Optimizer | Weight Init. | # Hidden Layers | # Neurons - Hidden Layer | Activation Function - Hidden Layer | # Epochs | Batch Size | Train Loss | Validation Loss | Train Recall | Validation Recall | Train Precision | Validation Precision | Train F1-Score | Validation F1-Score | Time (Secs) |
|---|
Defining Epochs and batch_size for models¶
Initial Model Building (Model 0)¶
- Let's start with a neural network consisting of
- just one hidden layer
- activation function of ReLU
- SGD as the optimizer
Building an initial model with one hidden layer and 40 neurons.¶
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 40) │ 1,640 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 1) │ 41 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 1,681 (6.57 KB)
Trainable params: 1,681 (6.57 KB)
Non-trainable params: 0 (0.00 B)
Epoch 1/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.2037 - val_loss: 0.1127 Epoch 2/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1019 - val_loss: 0.0985 Epoch 3/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0873 - val_loss: 0.0915 Epoch 4/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0797 - val_loss: 0.0869 Epoch 5/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0746 - val_loss: 0.0834 Epoch 6/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0709 - val_loss: 0.0806 Epoch 7/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0678 - val_loss: 0.0782 Epoch 8/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0653 - val_loss: 0.0762 Epoch 9/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0631 - val_loss: 0.0745 Epoch 10/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0612 - val_loss: 0.0730 Epoch 11/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.0596 - val_loss: 0.0718 Epoch 12/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0583 - val_loss: 0.0707 Epoch 13/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0572 - val_loss: 0.0697 Epoch 14/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0562 - val_loss: 0.0689 Epoch 15/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0552 - val_loss: 0.0682 Epoch 16/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0544 - val_loss: 0.0675 Epoch 17/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0536 - val_loss: 0.0669 Epoch 18/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0529 - val_loss: 0.0664 Epoch 19/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0523 - val_loss: 0.0659 Epoch 20/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0517 - val_loss: 0.0655 Epoch 21/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0511 - val_loss: 0.0650 Epoch 22/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0506 - val_loss: 0.0647 Epoch 23/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0502 - val_loss: 0.0644 Epoch 24/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0497 - val_loss: 0.0641 Epoch 25/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0493 - val_loss: 0.0638 Epoch 26/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0490 - val_loss: 0.0636 Epoch 27/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0486 - val_loss: 0.0634 Epoch 28/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0483 - val_loss: 0.0632 Epoch 29/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0480 - val_loss: 0.0629 Epoch 30/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0477 - val_loss: 0.0627 Epoch 31/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0474 - val_loss: 0.0625 Epoch 32/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0471 - val_loss: 0.0622 Epoch 33/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0468 - val_loss: 0.0620 Epoch 34/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0466 - val_loss: 0.0619 Epoch 35/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0463 - val_loss: 0.0617 Epoch 36/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0461 - val_loss: 0.0616 Epoch 37/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0459 - val_loss: 0.0614 Epoch 38/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0456 - val_loss: 0.0612 Epoch 39/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0454 - val_loss: 0.0611 Epoch 40/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0452 - val_loss: 0.0610 Epoch 41/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0450 - val_loss: 0.0609 Epoch 42/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0448 - val_loss: 0.0607 Epoch 43/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0446 - val_loss: 0.0606 Epoch 44/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0444 - val_loss: 0.0605 Epoch 45/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0442 - val_loss: 0.0605 Epoch 46/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0441 - val_loss: 0.0604 Epoch 47/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0439 - val_loss: 0.0603 Epoch 48/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0437 - val_loss: 0.0603 Epoch 49/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0436 - val_loss: 0.0602 Epoch 50/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0434 - val_loss: 0.0602
Time taken to train the model in seconds 30.228654146194458
500/500 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step
125/125 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step
Out[65]:
| 0 | |
|---|---|
| Model Name | model_0 |
| Optimizer | SGD |
| Weight Init. | Xavier |
| # Hidden Layers | 1 |
| # Neurons - Hidden Layer | 40 |
| Activation Function - Hidden Layer | ReLU |
| # Epochs | 50 |
| Batch Size | 64 |
| Train Loss | 0.050457 |
| Validation Loss | 0.060174 |
| Train Recall | 0.9905 |
| Validation Recall | 0.99025 |
| Train Precision | 0.990403 |
| Validation Precision | 0.990197 |
| Train F1-Score | 0.99018 |
| Validation F1-Score | 0.98988 |
| Time (Secs) | 30.228654 |
Observation(s):
- The loss decreases steeply until epoch 2, after which the rate of decrease slows.
- The performance metrics are high, which indicates that the model performs well in classifying generator failures.
- Based on the recall, precision, and f1-score metrics, the model performs well and generalizes.
- Training metrics are slightly higher than validation metrics, which could indicate that the model is slightly overfitting; however, the difference is within the acceptable range of 2%.
Next steps
- Build a model with two hidden layers with 40 and 20 neurons respectively.
- Validate the performance.
Model Performance Improvement¶
Model 1 with two hidden layers of 40, 20 neurons respectively and default/simple SGD optimizer.¶
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 40) │ 1,640 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 20) │ 820 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_2 (Dense) │ (None, 1) │ 21 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 2,481 (9.69 KB)
Trainable params: 2,481 (9.69 KB)
Non-trainable params: 0 (0.00 B)
Epoch 1/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.2407 - val_loss: 0.1225 Epoch 2/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.1127 - val_loss: 0.0972 Epoch 3/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0877 - val_loss: 0.0866 Epoch 4/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0760 - val_loss: 0.0805 Epoch 5/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0692 - val_loss: 0.0767 Epoch 6/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0647 - val_loss: 0.0740 Epoch 7/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0615 - val_loss: 0.0722 Epoch 8/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0591 - val_loss: 0.0707 Epoch 9/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0571 - val_loss: 0.0696 Epoch 10/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0555 - val_loss: 0.0688 Epoch 11/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0542 - val_loss: 0.0680 Epoch 12/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0530 - val_loss: 0.0674 Epoch 13/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0519 - val_loss: 0.0669 Epoch 14/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0510 - val_loss: 0.0664 Epoch 15/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0502 - val_loss: 0.0660 Epoch 16/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0494 - val_loss: 0.0657 Epoch 17/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0487 - val_loss: 0.0653 Epoch 18/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0480 - val_loss: 0.0650 Epoch 19/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0474 - val_loss: 0.0648 Epoch 20/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0468 - val_loss: 0.0646 Epoch 21/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - loss: 0.0462 - val_loss: 0.0644 Epoch 22/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0456 - val_loss: 0.0643 Epoch 23/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0451 - val_loss: 0.0641 Epoch 24/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0446 - val_loss: 0.0640 Epoch 25/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0442 - val_loss: 0.0638 Epoch 26/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0438 - val_loss: 0.0637 Epoch 27/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0434 - val_loss: 0.0636 Epoch 28/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0430 - val_loss: 0.0635 Epoch 29/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0426 - val_loss: 0.0635 Epoch 30/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0423 - val_loss: 0.0633 Epoch 31/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0419 - val_loss: 0.0632 Epoch 32/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0416 - val_loss: 0.0631 Epoch 33/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0413 - val_loss: 0.0630 Epoch 34/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0410 - val_loss: 0.0629 Epoch 35/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0406 - val_loss: 0.0628 Epoch 36/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0403 - val_loss: 0.0627 Epoch 37/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0401 - val_loss: 0.0625 Epoch 38/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0398 - val_loss: 0.0624 Epoch 39/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0396 - val_loss: 0.0623 Epoch 40/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0393 - val_loss: 0.0622 Epoch 41/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0391 - val_loss: 0.0621 Epoch 42/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0389 - val_loss: 0.0620 Epoch 43/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0387 - val_loss: 0.0619 Epoch 44/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0385 - val_loss: 0.0618 Epoch 45/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0383 - val_loss: 0.0617 Epoch 46/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0381 - val_loss: 0.0616 Epoch 47/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0379 - val_loss: 0.0615 Epoch 48/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0377 - val_loss: 0.0614 Epoch 49/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0375 - val_loss: 0.0613 Epoch 50/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0373 - val_loss: 0.0612
Time taken to train the model in seconds 32.545371294021606
500/500 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step
125/125 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step
Out[75]:
| 1 | |
|---|---|
| Model Name | model_1 |
| Optimizer | SGD |
| Weight Init. | Xavier |
| # Hidden Layers | 2 |
| # Neurons - Hidden Layer | [40,20] |
| Activation Function - Hidden Layer | ReLU |
| # Epochs | 50 |
| Batch Size | 64 |
| Train Loss | 0.044266 |
| Validation Loss | 0.061169 |
| Train Recall | 0.9915 |
| Validation Recall | 0.991 |
| Train Precision | 0.991426 |
| Validation Precision | 0.990915 |
| Train F1-Score | 0.991245 |
| Validation F1-Score | 0.990714 |
| Time (Secs) | 32.545371 |
Observation(s):
- Performance of model with 2 hidden layers (40, 20) improved slightly compared to performance of model with one hidden layer.
- The loss decreases steeply between epochs 4 and 5, after which the rate of decrease slows.
- The performance metrics are high, which indicates that the model performs well in classifying generator failures.
- Based on the recall, precision, and f1-score metrics, the model performs well and generalizes.
- Training metrics are slightly higher than validation metrics, which could indicate that the model is slightly overfitting; however, the difference is within the acceptable range of 2%.
Next steps
- Build a model with two hidden layers with 40 and 20 neurons respectively.
- Add class_weight.
- Validate the performance.
Model 2 with two hidden layers of 40 and 20 neurons, default/simple SGD optimizer and "class_weight" parameter.¶
Calculating class weights, as the dataset is imbalanced¶
Out[76]:
{0: 1.0587612493382743, 1: 18.01801801801802}
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 40) │ 1,640 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 20) │ 820 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_2 (Dense) │ (None, 1) │ 21 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 2,481 (9.69 KB)
Trainable params: 2,481 (9.69 KB)
Non-trainable params: 0 (0.00 B)
Epoch 1/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.9468 - val_loss: 0.2370 Epoch 2/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4834 - val_loss: 0.1974 Epoch 3/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4112 - val_loss: 0.1749 Epoch 4/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.3754 - val_loss: 0.1648 Epoch 5/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.3514 - val_loss: 0.1578 Epoch 6/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.3315 - val_loss: 0.1538 Epoch 7/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.3137 - val_loss: 0.1539 Epoch 8/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.3027 - val_loss: 0.1503 Epoch 9/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2938 - val_loss: 0.1465 Epoch 10/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2852 - val_loss: 0.1432 Epoch 11/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2787 - val_loss: 0.1400 Epoch 12/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2733 - val_loss: 0.1385 Epoch 13/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2674 - val_loss: 0.1347 Epoch 14/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2628 - val_loss: 0.1320 Epoch 15/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2583 - val_loss: 0.1343 Epoch 16/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2541 - val_loss: 0.1290 Epoch 17/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2521 - val_loss: 0.1266 Epoch 18/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2447 - val_loss: 0.1269 Epoch 19/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2445 - val_loss: 0.1256 Epoch 20/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2403 - val_loss: 0.1235 Epoch 21/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2373 - val_loss: 0.1263 Epoch 22/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2344 - val_loss: 0.1229 Epoch 23/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2330 - val_loss: 0.1208 Epoch 24/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2284 - val_loss: 0.1200 Epoch 25/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.2266 - val_loss: 0.1164 Epoch 26/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.2223 - val_loss: 0.1233 Epoch 27/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2204 - val_loss: 0.1215 Epoch 28/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.2190 - val_loss: 0.1182 Epoch 29/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.2175 - val_loss: 0.1152 Epoch 30/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.2137 - val_loss: 0.1202 Epoch 31/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.2126 - val_loss: 0.1192 Epoch 32/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.2074 - val_loss: 0.1170 Epoch 33/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.2067 - val_loss: 0.1191 Epoch 34/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.2071 - val_loss: 0.1197 Epoch 35/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.2040 - val_loss: 0.1191 Epoch 36/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2008 - val_loss: 0.1231 Epoch 37/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1982 - val_loss: 0.1137 Epoch 38/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1965 - val_loss: 0.1170 Epoch 39/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1919 - val_loss: 0.1168 Epoch 40/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1912 - val_loss: 0.1152 Epoch 41/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.1899 - val_loss: 0.1115 Epoch 42/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.1848 - val_loss: 0.1124 Epoch 43/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.1834 - val_loss: 0.1194 Epoch 44/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1880 - val_loss: 0.1096 Epoch 45/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1786 - val_loss: 0.1068 Epoch 46/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1782 - val_loss: 0.1108 Epoch 47/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1749 - val_loss: 0.1100 Epoch 48/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1730 - val_loss: 0.1194 Epoch 49/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1737 - val_loss: 0.1098 Epoch 50/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1701 - val_loss: 0.1111
Time taken to train the model in seconds 34.72442436218262
500/500 ━━━━━━━━━━━━━━━━━━━━ 1s 995us/step
125/125 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step
Out[86]:
| 2 | |
|---|---|
| Model Name | model_2 w/CW |
| Optimizer | SGD |
| Weight Init. | Xavier |
| # Hidden Layers | 2 |
| # Neurons - Hidden Layer | [40,20] |
| Activation Function - Hidden Layer | ReLU |
| # Epochs | 50 |
| Batch Size | 64 |
| Train Loss | 0.192813 |
| Validation Loss | 0.111127 |
| Train Recall | 0.984062 |
| Validation Recall | 0.97675 |
| Train Precision | 0.985965 |
| Validation Precision | 0.979437 |
| Train F1-Score | 0.984668 |
| Validation F1-Score | 0.97771 |
| Time (Secs) | 34.724424 |
Observation(s):
- Performance of model with 2 hidden layers (40, 20) and class_weight led to degrade in performance of the model.
- The loss decreases steeply until about 2nd epoch, after which the rate of decrease slows. Training loss is higher than valdiation loss with no convergence.
- The performance metrics are good although lower compared to the prior models, which indicates that the model performance is acceptable in classifying generator failures.
- Training metrics are slightly higher than validation metrics, which could indicate that the model is slightly overfitting; however, the difference is within the acceptable range of 2%.
- Time taken to train the model has increased by about 16 seconds while the performance decreased.
Next steps
- Build a model with two hidden layers with 40 and 20 neurons respectively.
- Add class_weight.
- Add momentum value of 0.5
- Validate the performance.
Model 3 with two hidden layers (40 and 20 neurons, respectively), SGD with momentum=0.5 and class_weight parameter.¶
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 40) │ 1,640 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 20) │ 820 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_2 (Dense) │ (None, 1) │ 21 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 2,481 (9.69 KB)
Trainable params: 2,481 (9.69 KB)
Non-trainable params: 0 (0.00 B)
Epoch 1/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.7680 - val_loss: 0.2330 Epoch 2/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4215 - val_loss: 0.2024 Epoch 3/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.3740 - val_loss: 0.1934 Epoch 4/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.3491 - val_loss: 0.1797 Epoch 5/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.3286 - val_loss: 0.1770 Epoch 6/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.3129 - val_loss: 0.1774 Epoch 7/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.3019 - val_loss: 0.1635 Epoch 8/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.2911 - val_loss: 0.1664 Epoch 9/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2835 - val_loss: 0.1615 Epoch 10/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2758 - val_loss: 0.1613 Epoch 11/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2693 - val_loss: 0.1598 Epoch 12/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2662 - val_loss: 0.1465 Epoch 13/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2574 - val_loss: 0.1510 Epoch 14/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2537 - val_loss: 0.1498 Epoch 15/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2481 - val_loss: 0.1389 Epoch 16/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2408 - val_loss: 0.1465 Epoch 17/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2366 - val_loss: 0.1421 Epoch 18/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2332 - val_loss: 0.1364 Epoch 19/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2238 - val_loss: 0.1448 Epoch 20/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2205 - val_loss: 0.1406 Epoch 21/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2165 - val_loss: 0.1405 Epoch 22/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2143 - val_loss: 0.1451 Epoch 23/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2108 - val_loss: 0.1362 Epoch 24/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2029 - val_loss: 0.1414 Epoch 25/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.2053 - val_loss: 0.1314 Epoch 26/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.1970 - val_loss: 0.1541 Epoch 27/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1892 - val_loss: 0.1400 Epoch 28/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2035 - val_loss: 0.1353 Epoch 29/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1899 - val_loss: 0.1291 Epoch 30/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1944 - val_loss: 0.1302 Epoch 31/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1918 - val_loss: 0.1248 Epoch 32/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1837 - val_loss: 0.1252 Epoch 33/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1839 - val_loss: 0.1160 Epoch 34/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1789 - val_loss: 0.1582 Epoch 35/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1782 - val_loss: 0.1338 Epoch 36/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1729 - val_loss: 0.1301 Epoch 37/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1714 - val_loss: 0.1445 Epoch 38/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1597 - val_loss: 0.1390 Epoch 39/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1665 - val_loss: 0.1423 Epoch 40/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1653 - val_loss: 0.1237 Epoch 41/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1607 - val_loss: 0.1468 Epoch 42/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1660 - val_loss: 0.1261 Epoch 43/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.1690 - val_loss: 0.1472 Epoch 44/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - loss: 0.1585 - val_loss: 0.1595 Epoch 45/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1546 - val_loss: 0.1417 Epoch 46/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1563 - val_loss: 0.1182 Epoch 47/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1432 - val_loss: 0.1335 Epoch 48/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1354 - val_loss: 0.1501 Epoch 49/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1515 - val_loss: 0.1283 Epoch 50/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1335 - val_loss: 0.1358
Time taken to train the model in seconds 33.93616580963135
500/500 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step
125/125 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step
Out[96]:
| 3 | |
|---|---|
| Model Name | model_3 w/cw and mom |
| Optimizer | SGD |
| Weight Init. | Xavier |
| # Hidden Layers | 2 |
| # Neurons - Hidden Layer | [40, 20] |
| Activation Function - Hidden Layer | ReLU |
| # Epochs | 50 |
| Batch Size | 64 |
| Train Loss | 0.154531 |
| Validation Loss | 0.135783 |
| Train Recall | 0.97825 |
| Validation Recall | 0.97 |
| Train Precision | 0.982873 |
| Validation Precision | 0.975391 |
| Train F1-Score | 0.979621 |
| Validation F1-Score | 0.971856 |
| Time (Secs) | 33.936166 |
Observation(s):
- The model with 2 hidden layers (40, 20), using class_weight and a momentum (mom) of 0.5, showed a degradation in performance.
- Validation loss was higher than training loss, with convergence occurring around the 45th epoch.
- The performance metrics are good, although lower than those of prior models, which indicates that the model's performance is acceptable in classifying generator failures.
- Training metrics are slightly higher than validation metrics, which could indicate that the model is slightly overfitting; however, the difference is within the acceptable range of 2%.
- Time taken to train the model has increased by about 15 seconds compared to the best performing model thus far (model #1), while the performance decreased.
Next steps
- Build a model with two hidden layers with 40 and 20 neurons respectively.
- Add Adam optimizer.
- Validate the performance.
Model 4 with two hidden layers (40 and 20 neurons, respectively), and Adam optimizer.¶
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 40) │ 1,640 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 20) │ 820 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_2 (Dense) │ (None, 1) │ 21 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 2,481 (9.69 KB)
Trainable params: 2,481 (9.69 KB)
Non-trainable params: 0 (0.00 B)
Epoch 1/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 2s 3ms/step - loss: 0.3078 - val_loss: 0.0829 Epoch 2/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0670 - val_loss: 0.0694 Epoch 3/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0552 - val_loss: 0.0634 Epoch 4/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0492 - val_loss: 0.0601 Epoch 5/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0452 - val_loss: 0.0580 Epoch 6/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0424 - val_loss: 0.0567 Epoch 7/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0400 - val_loss: 0.0564 Epoch 8/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0384 - val_loss: 0.0563 Epoch 9/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0368 - val_loss: 0.0563 Epoch 10/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0357 - val_loss: 0.0563 Epoch 11/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0346 - val_loss: 0.0559 Epoch 12/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0337 - val_loss: 0.0561 Epoch 13/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0326 - val_loss: 0.0562 Epoch 14/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0318 - val_loss: 0.0573 Epoch 15/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0309 - val_loss: 0.0574 Epoch 16/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0302 - val_loss: 0.0575 Epoch 17/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0295 - val_loss: 0.0577 Epoch 18/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0290 - val_loss: 0.0580 Epoch 19/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0284 - val_loss: 0.0585 Epoch 20/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0279 - val_loss: 0.0591 Epoch 21/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0274 - val_loss: 0.0595 Epoch 22/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0272 - val_loss: 0.0593 Epoch 23/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0265 - val_loss: 0.0609 Epoch 24/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0263 - val_loss: 0.0613 Epoch 25/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0260 - val_loss: 0.0615 Epoch 26/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0255 - val_loss: 0.0619 Epoch 27/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0252 - val_loss: 0.0624 Epoch 28/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0250 - val_loss: 0.0635 Epoch 29/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0248 - val_loss: 0.0629 Epoch 30/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0244 - val_loss: 0.0649 Epoch 31/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0244 - val_loss: 0.0641 Epoch 32/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0238 - val_loss: 0.0660 Epoch 33/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0236 - val_loss: 0.0650 Epoch 34/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0233 - val_loss: 0.0654 Epoch 35/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0230 - val_loss: 0.0663 Epoch 36/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0227 - val_loss: 0.0674 Epoch 37/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0225 - val_loss: 0.0667 Epoch 38/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0223 - val_loss: 0.0672 Epoch 39/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0221 - val_loss: 0.0676 Epoch 40/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0217 - val_loss: 0.0679 Epoch 41/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0213 - val_loss: 0.0684 Epoch 42/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0212 - val_loss: 0.0686 Epoch 43/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0212 - val_loss: 0.0681 Epoch 44/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0209 - val_loss: 0.0679 Epoch 45/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0202 - val_loss: 0.0695 Epoch 46/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0202 - val_loss: 0.0713 Epoch 47/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0203 - val_loss: 0.0704 Epoch 48/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0200 - val_loss: 0.0712 Epoch 49/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0192 - val_loss: 0.0728 Epoch 50/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0191 - val_loss: 0.0729
Time taken to train the model in seconds 34.42833089828491
500/500 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step
125/125 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step
Out[106]:
| 4 | |
|---|---|
| Model Name | model_4 |
| Optimizer | Adam |
| Weight Init. | Xavier |
| # Hidden Layers | 2 |
| # Neurons - Hidden Layer | [40, 20] |
| Activation Function - Hidden Layer | ReLU |
| # Epochs | 50 |
| Batch Size | 64 |
| Train Loss | 0.022106 |
| Validation Loss | 0.0729 |
| Train Recall | 0.99525 |
| Validation Recall | 0.9895 |
| Train Precision | 0.99522 |
| Validation Precision | 0.989278 |
| Train F1-Score | 0.99518 |
| Validation F1-Score | 0.989315 |
| Time (Secs) | 34.428331 |
Observation(s):
- The performance of the model with 2 hidden layers (40, 20) and the Adam optimizer led to an improvement in training recall and F1-score, while the corresponding validation scores were lower.
- Validation loss initially decreased steeply, but then started increasing with no convergence.
- However, the performance metrics are still good, which indicates that the model's performance is acceptable in classifying generator failures.
- Training metrics are slightly higher than validation metrics, which could indicate that the model is slightly overfitting; however, the difference is within the acceptable range of 2%.
- Time taken to train the model has increased by about 18 seconds compared to the best performing model thus far (#1).
Next steps
- Build a model with two hidden layers with 40 and 20 neurons respectively.
- Add Adam optimizer instead of SGD.
- Add class_weight
- Validate the performance.
Model 5 with two hidden layers (40 and 20 neurons, respectively), Adam optimizer, and dropout.¶
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 40) │ 1,640 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout (Dropout) │ (None, 40) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 20) │ 820 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_1 (Dropout) │ (None, 20) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_2 (Dense) │ (None, 1) │ 21 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 2,481 (9.69 KB)
Trainable params: 2,481 (9.69 KB)
Non-trainable params: 0 (0.00 B)
Epoch 1/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 2s 3ms/step - loss: 0.2737 - val_loss: 0.0980 Epoch 2/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.1183 - val_loss: 0.0845 Epoch 3/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0989 - val_loss: 0.0744 Epoch 4/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0880 - val_loss: 0.0695 Epoch 5/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0804 - val_loss: 0.0656 Epoch 6/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0775 - val_loss: 0.0636 Epoch 7/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0695 - val_loss: 0.0620 Epoch 8/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0690 - val_loss: 0.0610 Epoch 9/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0704 - val_loss: 0.0598 Epoch 10/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0657 - val_loss: 0.0600 Epoch 11/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0618 - val_loss: 0.0589 Epoch 12/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0611 - val_loss: 0.0589 Epoch 13/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0573 - val_loss: 0.0598 Epoch 14/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0572 - val_loss: 0.0599 Epoch 15/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0548 - val_loss: 0.0590 Epoch 16/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0514 - val_loss: 0.0586 Epoch 17/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0546 - val_loss: 0.0580 Epoch 18/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0526 - val_loss: 0.0586 Epoch 19/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0533 - val_loss: 0.0575 Epoch 20/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0524 - val_loss: 0.0576 Epoch 21/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0522 - val_loss: 0.0591 Epoch 22/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0491 - val_loss: 0.0581 Epoch 23/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0506 - val_loss: 0.0592 Epoch 24/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0505 - val_loss: 0.0592 Epoch 25/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.0485 - val_loss: 0.0589 Epoch 26/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0497 - val_loss: 0.0592 Epoch 27/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0464 - val_loss: 0.0581 Epoch 28/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0492 - val_loss: 0.0582 Epoch 29/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0480 - val_loss: 0.0574 Epoch 30/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0468 - val_loss: 0.0594 Epoch 31/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0472 - val_loss: 0.0590 Epoch 32/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0460 - val_loss: 0.0590 Epoch 33/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0463 - val_loss: 0.0590 Epoch 34/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0494 - val_loss: 0.0594 Epoch 35/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0456 - val_loss: 0.0582 Epoch 36/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0448 - val_loss: 0.0591 Epoch 37/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0494 - val_loss: 0.0587 Epoch 38/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0460 - val_loss: 0.0587 Epoch 39/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0417 - val_loss: 0.0601 Epoch 40/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0459 - val_loss: 0.0594 Epoch 41/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0467 - val_loss: 0.0592 Epoch 42/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0445 - val_loss: 0.0583 Epoch 43/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0467 - val_loss: 0.0594 Epoch 44/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0432 - val_loss: 0.0588 Epoch 45/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0450 - val_loss: 0.0592 Epoch 46/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0459 - val_loss: 0.0593 Epoch 47/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0437 - val_loss: 0.0595 Epoch 48/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0421 - val_loss: 0.0602 Epoch 49/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0462 - val_loss: 0.0605 Epoch 50/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0438 - val_loss: 0.0609
Time taken to train the model in seconds 38.84878945350647
500/500 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step
125/125 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step
Out[116]:
| 5 | |
|---|---|
| Model Name | model_5 w/DO |
| Optimizer | Adam |
| Weight Init. | Xavier |
| # Hidden Layers | 2 |
| # Neurons - Hidden Layer | [40, 20] |
| Activation Function - Hidden Layer | ReLU |
| # Epochs | 50 |
| Batch Size | 64 |
| Train Loss | 0.051128 |
| Validation Loss | 0.060909 |
| Train Recall | 0.99325 |
| Validation Recall | 0.9915 |
| Train Precision | 0.99323 |
| Validation Precision | 0.991419 |
| Train F1-Score | 0.993076 |
| Validation F1-Score | 0.99125 |
| Time (Secs) | 38.848789 |
Observation(s):
- The performance of the model with 2 hidden layers (40, 20) with Adam optimizer and nueron dropout led to an improvement in both training and validation recall and F1-score.
- Good performance metrics indicate that the model's performance is acceptable in classifying generator failures.
- Training metrics are mildly higher than validation metricshowever, the difference is within the acceptable range of 2%.
- Time taken to train the model has increased by about 18 seconds compared to the best performing model thus far (model #1).
- By far, this is the best model thus far despite the increase in training time.
Next steps
- Build a model with two hidden layers with 40 and 20 neurons respectively.
- Add Adam optimizer
- Add BatchNormalization
- Validate the performance.
Model 6 with two hidden layers (40 and 20 neurons, respectively), Adam optimizer, and BatchNomalization.¶
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 40) │ 1,640 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization │ (None, 40) │ 160 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 20) │ 820 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_1 │ (None, 20) │ 80 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_2 (Dense) │ (None, 1) │ 21 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 2,721 (10.63 KB)
Trainable params: 2,601 (10.16 KB)
Non-trainable params: 120 (480.00 B)
Epoch 1/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 3s 4ms/step - loss: 0.5570 - val_loss: 0.1578 Epoch 2/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.1244 - val_loss: 0.0708 Epoch 3/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0573 - val_loss: 0.0577 Epoch 4/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0439 - val_loss: 0.0534 Epoch 5/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0388 - val_loss: 0.0522 Epoch 6/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0357 - val_loss: 0.0514 Epoch 7/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0336 - val_loss: 0.0509 Epoch 8/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0322 - val_loss: 0.0512 Epoch 9/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0308 - val_loss: 0.0523 Epoch 10/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0297 - val_loss: 0.0527 Epoch 11/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0285 - val_loss: 0.0534 Epoch 12/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0278 - val_loss: 0.0538 Epoch 13/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0272 - val_loss: 0.0549 Epoch 14/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0266 - val_loss: 0.0547 Epoch 15/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0262 - val_loss: 0.0559 Epoch 16/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0256 - val_loss: 0.0569 Epoch 17/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 2s 5ms/step - loss: 0.0250 - val_loss: 0.0579 Epoch 18/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0248 - val_loss: 0.0582 Epoch 19/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0240 - val_loss: 0.0594 Epoch 20/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0237 - val_loss: 0.0604 Epoch 21/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0235 - val_loss: 0.0611 Epoch 22/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0230 - val_loss: 0.0608 Epoch 23/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0225 - val_loss: 0.0625 Epoch 24/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0222 - val_loss: 0.0619 Epoch 25/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0221 - val_loss: 0.0621 Epoch 26/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0210 - val_loss: 0.0624 Epoch 27/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0215 - val_loss: 0.0624 Epoch 28/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0211 - val_loss: 0.0638 Epoch 29/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0204 - val_loss: 0.0644 Epoch 30/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0204 - val_loss: 0.0643 Epoch 31/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - loss: 0.0201 - val_loss: 0.0638 Epoch 32/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0197 - val_loss: 0.0626 Epoch 33/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0192 - val_loss: 0.0650 Epoch 34/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0189 - val_loss: 0.0647 Epoch 35/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0187 - val_loss: 0.0646 Epoch 36/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0183 - val_loss: 0.0646 Epoch 37/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0184 - val_loss: 0.0658 Epoch 38/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0179 - val_loss: 0.0650 Epoch 39/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0174 - val_loss: 0.0651 Epoch 40/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0175 - val_loss: 0.0670 Epoch 41/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0166 - val_loss: 0.0660 Epoch 42/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0163 - val_loss: 0.0662 Epoch 43/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0166 - val_loss: 0.0662 Epoch 44/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0160 - val_loss: 0.0688 Epoch 45/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0157 - val_loss: 0.0684 Epoch 46/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.0151 - val_loss: 0.0678 Epoch 47/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0148 - val_loss: 0.0697 Epoch 48/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0151 - val_loss: 0.0705 Epoch 49/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0144 - val_loss: 0.0704 Epoch 50/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0140 - val_loss: 0.0702
Time taken to train the model in seconds 45.297874450683594
500/500 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step
125/125 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step
Out[126]:
| 6 | |
|---|---|
| Model Name | model_6 w/BN |
| Optimizer | Adam |
| Weight Init. | Xavier |
| # Hidden Layers | 2 |
| # Neurons - Hidden Layer | [40,20] |
| Activation Function - Hidden Layer | ReLU |
| # Epochs | 50 |
| Batch Size | 64 |
| Train Loss | 0.015545 |
| Validation Loss | 0.07019 |
| Train Recall | 0.993938 |
| Validation Recall | 0.98925 |
| Train Precision | 0.99388 |
| Validation Precision | 0.989056 |
| Train F1-Score | 0.993829 |
| Validation F1-Score | 0.988895 |
| Time (Secs) | 45.297874 |
Observation(s):
- The performance of the model with 2 hidden layers (40, 20) with Adam optimizer and BatchNormalization performs similar to the prior models.
- Good performance metrics indicate that the model's performance is acceptable in classifying generator failures.
- Training metrics are mildly higher than validation metricshowever, the difference is within the acceptable range of 2%.
- Time taken to train the model has increased significantly by about 65 seconds compared to the best performing model thus far (model #1).
Next steps
- Build a model with two hidden layers with 40 and 20 neurons respectively.
- Add Adam optimizer
- Add BatchNormalization
- Add Dropout
- Validate the performance.
Model 7 with two hidden layers (40 and 20 neurons, respectively), Adam optimizer, BatchNomalization and Dropout.¶
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 40) │ 1,640 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization │ (None, 40) │ 160 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout (Dropout) │ (None, 40) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 20) │ 820 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_1 │ (None, 20) │ 80 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_1 (Dropout) │ (None, 20) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_2 (Dense) │ (None, 1) │ 21 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 2,721 (10.63 KB)
Trainable params: 2,601 (10.16 KB)
Non-trainable params: 120 (480.00 B)
Epoch 1/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 3s 4ms/step - loss: 0.6685 - val_loss: 0.1709 Epoch 2/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.2293 - val_loss: 0.0890 Epoch 3/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.1292 - val_loss: 0.0681 Epoch 4/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.0982 - val_loss: 0.0631 Epoch 5/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.0895 - val_loss: 0.0598 Epoch 6/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0822 - val_loss: 0.0596 Epoch 7/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0781 - val_loss: 0.0588 Epoch 8/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0709 - val_loss: 0.0577 Epoch 9/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0717 - val_loss: 0.0561 Epoch 10/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0683 - val_loss: 0.0568 Epoch 11/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0675 - val_loss: 0.0563 Epoch 12/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0694 - val_loss: 0.0560 Epoch 13/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0623 - val_loss: 0.0556 Epoch 14/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0636 - val_loss: 0.0547 Epoch 15/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0625 - val_loss: 0.0546 Epoch 16/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0582 - val_loss: 0.0552 Epoch 17/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0588 - val_loss: 0.0548 Epoch 18/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0557 - val_loss: 0.0537 Epoch 19/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0589 - val_loss: 0.0524 Epoch 20/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.0574 - val_loss: 0.0534 Epoch 21/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0533 - val_loss: 0.0534 Epoch 22/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0551 - val_loss: 0.0529 Epoch 23/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0566 - val_loss: 0.0520 Epoch 24/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0533 - val_loss: 0.0523 Epoch 25/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0546 - val_loss: 0.0525 Epoch 26/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0546 - val_loss: 0.0519 Epoch 27/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0521 - val_loss: 0.0525 Epoch 28/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0534 - val_loss: 0.0515 Epoch 29/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0515 - val_loss: 0.0515 Epoch 30/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0531 - val_loss: 0.0517 Epoch 31/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0530 - val_loss: 0.0513 Epoch 32/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0475 - val_loss: 0.0517 Epoch 33/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0482 - val_loss: 0.0510 Epoch 34/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0502 - val_loss: 0.0511 Epoch 35/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.0462 - val_loss: 0.0511 Epoch 36/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0486 - val_loss: 0.0499 Epoch 37/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0487 - val_loss: 0.0502 Epoch 38/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0499 - val_loss: 0.0508 Epoch 39/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0481 - val_loss: 0.0501 Epoch 40/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0481 - val_loss: 0.0503 Epoch 41/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0466 - val_loss: 0.0500 Epoch 42/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0460 - val_loss: 0.0483 Epoch 43/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0458 - val_loss: 0.0496 Epoch 44/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0466 - val_loss: 0.0489 Epoch 45/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0462 - val_loss: 0.0490 Epoch 46/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0438 - val_loss: 0.0496 Epoch 47/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0441 - val_loss: 0.0492 Epoch 48/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.0460 - val_loss: 0.0487 Epoch 49/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.0473 - val_loss: 0.0484 Epoch 50/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0456 - val_loss: 0.0490
Time taken to train the model in seconds 45.00650978088379
500/500 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step
125/125 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step
Out[136]:
| 7 | |
|---|---|
| Model Name | model_7 w/BN and DO |
| Optimizer | Adam |
| Weight Init. | Xavier |
| # Hidden Layers | 2 |
| # Neurons - Hidden Layer | [40,20] |
| Activation Function - Hidden Layer | ReLU |
| # Epochs | 50 |
| Batch Size | 64 |
| Train Loss | 0.051016 |
| Validation Loss | 0.049 |
| Train Recall | 0.993062 |
| Validation Recall | 0.991 |
| Train Precision | 0.993058 |
| Validation Precision | 0.990915 |
| Train F1-Score | 0.992869 |
| Validation F1-Score | 0.990714 |
| Time (Secs) | 45.00651 |
Observation(s):
- The performance of this model almost similar to the prior models.
- Good performance metrics indicate that the model's performance is acceptable in classifying generator failures.
- Training metrics are mildly higher than validation metricshowever, the difference is within the acceptable range of 2%.
- Time taken to train the model has increased by about 30 seconds compared to the best performing model thus far (model #1).
Next steps
- Build a model with two hidden layers with 40 and 20 neurons respectively.
- Add Adam optimizer
- Add he_normal weight initializer
- Add Dropout
- Validate the performance.
Model 8 with two hidden layers (40 and 20 neurons, respectively), Adam optimizer, he_normalizer weight initialization and Dropout.¶
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 40) │ 1,640 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout (Dropout) │ (None, 40) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 20) │ 820 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_1 (Dropout) │ (None, 20) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_2 (Dense) │ (None, 1) │ 21 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 2,481 (9.69 KB)
Trainable params: 2,481 (9.69 KB)
Non-trainable params: 0 (0.00 B)
Epoch 1/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 2s 3ms/step - loss: 0.9049 - val_loss: 0.1449 Epoch 2/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.2069 - val_loss: 0.1058 Epoch 3/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.1439 - val_loss: 0.0919 Epoch 4/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1221 - val_loss: 0.0845 Epoch 5/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.1052 - val_loss: 0.0774 Epoch 6/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0964 - val_loss: 0.0741 Epoch 7/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0889 - val_loss: 0.0707 Epoch 8/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0782 - val_loss: 0.0687 Epoch 9/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0742 - val_loss: 0.0666 Epoch 10/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - loss: 0.0710 - val_loss: 0.0665 Epoch 11/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0688 - val_loss: 0.0644 Epoch 12/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0714 - val_loss: 0.0634 Epoch 13/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0667 - val_loss: 0.0633 Epoch 14/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0614 - val_loss: 0.0626 Epoch 15/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0631 - val_loss: 0.0630 Epoch 16/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0601 - val_loss: 0.0627 Epoch 17/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0561 - val_loss: 0.0623 Epoch 18/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0593 - val_loss: 0.0621 Epoch 19/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0551 - val_loss: 0.0616 Epoch 20/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0539 - val_loss: 0.0613 Epoch 21/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0558 - val_loss: 0.0624 Epoch 22/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0565 - val_loss: 0.0623 Epoch 23/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0551 - val_loss: 0.0610 Epoch 24/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0581 - val_loss: 0.0604 Epoch 25/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0514 - val_loss: 0.0596 Epoch 26/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0524 - val_loss: 0.0592 Epoch 27/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0502 - val_loss: 0.0597 Epoch 28/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0502 - val_loss: 0.0602 Epoch 29/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0507 - val_loss: 0.0597 Epoch 30/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0506 - val_loss: 0.0596 Epoch 31/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0491 - val_loss: 0.0600 Epoch 32/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0508 - val_loss: 0.0601 Epoch 33/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0457 - val_loss: 0.0605 Epoch 34/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0479 - val_loss: 0.0609 Epoch 35/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0474 - val_loss: 0.0613 Epoch 36/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0465 - val_loss: 0.0623 Epoch 37/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0448 - val_loss: 0.0620 Epoch 38/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0465 - val_loss: 0.0627 Epoch 39/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0458 - val_loss: 0.0632 Epoch 40/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0483 - val_loss: 0.0631 Epoch 41/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0482 - val_loss: 0.0633 Epoch 42/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0440 - val_loss: 0.0641 Epoch 43/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0441 - val_loss: 0.0627 Epoch 44/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0443 - val_loss: 0.0626 Epoch 45/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0457 - val_loss: 0.0616 Epoch 46/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0480 - val_loss: 0.0621 Epoch 47/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0455 - val_loss: 0.0632 Epoch 48/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0418 - val_loss: 0.0628 Epoch 49/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0423 - val_loss: 0.0629 Epoch 50/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0465 - val_loss: 0.0628
Time taken to train the model in seconds 38.79898548126221
500/500 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step
125/125 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step
Out[146]:
| 8 | |
|---|---|
| Model Name | model_8 w/DO |
| Optimizer | Adam |
| Weight Init. | he_normalizer |
| # Hidden Layers | 2 |
| # Neurons - Hidden Layer | [40,20] |
| Activation Function - Hidden Layer | ReLU |
| # Epochs | 50 |
| Batch Size | 64 |
| Train Loss | 0.051609 |
| Validation Loss | 0.062825 |
| Train Recall | 0.99225 |
| Validation Recall | 0.99025 |
| Train Precision | 0.99226 |
| Validation Precision | 0.990241 |
| Train F1-Score | 0.991999 |
| Validation F1-Score | 0.989855 |
| Time (Secs) | 38.798985 |
Observation(s):
- The performance of this model almost similar to the prior models.
- Good performance metrics indicate that the model's performance is acceptable in classifying generator failures.
- Training metrics are mildly higher than validation metricshowever, the difference is within the acceptable range of 2%.
- Time taken to train the model has increased by about 35 seconds compared to the best performing model thus far (model #1).
Next steps
- Generate oversampling training datasets using SMOTE
- Build a model with two hidden layers with 40 and 20 neurons respectively.
- Add Adam optimizer
- Add he_normal weight initializer
- Add Dropout
- Validate the performance.
Model 9 with two hidden layers (40 and 20 neurons, respectively), Adam optimizer using oversampled training dataset (SMOTE).¶
Before OverSampling, count of label '1': 888 Before OverSampling, count of label '0': 15112 After OverSampling, count of label '1': 6044 After OverSampling, count of label '0': 15112 After OverSampling, the shape of train_X: (21156, 40) After OverSampling, the shape of train_y: (21156,)
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 40) │ 1,640 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 20) │ 820 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_2 (Dense) │ (None, 1) │ 21 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 2,481 (9.69 KB)
Trainable params: 2,481 (9.69 KB)
Non-trainable params: 0 (0.00 B)
Epoch 1/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 2s 3ms/step - loss: 0.3596 - val_loss: 0.0896 Epoch 2/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1310 - val_loss: 0.0762 Epoch 3/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.1078 - val_loss: 0.0707 Epoch 4/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0953 - val_loss: 0.0693 Epoch 5/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0863 - val_loss: 0.0689 Epoch 6/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0798 - val_loss: 0.0695 Epoch 7/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0748 - val_loss: 0.0704 Epoch 8/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0708 - val_loss: 0.0712 Epoch 9/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0674 - val_loss: 0.0720 Epoch 10/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0642 - val_loss: 0.0731 Epoch 11/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0612 - val_loss: 0.0739 Epoch 12/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0587 - val_loss: 0.0756 Epoch 13/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0568 - val_loss: 0.0767 Epoch 14/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0545 - val_loss: 0.0781 Epoch 15/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0528 - val_loss: 0.0796 Epoch 16/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0510 - val_loss: 0.0817 Epoch 17/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0496 - val_loss: 0.0828 Epoch 18/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0478 - val_loss: 0.0841 Epoch 19/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0459 - val_loss: 0.0857 Epoch 20/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0446 - val_loss: 0.0876 Epoch 21/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0437 - val_loss: 0.0892 Epoch 22/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0424 - val_loss: 0.0907 Epoch 23/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0413 - val_loss: 0.0919 Epoch 24/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0401 - val_loss: 0.0942 Epoch 25/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0390 - val_loss: 0.0959 Epoch 26/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0378 - val_loss: 0.0979 Epoch 27/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0373 - val_loss: 0.0993 Epoch 28/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0363 - val_loss: 0.1000 Epoch 29/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0356 - val_loss: 0.1010 Epoch 30/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0348 - val_loss: 0.1030 Epoch 31/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0344 - val_loss: 0.1046 Epoch 32/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0334 - val_loss: 0.1051 Epoch 33/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0329 - val_loss: 0.1066 Epoch 34/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0323 - val_loss: 0.1084 Epoch 35/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0318 - val_loss: 0.1099 Epoch 36/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0307 - val_loss: 0.1133 Epoch 37/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0302 - val_loss: 0.1156 Epoch 38/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0292 - val_loss: 0.1161 Epoch 39/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0290 - val_loss: 0.1183 Epoch 40/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0285 - val_loss: 0.1209 Epoch 41/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0287 - val_loss: 0.1205 Epoch 42/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0277 - val_loss: 0.1209 Epoch 43/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0278 - val_loss: 0.1236 Epoch 44/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0266 - val_loss: 0.1244 Epoch 45/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0265 - val_loss: 0.1269 Epoch 46/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0256 - val_loss: 0.1292 Epoch 47/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0253 - val_loss: 0.1309 Epoch 48/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0255 - val_loss: 0.1311 Epoch 49/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0248 - val_loss: 0.1292 Epoch 50/50 331/331 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0241 - val_loss: 0.1310
Time taken to train the model in seconds 46.69398093223572
662/662 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step
125/125 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step
Out[158]:
| 9 | |
|---|---|
| Model Name | model_9 w/OS |
| Optimizer | Adam |
| Weight Init. | Xavier |
| # Hidden Layers | 2 |
| # Neurons - Hidden Layer | [40,20] |
| Activation Function - Hidden Layer | ReLU |
| # Epochs | 50 |
| Batch Size | 64 |
| Train Loss | 0.025021 |
| Validation Loss | 0.130974 |
| Train Recall | 0.991255 |
| Validation Recall | 0.98525 |
| Train Precision | 0.991251 |
| Validation Precision | 0.985417 |
| Train F1-Score | 0.99124 |
| Validation F1-Score | 0.985327 |
| Time (Secs) | 46.693981 |
Observation(s):
- The performance of this model almost similar to the prior models.
- Good performance metrics indicate that the model's performance is acceptable in classifying generator failures.
- Training metrics are mildly higher than validation metricshowever, the difference is within the acceptable range of 2%.
- Time taken to train the model has increased by about 17 seconds compared to the best performing model thus far (model #1).
Next steps
- Compare performance of all 10 models
- Choose the model with best performance.
- Run the model using test data.
- Document the performance.
Model Performance Comparison and Final Model Selection¶
Now, in order to select the final model, we will compare the performances of all the models for the training and validation sets.
Out[159]:
| Model Name | Optimizer | Weight Init. | # Hidden Layers | # Neurons - Hidden Layer | Activation Function - Hidden Layer | # Epochs | Batch Size | Train Loss | Validation Loss | Train Recall | Validation Recall | Train Precision | Validation Precision | Train F1-Score | Validation F1-Score | Time (Secs) | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | model_0 | SGD | Xavier | 1 | 40 | ReLU | 50 | 64 | 0.050457 | 0.060174 | 0.990500 | 0.99025 | 0.990403 | 0.990197 | 0.990180 | 0.989880 | 30.228654 |
| 1 | model_1 | SGD | Xavier | 2 | [40,20] | ReLU | 50 | 64 | 0.044266 | 0.061169 | 0.991500 | 0.99100 | 0.991426 | 0.990915 | 0.991245 | 0.990714 | 32.545371 |
| 2 | model_2 w/CW | SGD | Xavier | 2 | [40,20] | ReLU | 50 | 64 | 0.192813 | 0.111127 | 0.984062 | 0.97675 | 0.985965 | 0.979437 | 0.984668 | 0.977710 | 34.724424 |
| 3 | model_3 w/cw and mom | SGD | Xavier | 2 | [40, 20] | ReLU | 50 | 64 | 0.154531 | 0.135783 | 0.978250 | 0.97000 | 0.982873 | 0.975391 | 0.979621 | 0.971856 | 33.936166 |
| 4 | model_4 | Adam | Xavier | 2 | [40, 20] | ReLU | 50 | 64 | 0.022106 | 0.072900 | 0.995250 | 0.98950 | 0.995220 | 0.989278 | 0.995180 | 0.989315 | 34.428331 |
| 5 | model_5 w/DO | Adam | Xavier | 2 | [40, 20] | ReLU | 50 | 64 | 0.051128 | 0.060909 | 0.993250 | 0.99150 | 0.993230 | 0.991419 | 0.993076 | 0.991250 | 38.848789 |
| 6 | model_6 w/BN | Adam | Xavier | 2 | [40,20] | ReLU | 50 | 64 | 0.015545 | 0.070190 | 0.993938 | 0.98925 | 0.993880 | 0.989056 | 0.993829 | 0.988895 | 45.297874 |
| 7 | model_7 w/BN and DO | Adam | Xavier | 2 | [40,20] | ReLU | 50 | 64 | 0.051016 | 0.049000 | 0.993062 | 0.99100 | 0.993058 | 0.990915 | 0.992869 | 0.990714 | 45.006510 |
| 8 | model_8 w/DO | Adam | he_normalizer | 2 | [40,20] | ReLU | 50 | 64 | 0.051609 | 0.062825 | 0.992250 | 0.99025 | 0.992260 | 0.990241 | 0.991999 | 0.989855 | 38.798985 |
| 9 | model_9 w/OS | Adam | Xavier | 2 | [40,20] | ReLU | 50 | 64 | 0.025021 | 0.130974 | 0.991255 | 0.98525 | 0.991251 | 0.985417 | 0.991240 | 0.985327 | 46.693981 |
Observation(s):
- The performance of all models is good and comparable, and they generalize well.
- The top 3 best performing models are:
- Model_0, with one hidden layer (40 neurons) and the default SGD optimizer, had recall and F1-scores for its training and validation datasets that were almost identical. It took about 34 seconds to train the model.
- Model_1, with two hidden layers (40 and 20 neurons, respectively) and the default SGD optimizer, had recall and F1-scores for its training and validation datasets that were almost identical. It took about 33 seconds to train, which is similar to Model_0.
- Model_5, with two hidden layers (40 and 20 neurons, respectively) with the Adam optimizer and dropout, had recall and F1-scores for its training and validation datasets that were almost identical. It took about 50 seconds to train, which is slightly higher compared to Model_0 and Model_1.
Model chosen to move forward
- Considering that the Adam optimizer is widely used and is the default choice for modern neural network models, I chose Model_5 with the Adam optimizer and dropout as the best model to move forward.
- The next step is to test the performance of the model using unseen test data.
Now, let's check the performance of the final model on the test set.
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 40) │ 1,640 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout (Dropout) │ (None, 40) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 20) │ 820 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_1 (Dropout) │ (None, 20) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_2 (Dense) │ (None, 1) │ 21 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 2,481 (9.69 KB)
Trainable params: 2,481 (9.69 KB)
Non-trainable params: 0 (0.00 B)
Epoch 1/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 2s 3ms/step - loss: 0.3236 - val_loss: 0.1020 Epoch 2/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1345 - val_loss: 0.0845 Epoch 3/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.1068 - val_loss: 0.0753 Epoch 4/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0854 - val_loss: 0.0684 Epoch 5/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0819 - val_loss: 0.0660 Epoch 6/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0785 - val_loss: 0.0646 Epoch 7/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0751 - val_loss: 0.0615 Epoch 8/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0693 - val_loss: 0.0604 Epoch 9/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0698 - val_loss: 0.0602 Epoch 10/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0648 - val_loss: 0.0589 Epoch 11/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0581 - val_loss: 0.0581 Epoch 12/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0590 - val_loss: 0.0571 Epoch 13/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0599 - val_loss: 0.0572 Epoch 14/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0579 - val_loss: 0.0577 Epoch 15/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0547 - val_loss: 0.0564 Epoch 16/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0518 - val_loss: 0.0564 Epoch 17/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0545 - val_loss: 0.0567 Epoch 18/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0517 - val_loss: 0.0552 Epoch 19/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0511 - val_loss: 0.0564 Epoch 20/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0503 - val_loss: 0.0563 Epoch 21/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0483 - val_loss: 0.0561 Epoch 22/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0510 - val_loss: 0.0565 Epoch 23/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0480 - val_loss: 0.0577 Epoch 24/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0512 - val_loss: 0.0566 Epoch 25/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0464 - val_loss: 0.0574 Epoch 26/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0457 - val_loss: 0.0574 Epoch 27/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0458 - val_loss: 0.0570 Epoch 28/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0484 - val_loss: 0.0580 Epoch 29/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0447 - val_loss: 0.0570 Epoch 30/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0461 - val_loss: 0.0576 Epoch 31/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0423 - val_loss: 0.0577 Epoch 32/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0446 - val_loss: 0.0585 Epoch 33/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.0446 - val_loss: 0.0584 Epoch 34/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0454 - val_loss: 0.0589 Epoch 35/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0447 - val_loss: 0.0586 Epoch 36/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0453 - val_loss: 0.0589 Epoch 37/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0417 - val_loss: 0.0593 Epoch 38/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.0423 - val_loss: 0.0595 Epoch 39/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0449 - val_loss: 0.0599 Epoch 40/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0455 - val_loss: 0.0591 Epoch 41/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0464 - val_loss: 0.0580 Epoch 42/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0437 - val_loss: 0.0596 Epoch 43/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0430 - val_loss: 0.0607 Epoch 44/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0413 - val_loss: 0.0592 Epoch 45/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0423 - val_loss: 0.0597 Epoch 46/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0410 - val_loss: 0.0601 Epoch 47/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0395 - val_loss: 0.0609 Epoch 48/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0425 - val_loss: 0.0613 Epoch 49/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0425 - val_loss: 0.0614 Epoch 50/50 250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.0411 - val_loss: 0.0608
500/500 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step 125/125 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step 157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
Classification Report - Train data
precision recall f1-score support
0 0.99 1.00 1.00 15112
1 0.99 0.88 0.93 888
accuracy 0.99 16000
macro avg 0.99 0.94 0.96 16000
weighted avg 0.99 0.99 0.99 16000
Classification Report - Validation data
precision recall f1-score support
0 0.99 1.00 1.00 3778
1 0.98 0.85 0.91 222
accuracy 0.99 4000
macro avg 0.99 0.92 0.95 4000
weighted avg 0.99 0.99 0.99 4000
Classification Report - Unseen/Test data
precision recall f1-score support
0 0.99 1.00 1.00 4718
1 0.98 0.85 0.91 282
accuracy 0.99 5000
macro avg 0.99 0.93 0.95 5000
weighted avg 0.99 0.99 0.99 5000
Observation(s):
- The performance appears to be very strong, especially for the class imbalanced datasets.
- The model excels at identifying non-failures, with a near-perfect score across all metrics.
- For generator failures, the model has excellent precision, with few false positives. However, the recall is slightly lower, which indicates that the model misses some of the actual generator failures.
- Overall, the F1-scores indicate that this is a highly effective model for predicting generator failures.
Actionable Insights and Recommendations¶
Write down some insights and business recommendations based on your observations.
Actionable Insights
- ReneWind can deploy the final model to identify whether a given generator will fail or not, as it performs better than other models.
- The final model, chosen for its strong performance, has a recall score of 85%, meaning it correctly identifies 85% of actual generator failures.
- The remaining 15% of false positives will lead to incurring inspection costs while those generators work fine. Considering the cost of inspection is cheaper than repairing or replacing generators, a business decision must be made to determine if 15% false positives are acceptable.
- After deploying the model in production, continuously monitor its performance and make any necessary adjustments to ensure it doesn't drop below the threshold set by your organization.
- Also, monitor the quality and characteristics of the data to ensure the model's performance isn't degraded.
Recommendations
- Considering that the datasets contained ciphered data and lacked a data dictionary, it was difficult to understand the variables and their significance to the generator equipment. This created a "black box" scenario. Therefore, it is recommended to obtain meaningful data with a data dictionary that explains the purpose of each variable. This will enable further analysis and provide opportunities for more targeted model tuning.
- Establish a Service Level Agreement (SLA) to ensure the timely inspection and replacement of faulty generators based on their assigned risk level if one is not available already.