PSI - Issue 41

10

America Califano et al. / Procedia Structural Integrity 41 (2022) 145–157 Author name / Structural Integrity Procedia 00 (2019) 000–000

154

Figure 7 - ROC curves for the first (a) and second (b) test attempts carried out on the XGBoost classification model trained with different complementary configuration sets. The numbers for the tested configurations are reported in bold near the curves. The legend describes the values of AUC computed for each ROC curve. The FNR vs. FPR curves are reported for the twelve configurations for the first (c) and second (d) tests. The legend described the values of the area under each of these new curves, namely 1-AUC. According to the definition of the ROC curve, the results shown in Figure 7 are representative of good prediction capabilities, but surely improvable. This means that the simple preliminary XGBoost classification model that has been implemented is robust, as its prediction capabilities are barely sensitive to the training and test sets. It has to be reminded that the training and test phases previously described have been carried out considering data belonging to the plane z = 6 extracted as bidimensional section of the 3D model. To further test the capabilities of the XGBoost model, quick analyses on data belonging to different z planes have been carried out by considering as training set the configurations n. 1, 2, 3, 5, 6 and 7 and as test set all the others. The results in terms of ROC curve for the analyses carried out with data belonging to plane z = 8 and z = 2 are reported in Figure 8. The ROC curve for plane z = 8 (Fig. 8a) is completely overlapped with the one obtained for plane z = 6 (Fig. 7a). In addition, the average FNR is 28% in this case as well (Figure 8c). The results for plane z = 2 (Fig. 8b), both in terms of ROC curve and AUC, are slightly better than the previous ones and a lower value for the average FNR has been computed (about 19%, Figure 8d). These last results show that the classification model is sensitive to the 3-dimensionality of the considered problem and, so, this aspect surely needs to be further explored.