PSI - Issue 70

Ch. Vihas et al. / Procedia Structural Integrity 70 (2025) 461–468

466

Model validation is the process of assessing the trained machine learning model's performance on unseen data. It helps us understand whether the model is underfitting or overfitting (Nguyen-Sy, et al (2023)). Figure 3 illustrates the performance of individual models in predicting the compressive strength of concrete (measured in MPa) across three different datasets. Each plot compares the predicted and actual compressive strength of various models. In the training set, all the ML models display blue dots aligned closely to the red dashed ideal line, indicating the predictions are nearly perfect for the training data. For the prediction set, all the ML models exhibit a near-perfect linear relationship with the actual values, demonstrating that the models learned the patterns of the training data very well. In the validation set, the orange dots remain fairly close to the ideal line, although there is a slight spread. This suggests the model performs well on hidden validation data, though there might be slight deviations, possibly due to overfitting or minor inconsistencies in the validation set. Nevertheless, the model appears to generalize well. In the test set, the green dots represent the test data predictions. The pattern remains close to the ideal line, although some outliers exist, particularly for higher compressive strength values. These outliers may indicate challenges the model encounters in predicting extreme values in the test set, but overall, the predictions are quite accurate. The performance metrics were measured for all four machine learning models and are presented in Table 1. From Figure 4, it is observed that XGBoost consistently outperforms the other ML models across training, testing, and validation datasets. It achieves the highest R² value of 0.9751 on validation, indicating strong predictive accuracy, and records the lowest error values in terms of MSE, R², MAE, and RMSE, particularly on the validation set. This suggests that XGBoost fits the training data well and generalizes well on unseen data. Random Forest shows decent performance, with high R² on training (0.9925) but a notable drop on testing and validation, indicating slight overfitting. ANN exhibits moderate performance, with reasonable training metrics but poorer generalization, as seen in the higher RMSE and MAE on testing and especially in the questionable validation RMSE, which may suggest a data inconsistency. Finally, SVM performs the worst among all models, with consistently low R² scores and high error values across all the datasets, and it struggles to identify the patterns in the data. Overall, XGBoost emerges as the most reliable and accurate model for this dataset. 3.2 Performance Metrics

Table 1 . Performance Metrics of Various ML Models

Sl. No.

Machine Learning Model

Dataset

R 2

MSE

RMSE

MAE

1.

Random Forest

Training

0.9925

1.8370

1.3554

0.8143

Testing

0.9163

16.442

4.0549

2.7962

Validation

0.9177

13.256

3.6409

2.6776

2.

XG BOOST

Training

0.9973

0.6668

0.8166

0.6240

Testing

0.9502

9.7794

3.1272

1.9455

Validation

0.9751

4.0164

2.0041

1.4354

3.

ANN

Training

0.9568

10.5923

3.2546

2.2801

Testing

0.9093

17.824

4.4287

3.2667

Validation

0.8783

3.2667

19.6137

4.4287

4.

SVM

Training

0.5303

115.281

10.7369

7.8151

Testing

0.5002

98.2297

9.9111

6.9981

Validation

0.4701

85.363

9.2392

7.6421