PSI - Issue 64

Available online at www.sciencedirect.com Available online at www.sciencedirect.com ScienceDirect Structural Integrity Procedia 00 (2023) 000 – 000

www.elsevier.com/locate/procedia

ScienceDirect

Procedia Structural Integrity 64 (2024) 1–5

SMAR 2024 – 7th International Conference on Smart Monitoring, Assessment and Rehabilitation of Civil Structures Editorial – SMAR 2024 Enzo Martinelli a, *, Moslem Shahverdi b , Alper Ilki c , Masoud Motavalli b a Department of Civil Engineering, University of Salerno, via Giovanni Paolo II, 132, 84084 Fisciano (SA), b Empa, Swiss Federal Laboratories for Materials Science and Technology, Ueberlandstrasse 129, 8600 Dübendorf, Switzerlandc c Department of Civil Engineering, Istanbul Technical University,34467 Maslak, Istanbul, Turkey Italy The International Conference on Smart Monitoring, Assessment, and Rehabilitation of Civil Structures (SMAR) serves as a global platform for scientists, engineers, enterprises, and infrastructure managers to present and discuss advancements in testing and monitoring technology, structural modelling and assessment methods, and the application of advanced materials for structural rehabilitation. SMAR 2024, the seventh conference in this series, was held in Salerno, Italy, from 4 to 6 September 2024. It was co-organized by the University of Salerno (UniSA), Italy, and the Swiss Federal Laboratories for Materials Science and Technology (Empa), Switzerland. The SMAR conference series was launched in 2011 in Dubai, UAE, and followed up in Istanbul, Turkey (2013), Antalya, Turkey (2015), Zurich, Switzerland (2017), Potsdam, Germany (2019), and Shanghai, China (2022). The SMAR2024 International Scientific Committee (ISC, Table 1) received almost 400 abstracts and, after a rigorous review process involving at least two reviewers for each paper, accepted 280 full papers. Researchers from institutions across more than 35 Countries (Fig. 1) contributed to the proceedings. The conference hosted more than 250 scientists and experts worldwide to present their solutions and findings in the following areas: ‒ Structural health monitoring; ‒ Performance and damage assessment; ‒ Damage control; ‒ Structural strengthening and repair; ‒ Durability issues related to harsh environments;

‒ Shape memory alloys in civil structures; ‒ Practical applications and case studies.

* Corresponding author. Tel.: +39-089+96-4098. E-mail address: e.martinelli@unisa.it

2452-3216 © 2024 The Authors. Published by ELSEVIER B.V. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0) Peer-review under responsibility of SMAR 2024 Organizers

2452-3216 © 2024 The Authors. Published by ELSEVIER B.V. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0) Peer-review under responsibility of SMAR 2024 Organizers 10.1016/j.prostr.2024.09.188

Enzo Martinelli et al. / Procedia Structural Integrity 64 (2024) 1–5 Author name / Structural Integrity Procedia 00 (2019) 000 – 000

2

2

Table 1: SMAR 2024 International Scientific Committee

First Name

Last Name

Affiliation

Country First Name

Last Name

Affiliation

Country

‪ Riadh

Al-Mahaidi

Swinburne University of Technology Australia

Maria Pina Limongelli

Politecnico di Milano

Italy

Giuseppina Amato

Queen's University Belfast

UK

Ken J.

Loh

University of California - San Diego USA

Ueli

Angst

ETH Zurich

Switzerland Geert

Lombaert Manshadi Martinelli

KU Leuven

Belgium

Yu Jan

Bai

Monash University

Australia

Behzad D.

BBR VT International Ltd

Switzerland

Bien

Wroclaw University of Technology Poland

Enzo

University of Salerno

Italy

Antonio Matteo

Bilotta

University of Naples "Federico II"

Italy

Urs

Meier

Empa

Switzerland

Breveglieri Briseghella Brunner Caggiano

Empa

Switzerland Marco

Menegotto

CIPRA srl re-fer AG

Italy

Bruno

Fuzhou University

China

Julien

Michels

Switzerland

Andreas Antonio

Empa

Switzerland Ayaho

Miyamoto Mobasher Motavalli

Yamaguchi University Arizona State University

Japan USA

University of Genoa

Italy

Barzin

Zehra Erdem

Canan Girgin

Yıldız‪Technical‪University

Turkey Turkey

Masoud

Empa

Switzerland

Canbay Carloni

Middle East Technical University Case Western Reserve University

John

Myers Nanni Nigro

Missouri University University of Miami

USA USA Italy

Christian

USA

Antonio Emidio

Joan Ramon Casas

Polytechnic University of Catalonia Spain

University of Naples "Federico II"

Zekai

Celep

Istanbul Technical University Istanbul Technical University

Turkey Turkey

Ehsan

Noroozinejad University of British Columbia

Canada

Oguz Cem Celik

Eva

Oller

Polytechnic University of Catalonia Spain

Eleni Dawn

Chatzi Cheng

ETH Zurich

Switzerland Piotr

Omenzetter

University of Aberdeen

UK

University of California - Davis

USA

Kutay

Orakcal

Bogazici University

Turkey Greece

Dimitrios

Chronopoulos

KU Leuven

Belgium Costas

Papadimitriou University of Thessaly

Antoni

Cladera Colombi

University of the Balearic Islands

Spain

Carlo

Pellegrino

University of Padua University of Salerno

Italy Italy

Pierluigi

Politecnico di Milano University of Porto

Italy

Marco

Pepe

Alvaro

Cunha

Portugal

Yuri

Petryna

TU Berlin

Germany

Christoph Tommaso

Czaderski D'Antino Di Prisco

Empa

Switzerland M. Dolores

G. Pulido

Eduardo Torroja Institute - CSIC Spain King Fahd University of Petroleum Saudi Arabia

Politecnico di Milano Politecnico di Milano University of Patras

Italy Italy

Muhammad K. Rahman

Marco

Saim

Raza Saiidi Sarker Sezen Shafei Smith Smith Soyoz

Empa

Switzerland

Stephanos

Dritsos

Greece

M. Saiid

University of Nevada - Reno

USA

Tamer Raafat

El Maaddawy United Arab Emirates University

UAE

Prabir

Curtin University

Australia

El-Hacha Ferrara Ferrier Focacci

University of Calgary Politecnico di Milano

Canada

Halil

Ohio State University Iowa State University

USA USA

Liberato

Italy

Behrouz

Emmanuel Francesco

University "Claude Bernard" - Lyon 1 France

Moslem Shahverdi

Empa

Switzerland

eCampus University

Italy

Ian F.C.

TU Munich

Germany Australia

Paula

Folino

University of Buenos Aires University of Salento Xiamen University University of Hannover

Argentina

Scott

University of Adelaide Boğaziçi‪University University of Cagliari

Mariaenrica Frigione

Italy

Serdar Flavio

Turkey

Jing

Gao

China

Stochino

Italy

Elyas

Ghafoori

Germany

Andreas Giovanni

Taras

ETH Zurich

Switzerland Switzerland

Amir K.

Ghorbani-Tanha University of Tehran

Iran

Terrasi

Empa

Mark

Green

Queen's University Tongji University

Canada

Romildo Dias Toledo Filho

Federal University of Rio de Janeiro Brazil Polytechnic University of Catalonia Spain

Xianglin Issam E

Gu

China USA

Nicola

Tosic

Harik

University of Kentucky

Thanasis

Triantafillou

University of Patras University of Perugia

Greece

Alper

Ilki

Istanbul Technical University

Turkey

Filippo

Ubertini

Italy

Daniele Thomas

Inaudi Keller

Smartec SA

Switzerland Radu

Vacareanu

TU of Civil Engineering Bucharest

Romania Portugal

EPFL

Switzerland Humberto

Varum

University of Porto

Chul-Woo Kim

Kyoto University

Japan

Mateusz

Wyrzykowski

Empa

Switzerland

Renata Philipp

Kotynia

Lodz University of Technology

Poland

Ufuk

Yazgan

Istanbul Technical University

Turkey

Krooß

University of Kassel

Germany Ecuador

Qian-Qian Yu

Tongji University

China

Eva

Lantsoght

San Francisco de Quito University

Xiao Lin

Zhao

University of South Wales

Australia

Janet

Lees

University of Cambridge

UK

Enzo Martinelli et al. / Procedia Structural Integrity 64 (2024) 1–5 Author name / Structural Integrity Procedia 00 (2019) 000 – 000

3

3

Fig. 1. The World of SMAR 2024

Moreover, seventeen mini-symposia were organized on the following topics: MS1: Multifunctional materials for sustainable constructions: integrated thermal, structural, and sensing systems; MS2: Research and development of Iron-based Shape Memory alloys and their engineering application technology in China; MS3: Digital Manufacturing in Construction; MS4: Intelligent digitalization in structural health monitoring and lifetime maintenance of complex structures; MS5: Smart FRP and steel structures; MS6: Innovative Methods in Strengthening of Concrete Bridges using FRP; MS7: Bio-based composites for rehabilitation and retrofitting of buildings and structures; MS8: Advances in the investigation of the bond mechanism of externally bonded composites and FRP bars; MS9: Advances in Fiber Optical Sensing Solutions for Infrastructure, Geotechnics, and Earth Sciences; MS10: Economic assessment and Life-Cycle performance in building and civil engineering works; MS11: Seismic-Fire combined assessment and optimization of interventions for buildings and infrastructures; MS12: Innovative solutions for fatigue strengthening of existing structures; MS13: Natural fibres for eco-compatible solutions in seismic and energy upgrading of masonry structures; MS14: Advancements in Risk and Reliability Assessment of Existing Structures; MS15: Shape Memory Alloys (SMAs) for Engineering Applications; MS16: systems and methods for transport infrastructure surveillance and monitoring; MS17: Advancements in Object Digitization and Analysis: A Mini-Symposium on Innovative Tools and Methods. The scientific quality of SMAR 2024 was also testified by the 6 Keynote Lectures who are recognised scholars working in prestigious academic Institutions or private Companies. Their names and the titles of their speeches are listed below: Prof. Fae Azhari (University of Toronto, Canada): Transcending discreteness in structural monitoring ; Prof. Konrad Bergmeister (University of Natural Resources & Life Sciences Vienna, Austria): AI-enhanced digital inspection of bridges ; Dr. Maria Gabriella Castellano (FIP MEC srl, Italy): Seismic retrofit of buildings in Italy through seismic isolation or energy dissipation ;

Enzo Martinelli et al. / Procedia Structural Integrity 64 (2024) 1–5 Author name / Structural Integrity Procedia 00 (2019) 000 – 000

4

4

Prof. Shirley Dyke (Purdue University, USA): Empowering engineers by leveraging AI in structural engineering and monitoring ; Prof. Urs Meier (Empa, Switzerland): Advancements in carbon fiber reinforced polymer tendons for structural rehabilitation ; Prof. Carlo Pellegrino (University of Padua, Italy): Management of road bridge networks in Italy by means of integrated SHM systems . Taking into consideration both the aforementioned keynote lectures and all the full papers, there is a balanced focus on both "structural health monitoring" and "strengthening of structures". The two best papers presented in each one of these two areas are awarded the Mirko- Roš‪Medal‪during‪the‪closing‪ceremony (Fig. 2).

Fig. 2. SMAR Conferences: Mirko- Roš‪Medal

We wish to express our gratitude to all authors for their contributions, which will serve as valuable references for practitioners, researchers, students, and academics. Special thanks to the members of the International Scientific Committee for their meticulous review of the papers. We also acknowledge the support of the SMAR 2024 Institutional and Technical sponsors and exhibitors (Fig. 3).

Fig. 3. Institutional and technical sponsors of SMAR 2024

5

Enzo Martinelli et al. / Procedia Structural Integrity 64 (2024) 1–5 Author name / Structural Integrity Procedia 00 (2019) 000 – 000

5

Our appreciation goes to our colleagues on the Organizing Committee, UniSA, Empa, and the conference secretaries Francesco Nigro, Paula Barboza and Ursula Sieber, for their tireless efforts and quick responses to the demands of the conference. SMAR 2024 Honorary Chairs: Masoud Motavalli (Empa, Switzerland), Alper Ilki (ITU, Turkey), SMAR 2024 Chairs: Enzo Martinelli (UniSA, Italy), and Moslem Shahverdi (Empa, Switzerland).

Available online at www.sciencedirect.com Structural Integrity Procedia 00 (2023) 000 – 000 Available online at www.sciencedirect.com ScienceDirect

www.elsevier.com/locate/procedia

ScienceDirect

Procedia Structural Integrity 64 (2024) 220–227

SMAR 2024 – 7th International Conference on Smart Monitoring, Assessment and Rehabilitation of Civil Structures 3D Deep Learning for Segmentation of Masonry Tunnel Joints Jack Smith a, *, Chrysothemis Paraskevopoulou a a University of Leeds, UK Abstract Historic masonry lined tunnels form a large proportion of the world’s railway tunnel stock. However, as many of these date fr om the second industrial revolution over 150 years ago, they typically contain large areas of structural deterioration. Masonry spalling is a pervasive form of surface damage and its severity, defined by the depth of spalling, is indicative of a tunnel’s structu ral condition. Current tunnel spalling condition assessment procedures are largely manual, so the extent of spalling observed on many historic tunnels presents a challenge for timely and cost-effective assessment. Automated machine learning based workflows have shown substantial potential for automating and reducing the subjectivity of the assessment process. A key step in these workflows involves segmenting the location of masonry joints from 3D point clouds of the tunnel lining in order to isolate masonry block locations. The most prevalent method is to unroll 3D tunnel lining data into 2D before applying U-Net based convolutional neural networks to segment joint locations. However, recent developments in 3D point based neural networks enable semantic segmentation to be conducted directly on the input point cloud. Point based methods such as KPCONV provide 3D feature characterization and enable semantic segmentation of a wider variety of tunnel geometries by default, since a handcrafted unrolling strategy is not required. This study conducted a performance comparison between 3D KPCONV, 2D U-Net, and XGBOOST feature classifier based joint identification techniques. In order to effectively compare a real-world use-case where time consuming manual data labelling should be minimized, the methods were only trained on a 9.94m section of tunnel. It was found that a 2D U-Net combined with tunnel unrolling workflow could be more successfully trained on the case study dataset and due to effective transfer learning, achieved superior performance to KPCONV and XGBOOST methods. © 2024 The Authors. Published by ELSEVIER B.V. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0) Peer- under responsibility of SMAR 2024 Organizers Keywords: Deep Learning, Masonry, Tunnels, Railways, Historic Structures © 2024 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0) Peer-review under responsibility of SMAR 2024 Organizers

* Corresponding author. Tel.: +44 07869466570

E-mail address: eejmws@leeds.ac.uk

2452-3216 © 2024 The Authors. Published by ELSEVIER B.V. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0) Peer-review under responsibility of SMAR 2024 Organizers

2452-3216 © 2024 The Authors. Published by ELSEVIER B.V. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0) Peer-review under responsibility of SMAR 2024 Organizers 10.1016/j.prostr.2024.09.233

Jack Smith et al. / Procedia Structural Integrity 64 (2024) 220–227 Smith, J., Paraskevopoulou, C. Structural Integrity Procedia 00 (2019) 000 – 000

221

2

1. Introduction It is vital that we efficiently operate and maintain our underground infrastructure to ensure it is sustainable for future generations (Paraskevopoulou et al., 2019). Railway tunnels form a key part of our transportation connectivity and a significant proportion of the world’s underground infrastructure . However, in many countries the majority of these tunnels are masonry lined and were constructed over 150 years ago. The condition of these tunnels must be regularly assessed to inform maintenance activities and ensure safety. Condition assessments generally involve an in person structural inspection where an assessing engineer manually annotates the type, severity and location of structural damage onto a plan of the structure. To avoid disruptive railway closures, these mostly take place overnight. Accurate and regular assessments help asset managers to understand how much structural rehabilitation is required (Atkinson et al., 2021), reducing the need for costly (Paraskevopoulou et al., 2021; 2022) and carbon intensive new constructions. Historic masonry tunnels typically feature large areas of low severity spalling and efflorescence damage, which is time-consuming to document and can obscure other damages (Chiu et al., 2015). This makes them challenging to accurately inspect within the time constraints of an overnight closure. As a result, inspections incur a substantial labour cost and their outputs can be subjective to the assessing engineer’s judgement . With the global push for net zero and increasingly tight operational budgets, there is mounting pressure for the condition assessments to be more repeatable and cost-effective. Digitalisation of the assessment process can help achieve these aims. There has been a considerable recent effort to automate railway infrastructure condition assessment tasks. While most research has focused on using photographic data or specialist sensors such as GPR and Ultrasound for digital condition assessments (Llanca et al., 2017), terrestrial lidar scanning (TLS) offers significant potential. TLS is increasingly routinely undertaken during structural inspections for documentation and measurement purposes, so further utilizing this data would require no additional equipment or change to established inspection procedures. Research has shown that using tunnel lining 3D point clouds obtained by lidar to directly identify tunnel lining damage can be significantly more efficient than manual methods (Kaartinen et al., 2022). Further to this, fully automated tunnel assessment procedures have been proposed that use recent advances in computer vision using machine learning to identify and locate damage locations on these point clouds (Zhou et al., 2021; Smith et al., 2023b; Bahreini and Hammad, 2024) . Many of these tackle damage localization as a pixelwise semantic segmentation problem. This involves classifying individual pixels in images of a structure as undamaged or damaged. Nevertheless, most of these methods have focused on concrete lined tunnels, with only a few studies looking at damage on masonry infrastructure. Masonry damage segmentation has the additional combined challenges of non homogeneous surfaces, the frequent existence of multiple overlapping defect types and a variety of masonry joint geometries (Chiu et al., 2015). Examples of these are shown in Fig. 1. Certain studies have created multi step automated workflows to combat these challenges, such as Smith et al. (2023b) who combined a geometric block face plane fitting strategy with deep learning-based block identification for spalling severity segmentation. Identifying masonry joint locations is a necessary first step of this method, as it enables analysis to be conducted on each block independently and separates the blocks from the mortar areas. Masonry joint segmentation also enables individual block labelling and documentation, which provides additional localization information to assessing engineers and maintenance teams, while helping to build a model of the structure that can be used for numerical analysis (Loverdos and Sarhosis, 2023). While multiple studies have analysed automated masonry joint segmentation methods, these have largely involved photographic data (Ibrahim et al., 2020; Kajatin and Nalpantidis, 2021). While these are shown to be broadly effective, they do not make full use of the 3D nature of point cloud data. Recent advances in deep learning methods have enabled semantic segmentation to be conducted directly on 3D point clouds. An issue with many of these methods is that they require a substantial amount of data labelling to achieve sufficient performance. This is a manual task that would erode savings in the overall condition assessment labour time. Due to these issues, in addition to the difficulties in generalizing a model to the large variety of masonry types, this paper analyses the situation where a model is trained on a short section of tunnel, before application to automate analysis of the remaining area of the same tunnel. With a focus on performance with limited training data, this paper assesses the feasibility of using supervised machine learning for masonry joint semantic segmentation from lidar data. Different methods are analysed when only trained on a 9.94m section of tunnel and applied to segment joints within the rest of the tunnel. This paper compares 2D and 3D deep learning methods with feature-based machine learning methods.

Jack Smith et al. / Procedia Structural Integrity 64 (2024) 220–227 Smith, J., Paraskevopoulou, C./ Structural Integrity Procedia 00 (2019) 000 – 000

222

3

Fig. 1. Typical masonry walls showing analysis complications.

2. Background Automation of masonry block semantic segmentation from 3D point cloud data has only become technically feasible since developments in both computer vision algorithms and lidar technology. While not previously used on masonry structures, Hackel et al. (2016) created a popular method for semantic segmentation of 3D point clouds that does not use neural networks. Using local features obtained by eigenvalue analysis at varying radii as inputs, a machine learning classifier was trained to determine a segmentation from the image contours. To demonstrate the advantages of deep learning methods, we used this method as a baseline in this study. XGBOOST (Chen and Guestrin, 2016) was used as the classifier with inputs set as the features suggested by (Hackel et al., 2016). XGBOOST (Extreme Gradient boosting) is an ensemble decision tree method that iteratively generates new trees to compensate for errors in the previous trees. One of the first automated methods for masonry joint segmentation from 3D location data was created by Valero et al. (2018) who developed a workflow that operated directly on 3D point clouds using continuous wavelet transforms. While their method has the flexibility to deal with surfaces in any orientation and works well on rubble masonry, performance decreases when joints form only small deviations from the masonry block surface, such as found in many brick walls. More recently, deep learning using neural networks has proven itself as a valuable tool for automating repetitive tasks in the construction industry (Dong and Catbas, 2021; Sony et al., 2021; Sabato et al., 2023). Since the development of VGG by Simonyan and Zisserman, (2014), convolutional neural networks have surpassed traditional computer vision techniques for image analysis. By training image convolutions, effectively image filters applied over consecutive pixel neighbourhoods, convolutional neural networks achieve higher performance than fully connected networks. The memory requirements are reduced, so deeper networks are possible that can achieve a higher accuracy. Encoder-decoder style convolutional neural networks have achieved state of the art performance for 2D semantic segmentation tasks (Ronneberger et al., 2015). They consist of an image encoder, which builds a description of the image, and a decoder that recreates the image with the target areas segmented. The U-Net adds skip connections to this design which improves the accuracy of the segmentation by combining unaltered data from an equivalent stage of the encoder with fully encoded feature representations. The U-Net design has proven enduringly popular due to its excellent transfer learning performance. This enables it to be pretrained on a general task, before being fine-tuned using only a small amount of data from the target domain. Recent studies on damage and joint segmentation of tunnel linings have first unrolled a tunnel 3D point cloud before rasterizing it into a 2D depth map image ready for a 2D U Net style neural networks (Ji et al., 2022; Zhang et al., 2022; Smith et al., 2023a, 2023b). This method takes advantage of a tunnel lining forming a single surface, with joints and damages forming only small offsets that can be fully represented on a 2D depth map. The latter enables it to leverage the relatively developed field of 2D computer vision

Jack Smith et al. / Procedia Structural Integrity 64 (2024) 220–227 Smith, J., Paraskevopoulou, C. Structural Integrity Procedia 00 (2019) 000 – 000

223

4

at the expense of being able to operate on outlying geometries such as side passages. As the current state of the art, the workflow developed by (Smith et al., 2023b) was also analysed in this study. Further to these methods, neural networks have been developed that can operate directly on 3D point clouds. These must deal with three additional complications over 2D images: • Points are unordered. As a result, a network must be able to identify and group similar features for varying permutations of different shapes. • Clouds can be sparse. There must be an efficient way of grouping points when there are substantially varying point densities across a cloud. • Point clouds are unstructured. This means that computationally expensive nearest neighbour calculations must be conducted to characterize spatial relationships. Many network architectures have been developed to deal with these issues. KPCONV (Thomas et al., 2019) has been chosen for this study, as it achieves state of the art performance, while being easily tuneable to a target application. KPCONV applies a 3D convolution that can operate with a varying number of points directly to the point cloud. These convolutions can also be made deformable, which allows them to adapt their shape to different local geometries. A comparison of the key methods trialled in this study is shown in Table 1.

Table 1. Comparison between different masonry joint segmentation algorithms. Method Cloud preprocessing requirement Description

Relative cost of computation

Background

XGBOOST

K Nearest neighbour points must be calculated within specified radius. 3D roughness features must be calculated for each point, as outlined by Hackel et al. (2016). Tunnel point cloud must be unrolled and then rasterized into a 2D Image. Workflow developed by (Smith et al., 2023b) K Nearest neighbour points must be calculated within specified radius.

Medium

Initially released in 2014, the XGBOOST library (Chen and Guestrin, 2016) was developed to improve the performance of decision tree methods. Developed by Ronneberger et al., (2015), the U-Net was the first semantic segmentation network to show excellent transfer learning performance. Thomas et al., (2019) developed KPCONV in order to investigate how image convolutions could be adapted for sparse 3D point clouds

Boosting decision tree ensemble method. (point feature based machine learning)

U-NET

2D encoder-decoder convolutional neural network (2D image deep learning)

Medium with GPU

KPCONV

High with GPU

3D encoder-decoder convolutional neural network with adjustments to work with sparse and uneven point clouds (3D point cloud deep learning)

3. Dataset A 3D point cloud of a 19.88m section of masonry lined tunnel obtained by TLS was provided by Bedi Consulting Ltd. for this research. The dimensions of the tunnel segment are shown in Fig. 2a. The tunnel is located in the west of England and carries a double track railway line through an urban area. The tunnel is predominantly lined with limestone blocks dating from its construction in the 1850s and contains multiple historic repairs. As a result, some areas of the sidewalls have been infilled with blocks of smaller brick masonry and there is a significant volume of spalling present. Spalling severity is classified into multiple categories depending on the spalling depth. For Stone masonry, low severity is typically defined as <40mm deep and its location only needs to be monitored. Higher severities such as Medium (>40mm) and high (>100mm) will require surface repairs or local reconstruction. This tunnel contains large areas of low severity spalling and small areas of medium severity spalling. Efflorescence is also present, but with no other major defects the overall condition is acceptable for continued railway operation. The masonry joint locations were labelled onto this point cloud as shown in Fig. 2b.

Jack Smith et al. / Procedia Structural Integrity 64 (2024) 220–227 Smith, J., Paraskevopoulou, C./ Structural Integrity Procedia 00 (2019) 000 – 000

224

5

Fig. 2. Tunnel point cloud used for training and testing the algorithms.

The tunnel was split into a training and a testing section, each 9.94m long and with 20% of the training section reserved as a validation set. Four algorithms were tested: pointwise XGBOOST, 2D U-Net with tunnel unrolling and 3D KPCONV with and without tunnel unrolling. In the unrolled cases, A cylinder was fit to the tunnel profile using principal component analysis. The tunnel was then unrolled around a cylinder to leave the profile as shown in Fig. 3. While this is not completely flattened, this representation enables rasterisation onto a 2D image without substantial distortions. As the distortion is present in both the training and testing data, the neural networks learnt to operate on images with the distortions present. Early stopping was used for all of the algorithms and training was conducted until there was no improvement in accuracy within the trailing 20 epochs.

Fig. 3. Unrolled 3D tunnel point cloud with Z co-ordinate set as scalar field.

4. Methods 4.1. XGBOOST

Features were created for the XGBOOST classifier using the method outlined by Hackel et al. (2016). 0.05m and 0.01m were selected as the nearest neighbor radii., as the average joint width within the dataset is 0.01m. These distances should therefore characterize both features solely within a joint and those that represent how a joint appears within the context of the broader point neighborhood. In addition to the spatial features developed by Hackel et al. (2016), surface roughness was used. This is defined as the offset of the target point from a 2D plane fit to the remaining points within the neighborhood. After the features were calculated, the number of block and joint points was balanced to ensure equal importance during the training. The XGBOOST model was then trained with a learning rate of 0.1 for 599 epochs.

Jack Smith et al. / Procedia Structural Integrity 64 (2024) 220–227 Smith, J., Paraskevopoulou, C. Structural Integrity Procedia 00 (2019) 000 – 000

225

6

4.2. U-Net In order to apply the U-Net, the Z co-ordinate of each point was first projected onto a 2D plane and rasterized into a depth map image. This was split into 256x256 image patches which were used in a batch size of 4 for training the U-Net model. The model was pretrained on ImageNet to reduce the number of training epochs required. In order to improve the performance when trained on a small dataset, augmentations were applied to the training data; These were random vertical and horizontal flips, random brightness and contrast shifts (representing changes in tunnel profile and location of the depth map) and the addition of gaussian noise. Training was conducted for up to 400 epochs with a learning rate of 0.001, a Dice score-based loss function, and an ADAM optimizer. 4.3. KPCONV The KPCONV network was examined with both a rigid and flexible kernel. This enabled us to study the impact of kernel deformations on the results. To examine the how well KPCONV can generalize features to different plane orientations with limited but representative training data, the method was assessed with both the raw input tunnel point cloud and the unrolled point cloud created for the U-Net method. 3D versions of the 2D data augmentations applied to the 2D image data described in section 3.3 were applied to the point cloud, with the addition of random rotations around the Z axis to account for any possible tunnel orientation. The network was trained with a learning rate of 0.001. The maximum radius of the convolution was selected iteratively as 0.15m. Due to the substantial computational cost for training KPCONV, a high-performance computer capped at 48 hours activity was used for training up to 1500 epochs. 5. Results Each algorithm was tested on a 9.94m section of tunnel that was pre-processed using the same methods as the training data. Performance was assessed using the Intersection Over Union (IOU) score, which is a metric that compares the amount of overlap between the ground truth and output segmentation with the differences between them. A score of 1 is a perfect segmentation, however 0.5 is typically considered a good benchmark. Where is the number of true positives, is the number of false positives and is the number of false negatives, the IOU is calculated as shown in equation 1. = /( + + ) (1) The results are shown in Table 2 and the ground truth joint locations and algorithm outputs are shown in Fig. 4. Table 2. U-Net and KPCONV results. Network Point cloud flattened IOU Training GPU (Nvidia) Training time (mins) U-Net Yes 0.4068 GTX 970 34 XGBOOST No 0.185 None 100 KPCONV Flexible No 0.3373 V100 2880 KPCONV Flexible Yes 0.3761 V100 2880 KPCONV Rigid No 0.2472 V100 2880 KPCONV Rigid Yes 0.3001 V100 2880 While some joint locations are identified in the XGBOOST output, the IOU was substantially lower than the other methods and it can be seen in Fig. 4b that the algorithm could not distinguish between joints and other anomalous areas that effect surface roughness properties such as efflorescence and spalling. An additional downside of the XGBOOST method was the substantial computation time on the test dataset, which was 150% to 200% more than the KPCONV and U-Net methods. This was because the eigen features and surface roughness values had to be evaluated before the trained XGBOOST model could be applied.

Jack Smith et al. / Procedia Structural Integrity 64 (2024) 220–227 Smith, J., Paraskevopoulou, C./ Structural Integrity Procedia 00 (2019) 000 – 000

226

7

Fig. 4. Algorithm outputs on a section of the test data compared with the ground truth.

Rigid KPCONV did not achieve an IOU substantially superior to XGBOOST, although the IOU improved when applied to the unrolled point cloud. This was likely because the network did not need to characterize features in as many orientations. Applying flexible convolutions also improved the KPCONV IOU, as the convolutions were able to adapt to the single surface nature of the data. Despite the substantial training time and accurate segmentation in some areas, the IOU of the best performing KPCONV network did not surpass the U-Net, as it failed in areas where the joints were very narrow. It is likely that the superior transfer learning ability of the U-Net is a key factor in its performance. While it is possible that a deeper version of KPCONV would be better able to characterize the multi scale features, this may lead to overfitting given the limited training data. With the small volume of training data and limited amount of training data augmentations, none of the methods could accurately segment the smaller brick joints. 6. Conclusion This study investigated three machine learning algorithms for masonry joint segmentation. A key challenge was achieving acceptable performance with limited training data. For the dataset analysed, a 3D point cloud deep learning framework, KPCONV was shown to achieve results approaching those of a 2D image-based U-Net. This suggested that small scale surface features were equally well characterized when viewed as a 2D depth map rather than as the raw 3D data. As a result, due to the substantial computational costs of KPCONV training, an unrolling and rasterizing method using off the shelf 2D image based semantic segmentation algorithms such as U-Net is recommended for analysing surface features in tunnel structures. While the U-Net and KPCONV methods perform well on the stone tunnel lining case study, it is possible that performance would decrease when the method is applied to brick masonry, due to the joint widths approaching accuracy limits of the TLS system used. We suggest that further work should be conducted to investigate how well each method operates on different masonry block materials and geometries and how effectively it can generalize when applied to types of masonry unseen in the training data. Acknowledgements The authors would like to thank Network Rail and Bedi Consulting Ltd. for collecting and sharing point cloud data of operational railway tunnels. This project was funded by EPSRC Environment DTP grant EP/T517860/1.

Jack Smith et al. / Procedia Structural Integrity 64 (2024) 220–227 Smith, J., Paraskevopoulou, C. Structural Integrity Procedia 00 (2019) 000 – 000

227

8

References

Atkinson, C., Paraskevopoulou, C., Miller, R., 2021. Investigating the rehabilitation methods of Victorian masonry tunnels in the UK. Tunnelling and Underground Space Technology 108, 103696. https://doi.org/10.1016/J.TUST.2020.103696 Bahreini, F., Hammad, A., 2024. Dynamic graph CNN based semantic segmentation of concrete defects and as-inspected modeling. Autom Constr 159, 105282. https://doi.org/10.1016/J.AUTCON.2024.105282 Chen, T., Guestrin, C., 2016. XGBoost: A Scalable Tree Boosting System. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 13-17-August-2016, 785 – 794. https://doi.org/10.1145/2939672.2939785 Chiu, Y.C., Wang, T.T., Huang, T.H., 2015. Investigating continual damage of a nineteenth century masonry tunnel. http://dx.doi.org/10.1680/feng.13.00030 167, 109 – 118. https://doi.org/10.1680/FENG.13.00030 Dong, C.Z., Catbas, F.N., 2021. A review of computer vision – based structural health monitoring at local and global levels. Struct Health Monit 20, 692 – 743. https://doi.org/10.1177/1475921720935585/ASSET/IMAGES/10.1177_1475921720935585-IMG6.PNG Hackel, T., Wegner, J.D., Schindler, K., 2016. FAST SEMANTIC SEGMENTATION OF 3D POINT CLOUDS WITH STRONGLY VARYING DENSITY, in: ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci. pp. 177 – 184. https://doi.org/10.5194/isprsannals-III-3-177-2016 Ibrahim, Y., Nagy, B., Benedek, C., 2020. Deep Learning-Based Masonry Wall Image Analysis. Remote Sensing 2020, Vol. 12, Page 3918 12, 3918. https://doi.org/10.3390/RS12233918 Ji, A., Chew, A.W.Z., Xue, X., Zhang, L., 2022. An encoder-decoder deep learning method for multi-class object segmentation from 3D tunnel point clouds. Autom Constr 137, 104187. https://doi.org/10.1016/J.AUTCON.2022.104187 Kaartinen, E., Dunphy, K., Sadhu, A., 2022. LiDAR-Based Structural Health Monitoring: Applications in Civil Infrastructure Systems. Sensors (Basel) 22, 4610. https://doi.org/10.3390/S22124610 Kajatin, R., Nalpantidis, L., 2021. Image Segmentation of Bricks in Masonry Wall Using a Fusion of Machine Learning Algorithms, in: ICPR 2021: Pattern Recognition. ICPR International Workshops and Challenges. Springer Science and Business Media Deutschland GmbH, pp. 446 – 461. https://doi.org/10.1007/978-3-030-68787-8_33/FIGURES/5 Llanca, D., Breul, P., Bacconnet, C., 2017. Improving the diagnosis methodology for masonry tunnels. Tunnelling and Underground Space Technology 70, 55 – 64. https://doi.org/10.1016/J.TUST.2017.07.002 Loverdos, D., Sarhosis, V., 2023. Geometrical digital twins of masonry structures for documentation and structural assessment using machine learning. Eng Struct 275, 115256. https://doi.org/10.1016/J.ENGSTRUCT.2022.115256 Paraskevopoulou, C., Cornaro, A., Admiraal, H., Paraskevopoulou, A., 2019. Underground space and urban sustainability: an integrated approach to the city of the future, in: International Conference on Changing Cities IV. University of Thessaly, pp. 198 – 208. Paraskevopoulou, C., Dallavalle, M., Konstantis, S., Spyridis, P., Benardos, A., 2022. Assessing the failure potential of tunnels and the impacts on cost overruns and project delays. Tunnelling and Underground Space Technology 123, 104443. https://doi.org/10.1016/J.TUST.2022.104443 Paraskevopoulou, C., Skolidis, A., Parsons, S., Marinos, V., 2021. Integrating uncertainty into geotechnical design of underground openings in tectonically undisturbed but lithologically varied sedimentary environments. Tunnelling and Underground Space Technology 113, 103979. https://doi.org/10.1016/J.TUST.2021.103979 Ronneberger, O., Fischer, P., Brox, T., 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation, in: Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015 . Munich, pp. 234 – 241. Sabato, A., Dabetwar, S., Kulkarni, N.N., Fortino, G., 2023. Noncontact Sensing Techniques for AI-Aided Structural Health Monitoring: A Systematic Review. IEEE Sens J 23, 4672 – 4684. https://doi.org/10.1109/JSEN.2023.3240092 Simonyan, K., Zisserman, A., 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings. Smith, J., Paraskevopoulou, C., Bedi, A., Invernici, M., 2023a. Deep learning for masonry lined tunnel condition assessment. Expanding Underground - Knowledge and Passion to Make a Positive Impact on the World- Proceedings of the ITA-AITES World Tunnel Congress, WTC 2023 2910 – 2917. https://doi.org/10.1201/9781003348030-351 Smith, J., Paraskevopoulou, C., Cohn, A., Kromer, R., Bedi, A., Invernici, M., 2023b. Automated masonry spalling severity segmentation in historic railway tunnels using deep learning and a block face plane fitting approach, (preprint). Leeds. Sony, S., Dunphy, K., Sadhu, A., Capretz, M., 2021. A systematic review of convolutional neural network-based structural condition assessment techniques. Eng Struct 226, 111347. https://doi.org/10.1016/J.ENGSTRUCT.2020.111347 Thomas, H., Qi, C.R., Deschaud, J.E., Marcotegui, B., Goulette, F., Guibas, L., 2019. KPConv: Flexible and deformable convolution for point clouds. Proceedings of the IEEE International Conference on Computer Vision 6410 – 6419. https://doi.org/10.1109/ICCV.2019.00651 Valero, E., Bosché, F., Forster, A., 2018. Automatic segmentation of 3D point clouds of rubble masonry walls, and its application to building surveying, repair and maintenance. Autom Constr 96, 29 – 39. https://doi.org/10.1016/J.AUTCON.2018.08.018 Zhang, Z., Ji, A., Wang, K., Zhang, L., 2022. UnrollingNet: An attention-based deep learning approach for the segmentation of large-scale point clouds of tunnels. Autom Constr 142, 104456. https://doi.org/10.1016/J.AUTCON.2022.104456 Zhou, M., Cheng, W., Huang, H., Chen, J., 2021. A Novel Approach to Automated 3D Spalling Defects Inspection in Railway Tunnel Linings Using Laser Intensity and Depth Information. Sensors (Basel) 21, 5725. https://doi.org/10.3390/S21175725

Available online at www.sciencedirect.com Available online at www.sciencedirect.com ScienceDirect Structural Integrity Procedia 00 (2023) 000 – 000

www.elsevier.com/locate/procedia

ScienceDirect

Procedia Structural Integrity 64 (2024) 901–907

SMAR 2024 – 7th International Conference on Smart Monitoring, Assessment and Rehabilitation of Civil Structures 3D-printing technology for integrating the rehabilitation and monitoring of civil structures with fiber optic Valentina Tomei a, *, Ernesto Grande a , Maura Imbimbo a , Michele A. Caponero b a Department of Civil and Mechanical Engineering, University of Cassino and Southern Lazio, via G. Di Biasio 43, 00143, Cassino, FR (Italy) b Centro Ricerche Frascati, ENEA, Frascati, Rome (Italy) Abstract In recent years, 3D printing technologies have become increasingly widespread in the field of architectural restoration of historical buildings and monuments. This is due to 3D printing's capability to accurately reproduce complex shapes with simplicity. Currently, within this field, 3D printing technology is generally employed for the physical reproduction of decorative/architectural components, while the reproduction of structural elements is still in its early stages. In this context, the paper presents an experimental study on 3D-printed samples made of PLA (Polylactic Acid Material), equipped with fiber optic sensors introduced inside the samples during the printing process. This innovative approach facilitates reintegration interventions while enabling real time monitoring of structures, marking a significant advancement in the field. However, since the fiber optic is incorporated within the element during the printing process, it is crucial to assess its potential impact on the mechanical properties of the samples and reliability of measurement system itself. Experimental tensile tests are then performed on samples carried out by considering both different printing paths and, moreover, different arrangements of the fiber optic inside the samples. Lastly, the study also examines the impact of artificially induced aging effects to assess their influence on both the mechanical properties of the samples and the reliability of measurements obtained from the fiber optic sensors. © 2024 The Authors. Published by ELSEVIER B.V. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0) Peer-review under responsibility of SMAR 2024 Organizers Keywords: 3D-printing; Tensile Tests; PLA; Dog-bone samples; Health Monitoring; Optical Fiber; FBG a b © 2024 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0) Peer-review under responsibility of SMAR 2024 Organizers

* Corresponding author. E-mail address: v.tomei@unicas.it

2452-3216 © 2024 The Authors. Published by ELSEVIER B.V. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0) Peer-review under responsibility of SMAR 2024 Organizers

2452-3216 © 2024 The Authors. Published by ELSEVIER B.V. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0) Peer-review under responsibility of SMAR 2024 Organizers 10.1016/j.prostr.2024.09.365

Valentina Tomei et al. / Procedia Structural Integrity 64 (2024) 901–907 Author name / Structural Integrity Procedia 00 (2019) 000 – 000

902 2

1. Introduction 3D printing is rapidly spreading in various fields, among which that of cultural heritage preservation. Advancements in technology and materials have made 3D printing more accessible, leading to the introduction of this in different research fields, showing its high potentialities. In the field of cultural heritage applications, recent studies [Xu et al. (2017) and Higueras et al. (2021)], particularly explored the use of 3D printing to rebuild missing components of historical artifacts. In a larger scale, Fotia et al. (2021) even used 3D printing to recreate an entire village in scale based on scans from drones and laser scanners. Beyond specific applications, other research activities focus on the mechanical characterization of the materials composing 3D printed components, in terms of stiffness, strength and ductility, also considering the effect on these of the printing parameters, such as the direction of the printed layers [Monaldo et al. (2023),Tomei et al. 2024)]. This is a fundamental step in the exploration of 3D printed components for architectural/structural applications. The present study explores the feasibility of utilizing 3D printing for the restoration of historical structures and health monitoring purposes. To achieve this, tensile tests were conducted on samples derived from a 3D printing process here proposed, designed to facilitate the incorporation of fiber optic filaments within the samples. Moreover, considering the potential exposure of 3D printed components to environmental effects, some of the samples underwent Accelerated The aim of this study is to explore the tensile behavior of 3D-printed components realized through Additive Manufacturing (AM). The investigation focuses on several factors that could affect the tensile behavior of the samples: the printing path, the presence of optical fiber sensors (FOSs) introduced into the sample during the printing process, the effects of aging. For this purpose, dog-bone samples were manufactured using Additive Manufacturing (AM) technology, specifically employing the Fused Filament Technique (FFT) and PolyLactic Acid (PLA) material. The samples were printed utilizing two distinct printing paths, referred to hereafter as 'Vertical' and 'Horizontal'. Additionally, some of these samples were equipped with fiber optics inserted during the printing process. Among these, a subset featured Fiber Bragg Grating (FBG) sensors within the fiber optics. Lastly, to evaluate the effects of aging, certain samples underwent Accelerated Aging tests prior to the tensile testing phase. H_dog-bone V_dog-bone H_F_dog-bone Aging tests before the tensile tests (Amza et al. (2021)). 2. 3D-printing process and description of the samples

inner zone printing -45 )

second layer printing

printing plane

edge printing

inner zone printing (+45 )

first layer printing

edge printing

(a) (c) Figure 1. Printing processes: (a) H_dog-bone samples, H_Ag_dog-bone samples; (b) V_dog-bone samples; (c) H_Fh_dog-bone samples, H_Fd_dog-bone samples, H_Fh(s)_dog-bone samples. (b)