Smart City – Innovation, Connectivity and Integration of

Road Infrastructure

Bill ButtlarUniversity of Missouri-Columbia

Buttlar Research Group (esp. H. Majidifard, B. Jahangiri, K. Barri), Yaw Adu-

Gyamfi, Amir Alavi, Shantanu Chakraburtty, Henrique Reis, Missouri

Department of Transportation, Illinois Tollway, Missouri Asphalt Pavement

Association, National Science Foundation, American Society of Civil

Engineers, Virgin Hyperloop One

Acknowledgements

What is a ‘Smart City?’

3

“A smart city is an urban development vision to integrate multiple information and communication technology (ICT) solutions in a secure fashion to manage a city’s asset, including: schools, libraries, transportation systems, hospitals, power plants, water supply networks, waste management, law enforcement, and other community services.

The goal of building a smart city is to improve quality of life by using technology to improve the efficiency of services and meet residents’ needs. ICT allows city officials to interact directly with the community and the city infrastructure and to monitor what is happening in the city, how the city is evolving, and how to enable a better quality of life.”

en.wikipedia.org/wiki/Smart_city

http://www.barcinno.com/smart-city-barcelona/

Smart Infrastructure Applications

4

Energy-Harvesting Sidewalk (UK)

Infrastructure HealthDiagnosis and Self-Repair

Vehicle Charging

3D Scanning/Paving

Intelligent Infrastructure Research at MAPIL

Active Digital Asset Management System

Mobile Molecular Scanner

Smart / Nano-Modified Asphalt (Windgo) Partners:

Smartphone Pavement Assessment

Machine Learning

Infrastructure HealthDiagnosis and Self-Repair: Acoustic Emission

Source Location

Behnia,B., Dave, E.V., Buttlar, W.G., Reis. H., “Acoustic Emissions (AE) Technique for Evaluation of Embrittlement Temperature of Asphalt Binders: Development and Field Calibration”, International Journal of Road Materials and Pavement Design, Vol. 14, pp. 57-78, 2013.Sun, Z., Behnia, B., Buttlar, W. G., and Reis, H. "Assessment of Low-Temperature Cracking in Asphalt Materials Using an Acoustic Emission Approach," Journal of Testing and Evaluation, Vol. 45, No. 6, 2017, pp. 1948-1958.

Multifunctional Pavement Using RFID Sensing Technology

• Wireless sensor networks

• Radio frequency identification technology

• Battery free

scheme of the sensor tag

Realized prototype of the sensor(credit: Chakrabartty research group, Washington University in St. Louis

Attaching the tags on DC(T) samples and monitoring the crack propagation

Smart Materials

InputProcess

Definition:A material that exhibits one or more properties that can be significantly changed in a controlled fashion by external stimuli.

Output

Key point: Truly smart materials will have ‘intelligence’ built in at the material scale: Examples: smart roofing shingle, self-healing concrete

𝟏𝟏𝟏𝟏𝟑𝟑𝟑𝟑

Machine Learning Is Changing The World

Machine Learning in Transportation

• Transportation

Elimination of human error in the driving process should make our journeys faster and safer.

Optimization of shipping routes speeds delivery, and lowers cost/environmental footprint.

Example: Development of a model using an innovative machine

learning technique ~ Genetic Programming (GP), to predict the

fracture energy of asphalt mixture specimens at low temperatures

0

1

2

3

4

0 5 10Load

(kN

)

CMOD (mm)

Gf = Fracture energyGP

Graph GP

Linear GP

Tree GP

Mix Quality Characteristics:(voids, aggr. structure, asphalt content, recycling type/amt….)

Machine Learning in Materials

Majidifard, H., Jahangiri, B., Alavi, A., Buttlar, W., (2018), “New Machine Learning-based Prediction Models for Fracture Energy of Asphalt Mixture,” Measurement, 135 (2019) 438–451.

GP Method

In the case of GP, prior knowledge about the underlying

physical process based on engineering judgement can be

incorporated into the learning formulation, which greatly

enhances the usefulness of GP over other ML techniques,

such as Artificial Neural Networks (ANNs).

ANNs are black box models…you cannot recover the inside of the black box, whereas in GP, you get the highly nonlinear reln’s, plus the ability to put the solution in the form of a straight-forward equation. This is a breakthrough for practical engineering.

Gene Expression Programming

Invented by Koza (1992), GP is an extension to the genetic

algorithm (GA) approach, which can automatically

generate mathematical models, akin to Darwinian

evolutionary theory (millions of models generated, only the

best survive evolution).

GP

Graph GP

Linear GPTree GP

GP Approaches (Alavi and Gandomi 2011)

Our initial GEP-based prediction model

𝐺𝐺𝑓𝑓𝑗𝑗𝑚𝑚2 = 𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁 − 9 𝑁𝑁𝐴𝐴 − 𝑅𝑅𝑁𝑁𝑅𝑅 5.36 + 𝐴𝐴 − 𝐿𝐿𝐴𝐴𝑅𝑅𝐺𝐺 − 𝐿𝐿𝐴𝐴𝑅𝑅𝐺𝐺

+ 𝑁𝑁𝐴𝐴 𝐺𝐺4 − 1.7 + 𝑈𝑈𝐴𝐴𝑈𝑈 +𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁 − 𝑈𝑈𝐴𝐴𝑈𝑈

𝐴𝐴 + 𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁 𝑁𝑁𝐴𝐴 − 𝑁𝑁𝐴𝐴 + 6.45− 𝑅𝑅𝑁𝑁𝑁𝑁 + 𝑅𝑅𝑁𝑁𝑅𝑅

+𝐿𝐿𝐴𝐴𝑅𝑅𝐺𝐺𝑁𝑁𝐴𝐴 + 𝐿𝐿𝐴𝐴𝑅𝑅𝐺𝐺2 + 3.49𝐴𝐴 + 𝑁𝑁𝐴𝐴 × 𝑅𝑅𝑁𝑁𝑅𝑅

+ 𝐴𝐴 𝐴𝐴𝑅𝑅𝐴𝐴3 − 10𝐴𝐴 − 𝑅𝑅𝑁𝑁𝑅𝑅 × 𝑁𝑁𝐴𝐴 − 6.461 + 𝑈𝑈𝐴𝐴𝑈𝑈

* Derived after controlling millions of highly nonlinear models, which is not feasible via other nonlinear regression approaches

Training data Testing data

Measured vs. predicted Gf

Actually, three portions – Learning (70%), Validation (10%), and Testing (20%). Here we combined Learning and Validation in the figure (both are involved in model selection process).

Hamburg Machine Learning Project @ MAPIL

Table 1. Statistical parameters of the dependent and independent variables. Mix

type UTI (℃)

HTPG (℃)

AC (%)

ABR (%)

NMAS (mm)

RAP (%)

RAS (%) G AT CRC

(%) T

(℃) Passes R (mm)

Mean 1.1 87.8 58.9 5.9 28.8 11.3 20.9 8.0 1.2 1.2 2.3 52.2 13131 3.8 Median 1.0 86.0 58.0 5.7 32.5 12.5 20.4 0.0 1.0 1.0 0.0 50 10000 2.7 Range 1.0 18.0 24.0 2.8 48.4 14.3 35.3 33.0 1.0 1.0 10.0 24 15000 19.2 Max 2.0 98.0 70.0 7.9 48.4 19.0 35.3 33.0 2.0 2.0 10.0 64 20000 19.7 Min 1.0 80.0 46.0 5.1 0.0 4.8 0.0 0.0 1.0 1.0 0.0 40 5000 0.6

Sensitivity of Variables

Trends Predicted by Machine Learning Model

0.0

5.0

10.0

15.0

20.0

25.0

0 10000 20000 30000

Rut

Dep

th (m

m)

Number of Passes

4.5% AC5% AC5.5% AC

(b)

Pavement Management: You Know the Underlying Principle…

• Accurate pavement condition monitoring is important, Why?If the cracks are detected sooner, the maintenance process will be

less expensive.

Increased costdue to delayed

monitoring

Pavement Monitoring

• Traditional method

ARAN, laser-based sensing,

GPR

Cost :

A vehicle equipped with modern sensor and computing systems was purchased by the Ohio Department of Transportation for US$1,179,000 with an annual operating cost US$70,000 (Vavrik et al., 2013).

Deep Learning Approach for Automatic Pavement Distresses Detection using Google Street view images

Why Google images? Free and available for every road section

Why Automated system?

1) Significant cost benefit

2) Begins to remove human error and judgment from the calculation process

Annotating 7500 images with nine classes

Distress Type Distress IDReflective Crack D0Transvers Crack D1

Block Crack D2Longitudinal Crack D3

Alligator Crack D4Sealed Reflective Crack D5Lane Longitudinal Crack D6

Sealed Longitudinal Crack D7pothole D8

Table 1. Distress types versus their corresponding distress ID

* To our knowledge, this is the most comprehensive dataset annotated by pavement experts in the literature

Model accuracy

YOLO V2 D0 D1 D2 D3 D4 D5 D6 D7 D8D0 0.99 0.01 0.00 0.00 0.00 0.01 0.00 0.00 0.00D1 0.02 0.97 0.01 0.00 0.00 0.00 0.00 0.00 0.00D2 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.00D3 0.00 0.00 0.01 0.98 0.00 0.00 0.01 0.00 0.00D4 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 0.00D5 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00D6 0.00 0.00 0.00 0.00 0.00 0.00 0.99 0.00 0.00D7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00D8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00

Crack class name Precision Recall F1Reflective crack 0.93 0.76 0.84Transverse crack 0.9 0.83 0.86

Block crack 0.93 0.79 0.85Longitudinal crack 0.91 0.83 0.87

Alligator crack 0.91 0.74 0.82Sealed transverse crack 0.93 0.83 0.87

Sealed longitudinal crack 0.93 0.79 0.85Lane longitudinal crack 0.94 0.57 0.71

Pothole 0.96 0.78 0.86Average 0.93 0.77 0.84

Detection and classification results for nine distress types

Confusion matrices obtained on the classification dataset using YOLO v2

Development of a model using an innovative machine learning

technique ~ Genetic Programming (GP), to predict the Pavement

condition as PASER rating using the detected distresses from

developed crack detection model

Predicted PASERRating

GP

Graph GP

Linear GP

Tree GP

Number and types of detected

distresses per section

Pavement Condition Prediction Model Development based on YOLO Model outputs

𝑅𝑅𝑁𝑁𝑁𝑁𝐸𝐸𝑅𝑅= 𝑓𝑓[𝑑𝑑(1),𝑑𝑑(2),𝑑𝑑(3),𝑑𝑑(4),𝑑𝑑(5),𝑑𝑑(6),𝑑𝑑(7),𝑑𝑑(8),𝑑𝑑(9)]

where, d(1)= Reflective Crack, d(2)= Transverse Crack, d(3)= Block Crack, d(4)= Longitudinal Crack, d(5)= Alligator Crack, d(6) = Sealed Reflective Crack, d(7)= Lane Longitudinal Crack, d(8)= Sealed Longitudinal Crack,d(9)= Pothole. All the variable including d(1) to d(9) are the average number of distresses per each section.

Training dataset

Validation dataset

Testing dataset

Results from GEP prediction Model

Developing a U-Net based Model for Distress Quantification

a) Raw Image b) pre-trained U-Net output

a) Raw Image

c) Re-trained U-Net output

b) pre-trained U-Net output

c) Re-trained U-Net output

Benefits of Integrated YOLO and U-Net Model

• Considers both type and density of distresses

• Neither Yolo nor U-net modeling alone can assure accurate pavement condition assessment because:

• YOLO can detect types of cracks by drawing frames around them, but cannot differentiate the severity or density of the detected cracks.

• U-Net can quantify the density of cracks, but it can not discriminate crack types. As an example, longitudinal joint cracks and sealed cracks are not as detrimental to pavement condition as compared to wheel path longitudinal cracks. The U-net model is not able to address this issue.

30

https://fox2now.com/2019/10/03/hyperloop-test-pod-on-display-at-mizzou/

Is Hyperloop for Real? How Will it Integrate with Current Surface Transportation?

With Smart Cities of the Future?

Hyperloop One Pod at University of Missouri-Columbia, October 3-4, 2019

Hyperloop One Master Class, held at University of Missouri-Columbia, October 3, 2019

National Coverage on CNN

Hyperloop Smart Infrastructure Monitoring

• Integrating self-powered smart sensors in pylons and tube structures to ensure consistent response and to identify anomalies (i.e., micro-cracks)…wirelessly and economically

• Developing UAV monitoring system + solar-powered data transmission modules for collecting data and validating observations

• Developing machine learning algorithms for early prediction of structural fatigue, damage, cracking, and other anomalies

A network of low-cost, self-powered sensing nodes

Unveiled at ASCE National Convention, Oct. 10-13, 2019(to be shared with CAPSA, with permission from ASCE)

https://www.futureworldvision.org

Alternative Energy

Change on this scale can drive

confusion and dysfunction unless

industries, organizations, and

individuals are prepared to tackle

new realities

Autonomous Vehicles

Climate Change Smart Cities

High-Tech Construction/ Advanced Materials Policy & Funding

35

Floating City Example1. Concept Art for Flooded Cityscape

2. Systems View of Interactive Prototype

3. Floating City Detail Concept Art

4. Concept Art for Floating City Overview

5. Computer Model of City

1

2 3

4

www.MAPIL.Missouri.edu

Thanks!

buttlarw@Missouri.edu