DORIC An Architecture for Data-intensive Real-timeApplications

Miguel Kassick CadavizUniversity of Vale do Rio dos Sinos

Satildeo Leopoldo Brazilmiguelcadavizgmailcom

Kleinner FariasUniversity of Vale do Rio dos Sinos

Satildeo Leopoldo Brazilkleinnerfariasunisinosbr

Lucian Joseacute GonccedilalesUniversity of Vale do Rio dos Sinos

Satildeo Leopoldo Brazillucianjeduunisinosbr

Vinicius BischoffUniversity of Vale do Rio dos Sinos

Satildeo Leopoldo Brazilviniciusbischofeduunisinosbr

ABSTRACTThis study presents Doric a software architecture for data-intensivereal-time applications Dimensions of data-intensive real-time ap-plications are introduced as well as technologies that enable the im-plementation of such dimensions A case study involving a portableelectroencephalogram (EEG) enabled data collection based on real-istic scenarios found in data-intensive real-time applications TheDoric architecture was implemented using recent technologies (egApache Kafka) for building real-time data pipelines and stream-ing applications This prototype was evaluated in five scenarioscontaining different volumes of data The obtained results wereencouraging and show the potential for applying Doric as a struc-ture to foster the development of modern information systems inorganizations and to support serve as a guideline for new corporatearchitectures

CCS CONCEPTSbull Human-centered computing rarr HCI theory concepts andmodels

KEYWORDSSoftware Architecture Real Time Large Scale

ACM Reference FormatMiguel Kassick Cadaviz Kleinner Farias Lucian Joseacute Gonccedilales and ViniciusBischoff 2018 DORIC An Architecture for Data-intensive Real-time Appli-cations In Proceedings of ACM Woodstock conference (SBSIrsquo2018) JenniferB Sartor Theo DrsquoHondt and Wolfgang De Meuter (Eds) ACM New YorkNY USA Article 4 7 pages httpsdoiorg10475123_4

1 INTRODUCTIONIn Greece Doric denotes an architectural structure related to sup-portability of heavyweight constructions such as the Temple ofParthenon in Athens It is one of the three columns of ancient Greek

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page Copyrights for third-party components of this work must be honoredFor all other uses contact the ownerauthor(s)SBSIrsquo2018 Junho 2018 Caxias do Sul Rio Grande do Sul Brazilcopy 2018 Copyright held by the ownerauthor(s)ACM ISBN 123-4567-24-5670806httpsdoiorg10475123_4

and later Roman architecture used to support heavy marble struc-tures of the Greek temples the other two canonical columns werethe Ionic and the Corinthian In the context of modern informationsystems (IS) the reality is not different Their structures should bedesigned to withstand or support a heavy volume of data as mostof them are data-intensive real-time (DIRT) applications TheseDIRT applications are in a natural way related to Doric architec-tural structure with regards to how software architecture providesdata integrates heterogeneous data sources provides informationprocessing and analysis and storage support

Nowadays data are produced comes from devices such as porta-bles EEG (Electroencephalography) body sensors and cameras Inrecent years the trend of data production has intensified with therise of the internet of things [13] where interconnected sensorsand devices are in practically all everyday objects These devicesare beginning to become popular and to be applied in many areaswhere information systems are essential such as mental healthcare

For example portable devices such as Emotiv Epoc1 have beenused in several actions of mental health care such as collectionof mental data supervision of facial expressions and the mentalhealth status of patients These activities produce a big amount ofstructured and unstructured data Extracting valuable informationfrom these data implies into properly processing analyzing linkingand storing data [12] However this becomes a challenging taskdue to the long processing time in a scenario that users are notwilling to wait much for a server response

Although several works have been carried out by proposing tech-niques and architectures for information systems they do not treatinformation systems as a DIRT application This implies that hard-ware and software should operate under time constraints returnsresults quickly enough to influence the environment whereas thissystem receives and processes the data Still when developers needto implement DIRT applications traditional software architecturesare no longer adequate [6][10] There is a lack of a comprehensivearchitecture to properly represent important aspects of genericDIRT applications

This study therefore presents Doric a software architecture fordata-intensive real-time applications Dimensions of data-intensive

1Emotiv Epoc httpswwwemotivcom

SBSIrsquo2018 Junho 2018 Caxias do Sul Rio Grande do Sul Brazil M Cadaviz et al

real-time applications are introduced as well as technologies thatenable the implementation of such dimensions

A case study involving a portable electroencephalogram (EEG)enabled data collection based on realistic scenarios found in DIRTapplications The Doric architecture was implemented using re-cent technologies (eg Apache Kafka2) for building real-time datapipelines and streaming applications This prototype was evaluatedin five scenarios containing different volumes of data The obtainedresults were encouraging and show the potential for applying Doricas a structure to foster the development of modern informationsystems in organizations and to support serve as a guideline fornew corporate architectures

This article is organized as follows Section 2 presents the pro-posed architecture Section 3 describes the results and discussionof the empirical evaluation Section 4 presents the related worksFinally Section 5 presents the conclusion and final considerations

2 THE PROPOSED ARCHITECTUREThis section describes the proposed architecture Section 21 presentsan overview of the Doric architecture Section 22 introduces thearchitectural design by describing the Doricrsquos requirements andarchitectural components

21 General ViewFigure 1 presents the overview of the proposed architecture It iscomposed by six steps to process data and six architectural compo-nents These steps are described below

bull Step 1 Producers send information to the application serversto execute the data analysis

bull Step 2 The received data are processed Once the process isfinished the information are sent back to the servers Thenthe servers store this information

bull Step 3 This data flow is not mandatory An application mayuse this flow in the case of requiring a detailed analysis overa specific block of information

bull Step 4 The resulting data come from the analysis step thatare stored in a distributed database

bull Step 5 The consumers can receive the information withoutany treatment

bull Step 6 This data flow is used when consumers desire toaccess the result of the analysis

The objective of the proposed architecture is receiving a hugeamount of data in real-time from many producers at the sametime For this our architecture has a broker that receives the inputdata This broker controls the processing analyses and storage ofthese data providing them to consumers This component enablesthe relation between data source and consumer components EachDoric component will be described in Section 222

Table 1 shows as the Doric components can be supported bycurrent technologies The Doric architecture defines standards tosupport DIRT applications This will enable developers reusingaspects of the architecture in multiple domains such as smart cities[1] health systems [8] and smart highways [4] A system orga-nized in this manner allows developers mapping and modelling a

2Apache Kafka httpskafkaapacheorg

Figure 1 Proposed Architecture

structure in distinct blocks as proposed in [14] This enables theencapsulation of assignments and responsibilities of each block aswell as facilitating the maintenance and configuration of each stepThis is because the well-defined interfaces were designed with thepurpose to make feasible an architecture with low coupling

Table 1 Dimensions of DIRT applications (based on [14])

Data Sources

Publisher

and

Subscriber

Information

Processing

Analysis

and

Storage

Consumers

Data Centers Apache Kafka Apache Spark Apache Cassandra Health

Web Applications Amazon Kinesis Apache Storm Apache Hbase TI Operations

Air Pictures Google PubSub Apache Stamza Apache Druid Retail

Machines MapR Streams Twitter Heron Elasticsearch Government

Buildings Apache

DistributedLog

Amazon Kinesis Apache Solr Agriculture

Vehicles Google Dataflow MapR-DB Marketing

WearablesAzure Event

Apache Apex FiloDB Manufacture

Smartphones Apache Flink memsql Industry

Tablets Apache Flume Apache Kudu Science

Internet

of ThingsAzure Strem

Analytics

Google Cloud

Storage

Financial

Services

Apache Kafka

Streams

Google Cloud

Bigtable

Media

Agencies

DORIC An Architecture for Data-intensive Real-time Applications SBSIrsquo2018 Junho 2018 Caxias do Sul Rio Grande do Sul Brazil

22 Architectural projectThe proposed architecture was built in two steps In the first step weidentified the requirement an architecture must have The secondstep the technologies that implement each architecture module isdescribed These technologies are defined according to the definedrequirements in Step 1 These steps are described below

221 Requirements of the Proposed Architecture The require-ments were elaborated based on suggested features on the articlesand research reports We elaborated three non-functional require-ments (NFR) These requirements are described below

bull NFR01 support the process and reception of large-scaleinformation with integrity The architecture must sup-port and guarantee the correctness of informationrsquos that aretransferred in high volume

bull NFR02 able to return real-time responsesThe analyzedinformationrsquos must be provided in real-time to the consumersby the system

bull NFR03 the architecture must be extensible The archi-tecture must be well encapsulated and provide extensiblepoints This enables reuse of modules and the addition ofnew ones

222 Architectural components This section describes the Doriccomponents as well as suggests some technologies to implementthem Figure 3 shows how the components of the architecturewere organized Table 1 suggests some technologies to implementthe Doric components The chosen technologies seek to meet therequirements previously mentioned Note that each dimensionpresent in Table 1 corresponds to a Doric component which isimplemented by using technologies for building real-time datapipelines and streaming applications The following each compo-nent is briefly described

bull Data source The data source is not present in the architec-ture because they are external components This architectureis adapted to receive information from any data source

bull Publishersubscriber We choose the Apache Kafka forthis component Kafka is largely applied in the industry andis open source Kafka is suitable for two kinds of applica-tions first obtaining data in real time frommany applicationand systems safely Second systems that convert or reactto a certain data flux in real-time Kafka also promotes theabstraction of the data record from a determined topic Top-ics are categories for which the records are published Thetopics in Kafka are always multi-subscriber ie a topic canhave n subscribers to access their data The version ofApache Kafka used in this architecture is 01020 In addi-tion in this module the Apache ZooKeeper are implementedto cooperate with kafka [7] Apache ZooKeeper [3] providesseveral functionalities for distributed applications such asdistributed configuration management information coor-dination and locks The deployed version was ZooKeeper349 Both Kafka and ZooKeeper use Java 8 [11] release 131of the Java Runtime Environment (JRE) JRE consists on aJava virtual machine for running applications

bull Information Processering The Apache Spark [2] is re-sponsible to manage the information processing This tech-nology is powerful and open source

bull Data analysisWe implemented an application using Javato manage the data analysis This is because there is nosuggestion and consolidate techniques to conduct the dataanalysis

bull Storage of information These components implement theMongoDB [9] database This database is the state-of-art tech-nology for storing massive data and non-structured infor-mationrsquos We used the MongoDB version 344

bull Consumers Consumers are not defined in this architecturebecause they are external entities Any external entity canconsume the data provided by the architecture

Data Source PublisherSubscriber Consumers

Analysis

Informationrsquos Processing Storage

Figure 2 Implemented architecture

3 ASPECTS OF IMPLEMENTATION ANDEVALUATION

This section presents an empirical evaluation of the proposed archi-tecture The architecture is evaluated through realistic scenariosin which a large amount of data is generated in real time froma portable electroencephalogram (EEG) Section 31 describes theimplementation aspects of the proposed architecture Section 32 de-scribes the adopted evaluation method Finally Section 33 presentsthe results and discussion about the performance of the imple-mented system that adopted the proposed architecture

SBSIrsquo2018 Junho 2018 Caxias do Sul Rio Grande do Sul Brazil M Cadaviz et al

APACHESPARK

DATA SOURCES

MONGODB

ANALYSISAPPLICATION

CONSUMERSAPACHEKAFKA

Legend

A

Component

Provided Interface

Required Interface

Aspectual Connector

Figure 3 Component diagram

31 Implementation AspectsThe publishing and receiving stage implemented with ApacheKafka controls the entire data flux acting as the central nervoussystem of the structure but the available producer and consumerin the prototype work via prompt through the use of batch filesIn addition activities such as starting the Kafka server creating anew information topic or even listing messages also are managedthrough the prompt command lines

The prototype of the Doric architecture was constructed usingJava language Maven3 were used to manage the construction anddependencies of the project User interface was created using Swingjava library using Netbeans IDE The prototype was built to sim-ulate simple interactions with the Kafka cluster the sending ofinformation to the servers and consumers obtaining the analyzedinformation

Figure 4 shows an overview of the prototype It is possible toverify the existence of areas representing the dimensions of DIRTapplications Each of these areas implements some functions relatedto the respective area The list bellow describes these functionsaccording to each dimension

The DORIC components implemented in the prototype were(1) Publish and subscribe This component represented in

Figure 4(1) is related to the functions responsible for (a) changetopics it allows to switch between the existing topics inKafka As previously mentioned a topic is a category inwhich the input data are recorded (b) create topics it allowsthe creation of a new topic

(2) Data source We have used the EPOC Emotiv 4 as datasource This component represented in Figure 4(2) is re-sponsible for the functions related to manage of data sourcesThese functions enable (a) create new message generates asingle and unique message in the selected topic (b) importmessages performs the import of plain text messages fromthe file to the selected topic

3Maven httpsmavenapacheorg4EPOC Emotiv httpswwwemotivcomepoc

Figure 4 An overview of the prototype

(3) Consumer responsible to accommodate functions to man-age Consumers such as (a) last messages displays the lastmessages sent to the selected topic

32 Evaluation methodThe purpose of architecture evaluation is to present some results ob-tained after submitting information to the publishing and receivingcomponent Then the objective is to demonstrate the architectureto handle large-scale and real-time information The architecturewas built on a virtual machine with the following requirements

bull Windows 10 Operating Systembull Intel i7 (27 GHz) CPU with 4 coresbull 4 GB RAM DDR4 2133 MHz of memorybull 30 GB (5400 RPM) of Hard Disk

The scenario where this architecture is evaluated consists inprocessing and analyzing data from a health-care system Thissystem collects mental indicators data from a portable EEG Thenthe test is divided in two steps

The first assessment considers the architecturersquos ability to readlarge files For this several files with information of a wirelesselectroencephalogram in a csv file were submitted to the infra-structure The largest of them has 50 MB of information totalingmore than 50 thousand of lines records and around of 51 millioncharacters The second step tests the concept of large scale For thisseveral threads are being created to process the associated data ofthe producer to the system publish information simultaneously

33 ResultsMeasurements of time used in the experiment were carried onthe upper layers of the prototype thus creating a more realisticenvironment for the end user However this method of measuringjust considering the time spent in the transmission of informationsuch as processing time Several tests of single message submissionwere performed using the prototype which obtained an averageresponse time of 318 ms We did not observe a correlation betweenresponse time and the size of the message the type of contentneither the frequency of the topic structure

DORIC An Architecture for Data-intensive Real-time Applications SBSIrsquo2018 Junho 2018 Caxias do Sul Rio Grande do Sul Brazil

The evaluation of capacity of real-time response was performedwith five different file sizes Table 2 presents the obtained resultsThe import process was repeated fifty times for each file to obtainreliable values The size of the file influences in the processingtime The time increased when larger files were processed Wealso observed the time between sending the last record and thenotification of process completion In this case the average time alsoincreased according to the size of the file Specifically the averagetime increases significantly for 30 MB files Figure 5 illustrates theresults contained on Table2

Table 2 Test results of data provided from direct files

File

Size (MB)

Dispatch Time

Averarge (ms)

Receiving Time

Averarge (ms)

10 34366 12743

20 39682 140025

30 42562 201919

40 90672 222783

50 162845 22626

Next we also evaluated the architecturersquos capability to handlelarge-scale data For this we executed multiple threads to simulatethe role of data sources These threads simultaneously executedand provided the same topic Each thread creates a producer andgenerates a certain number of messages We monitored two aver-age times in this case (1) the average time spent on sending themessages to the server and (2) the average time the infrastructuretakes to signalize that it received the information Figure 5 depictsthe results Table 3 specifies the results obtained

File Size (MB)

Tim

e (

ms)

Figure 5 Receiving time average

The results evidence a gradual increase in the receiving timewhen there are more threads in execution The same occurs in themoment that the number of messages increases This behavior wasexpected because the tests were conducted on a virtual machineThere is a limitation regarding the hardware capability eg there isonly one CPU This implies that a larger number of threads causes

Table 3 Test results of data provided from threads

ThreadsMenssages

Number

Dispatch Time

Averarge (ms)

Receiving Time

Averarge (ms)

2 2 0 45083

4 2 0 52482

8 2 0 67111

16 2 0 68427

32 2 32 75904

1024 2 2592 182025

2 20 0 25002

4 20 0 21881

8 20 0 33174

16 20 0 97486

32 20 632 102732

1024 20 31448 200572

congestion on the processing queue There is more processes wait-ing on the queue because there is no available resource to executeall simultaneously Furthermore this overall hardware resource isshared with the operating system on which the virtual machine isdeployed This also impacts negatively in the average time

Figure 6 shows the results obtained Analyzing the extreme caseon the second test there are 20 messages being sent from 1024data sources simultaneously This is a total of 20480 messagesThese numbers of messages were processed in 200572 ms In thecase of transmitting messages from a single producer we reachedat the number of 106650 messages processed in 784ms Due toresource limitation the increasing of threads impacts negativelyon the performance of messages sent

Tim

e (

ms)

Number of Threads

2 Messages

20 Messages

Figure 6 Representation of receiving average time fromtread

4 RELATEDWORKSThere is a growing concern on which architecture must be used forapplications that process large volumes of data in real time Recent

SBSIrsquo2018 Junho 2018 Caxias do Sul Rio Grande do Sul Brazil M Cadaviz et al

literature has explored this challenge in the last years Howeverlittle has been done to clarify about how different architecturalapproaches might be brought together to address particular facetsof DIRT information systems

By doing so this section explores three architectures for DIRTapplications applied to different domains Section 41 describes anarchitecture for scientific applications Section 42 describes anarchitecture for weather applications Section 43 presents an ar-chitecture for application control Finally Section 44 presents acomparative analysis between the Doric and the architectures dis-cussed in order to identify the similarities and differences betweenthem as well as the research gaps the proposed architecture fulfills

41 Large-Scale and Real Time Processing forScientific Applications

Based on dynamic data-driven application systems (DDDAS) con-cept Cao and Li propose in [5] a framework to optimize the perfor-mance of data processing where data is generated in high volumeabout 10 MB per second almost 1 TB per day Another challengeis the issue of transportation since the equipment is spread overthousands of miles away In this context performance is a criticalfactor since the processing of the collected information must becompleted in 30 minutes

The architecture proposed is based on four layers (1) appli-cation allows the input of parameters and publishes the resultsin a user-friendly way (2) data processing performs additionaldata processing based on monitoring outputs (3) data monitor-ing dynamically accompanies information on a central monitorand (4) instrumentation receives elements collected from one ormore measurement systems and submits them to treatment stepssubsequently storing the assembly in the instrumentation layerAccording to [5] the frameworkrsquos strengths are in data reductionalgorithms which can reduce a block of data from 9063 MB to 29MB thus reducing possible bottlenecks in the transport area

The authors believe that in the near future the data scaleswill grow exponentially In this scenario although the concept ofDDDAS is directed to simulation applications the framework willhave the possibility of revealing its potential for several scientificapplications

42 Real-time Processing for Weather StationsIn China it is common to use autonomous weather stations withmore than 30000 units [16] Data-generating devices and controlterminals are commonly located in different locations such as fieldsislands reservations among others

Using the concept of clientserver all collected data is sent to adata server simultaneously In this system where there is a largedata transmission network that is transmitted with a small amountof information and in large scale making use of sequential transmis-sions it is necessary to implement an efficient real-time processingcapable of receiving and treating the data synchronously

Guangsheng and colleagues in [16] presents a detailed discussionon the design of a system for receiving and processing data fromautonomous meteorological stations in Guangdong province Eachobservation station produces a group of data every 5 minutes ondays of good weather or 1 minute in atypical situations Thus the

model must be auditable and extensible according to the technicalor operational needs of the meteorological stations in addition toavoiding the congestion in the observation data uploaded at criticalmoments For this the premise is that the entire processing anddata storage cycle must be completed within 1 minute or less

This system was deployed in more than 1700 weather stationsfor over a year Running with stability efficiency and reliability Allstations send their packets to the control center at the same timeevery minute The time for receiving processing and uploadingthe response for each cycle was less than 1 minute and the hard-ware requirements are low reducing maintenance and reducingthe complexity of the systems used

43 Real-Time Scheduling for Large ScaleApplication Control

In [15] Waknis and Sztipanovits argue that real-time applicationrequirements can be satisfied by a scheduling scheme appropriateto a model-based programming environment They developed anintelligent process control system which is a model-based systemand acts as a real-time supervisor of large-scale industrial processesable to control various activities monitor different operations andmaking use of the sensor information to diagnose a system failure[15]

The schema is constructed in three layers (1) model buildingtools layer which allows the qualitative and quantitative repre-sentation of the information (2) model interpretation layer wherethe information is stored in a descriptive way which is used formonitoring control and diagnosis besides having the capacity totransform the models into executable computational structures and(3) the environment execution layer which is implemented with themultigraph architecture using directional graphs The nodes of thegraph represent the computational blocks and their connectionsrepresent the flow of data between them Each block has a programsegment that is staggered according to demand and data flow

One notable feature that is offered by this framework is theability to dynamically reconfigure This is not limited only to ad-justments in system parameters allowing for architectural changesat runtime by the inclusion and removal of nodes and connectionsThe structure also presents interfaces for receiving informationfrom the external environment and integration with its consumersin addition it also has an interface implemented by each noderesponsible for the parallel and scalable processing of the inputdata and a similar function that processes the output data

44 Comparative AnalysisThis section presents a comparative analysis of the related worksThis comparison serves as a basis to identify the similarities anddifferences between the Doric and the selected works The com-parison criteria seek to reveal specific characteristics of the relatedworks which can be contrasted with the proposed architectureThe comparison criteria are discussed as follows

bull Data source Studies that seek to offer expandable archi-tectures allowing to support several data sources Modernarchitectures for information systems need to give their pro-tection to multiple types and formats of data

DORIC An Architecture for Data-intensive Real-time Applications SBSIrsquo2018 Junho 2018 Caxias do Sul Rio Grande do Sul Brazil

Table 4 Comparative Analysis

ModularizedData Source Publish and Subscrib Information Processing Analysis Storage Consumers

Expandable Expandable Scalable Expandable Scalable Expandable scalable Expandable Scalable Expandable

Real-time and Large-scale

processing for Scientific Applicationssim - sim + sim + sim + sim + -

Real-time and Large-scale

processing for Weather Stationssim - sim + sim + sim + sim sim +

Real-time Scheduling For

Large-scale Control Applicationssim sim + + sim + sim + sim + +

Proposed Architecture + + + + + + + + + + +

Legend ldquo+rdquo Supports ldquosimrdquo Partially Supports ldquo-rdquo Does not Support

bull PublishSubscribe Studies that are based on the publish-subscribe pattern

bull Information processing Studies that are scalable and ex-pandable for information processing Processing informationis fundamental to any architecture for the development ofmodern information systems

bull Analysis Studies that provide support for data analysisthrough machine learning techniques

bull Storage Studies that are extensible and scalable with regardto the type and volume of data storage

bull Consumers Studies that are flexible to support differenttypes of customers

Table 4 presents the comparative analysis considering thesecriteria It is observed that only the Doric fully meets the definedcriteria highlighting the contribution and the differential of thiswork

It is possible to verify that in spite of the great scalability of theevaluated projects there is a deficiency in the issue of expansionand reuse of the structure for this reason the proposed architec-ture aims not only to satisfy the demands presented but also toprovide a structure divided into modules and reusable making useof microservices and emphasizing the separation between its pro-cessing stages Such an implementation favors the construction ofexpandable reusable and easily understood structures contributingboth to existing systems and to those that will still be built

5 CONCLUSIONThe adoption of DIRT applications increased in recent years How-ever there is a lack of standardization and modularized architec-tures This implies that a specific infrastructure was needed to beprojected to attend a different demand of the applications In otherwords a different construction was made despite they have thesame essential purpose This purpose is to serve as a large-scaleand real-time architecture

To resolve this problem this work proposed the Doric a flexibleand modular architecture for data-intensive and real-time systemsThis architecture was built to accommodate the state-of-the-arttechnologies related to this purpose A prototype was implementedto conduct an empirical evaluation This evaluation demonstratedthat the architecture could process a large volume of data in a shortperiod of time This was possible even with a hardware limitationLastly the future work consists of deploying the architecture in anenvironment with improved hardware capability and evaluate thearchitecture adaptability in more scenarios

6 ACKNOWLEDGMENTSThank you to Unisinos for providing the spaces and resources toconduct this research

REFERENCES[1] Ricardo Alexandre Afonso WM Da Silva GHRP Tomas Kiev Gama Alezy

Oliveira Alexandre Alvaro and Vinicius Cardoso Garcia 2013 Br-SCMM Mod-elo Brasileiro de Maturidade para Cidades Inteligentes Simpoacutesio Brasileiro DeSistemas De Informaccedilatildeo (2013)

[2] Apache 2018 Apache Spark httpssparkapacheorg [online accessed onjanuary 2018]

[3] Apache 2018 Apache Zookeper httpszookeeperapacheorg [onlineaccessed on january 2018]

[4] Luciana Regina Bencke Anderson Luiz Fernandes Perez and Osvaldo da Costa Ar-mendaris 2017 Rodovias Inteligentes uma visatildeo geral sobre as tecnologias em-pregadas no Brasil e no mundo iSys-Revista Brasileira de Sistemas de Informaccedilatildeo10 4 (2017) 80ndash102

[5] Junwei Cao and Junwei Li 2011 Large-scale real-time data-driven scientificapplications In Networking and Distributed Computing (ICNDC) 2011 SecondInternational Conference on IEEE 116ndash121

[6] Dave Evans 2012 The Internet of Things how the next evolution of the internet ischanging everything (april 2011) White Paper by Cisco Internet Business SolutionsGroup (IBSG) (2012)

[7] Apache Kafka 2017 Apache Kafka a distributed system platform Introductionhttpkafkaapacheorgintrohtml [online accessed on january 2018]

[8] Juacutelio Menezes Jr and Cristine Gusmatildeo 2014 InteliMEDndashProposta de Sistema deApoio ao Diagnoacutestico Meacutedico para Dispositivos Moacuteveis iSys-Revista Brasileirade Sistemas de Informaccedilatildeo 6 1 (2014) 44ndash61

[9] Mongo 2018 MongoDB httpswwwmongodbcom [online accessed onjanuary 2018]

[10] Linda Northrop Peter Feiler Richard P Gabriel John Goodenough Rick LingerTom Longstaff Rick Kazman Mark Klein Douglas Schmidt Kevin Sullivan et al2006 Ultra-large-scale systems The software challenge of the future Technical Re-port CARNEGIE-MELLON UNIV PITTSBURGH PA SOFTWARE ENGINEERINGINST

[11] Oracle 2018 Java 8 httpwwworaclecomtechnetworkptjavajavasedownloadsjre8-downloads-2133155html [online accessed on january 2018]

[12] Dan Puiu Payam Barnaghi Ralf Toenjes Daniel Kuumlmper Muhammad IntizarAli Alessandra Mileo Josiane Xavier Parreira Marten Fischer Sefki KolozaliNazli Farajidavar et al 2016 Citypulse Large scale data analytics framework forsmart cities IEEE Access 4 (2016) 1086ndash1108

[13] The Statistics Portal STATISTA 2017 Internet of Things (IoT) number of con-nected devices worldwide from 2015 to 2025 (in billions) httpswwwstatistacomstatistics471264iot-number-of-connected-devices-worldwide

[14] STRATA 2015 Large-Scale Real-Time Applications [Strata + Hadoop WorldConferences]

[15] Prashant Waknis and Janos Sztipanovits 1993 Hard real-time scheduling forlarge scale process control applications In Proceedings of the IEEE Workshop onReal-Time Applications IEEE 71ndash75

[16] Guangsheng Wu Zhenlang Ao Jianyong Li and Qinqiang Zhou 2011 A Real-time Receiving and Distributed Processing System for Large-scale Burst Data In2011 Second International Conference on Networking and Distributed Computing(ICNDC) IEEE 111ndash115

DORIC An Architecture for Data-intensive Real-time Applications SBSIrsquo2018 Junho 2018 Caxias do Sul Rio Grande do Sul Brazil

22 Architectural projectThe proposed architecture was built in two steps In the first step weidentified the requirement an architecture must have The secondstep the technologies that implement each architecture module isdescribed These technologies are defined according to the definedrequirements in Step 1 These steps are described below

221 Requirements of the Proposed Architecture The require-ments were elaborated based on suggested features on the articlesand research reports We elaborated three non-functional require-ments (NFR) These requirements are described below

bull NFR01 support the process and reception of large-scaleinformation with integrity The architecture must sup-port and guarantee the correctness of informationrsquos that aretransferred in high volume

bull NFR02 able to return real-time responsesThe analyzedinformationrsquos must be provided in real-time to the consumersby the system

bull NFR03 the architecture must be extensible The archi-tecture must be well encapsulated and provide extensiblepoints This enables reuse of modules and the addition ofnew ones

222 Architectural components This section describes the Doriccomponents as well as suggests some technologies to implementthem Figure 3 shows how the components of the architecturewere organized Table 1 suggests some technologies to implementthe Doric components The chosen technologies seek to meet therequirements previously mentioned Note that each dimensionpresent in Table 1 corresponds to a Doric component which isimplemented by using technologies for building real-time datapipelines and streaming applications The following each compo-nent is briefly described

bull Data source The data source is not present in the architec-ture because they are external components This architectureis adapted to receive information from any data source

bull Publishersubscriber We choose the Apache Kafka forthis component Kafka is largely applied in the industry andis open source Kafka is suitable for two kinds of applica-tions first obtaining data in real time frommany applicationand systems safely Second systems that convert or reactto a certain data flux in real-time Kafka also promotes theabstraction of the data record from a determined topic Top-ics are categories for which the records are published Thetopics in Kafka are always multi-subscriber ie a topic canhave n subscribers to access their data The version ofApache Kafka used in this architecture is 01020 In addi-tion in this module the Apache ZooKeeper are implementedto cooperate with kafka [7] Apache ZooKeeper [3] providesseveral functionalities for distributed applications such asdistributed configuration management information coor-dination and locks The deployed version was ZooKeeper349 Both Kafka and ZooKeeper use Java 8 [11] release 131of the Java Runtime Environment (JRE) JRE consists on aJava virtual machine for running applications

bull Information Processering The Apache Spark [2] is re-sponsible to manage the information processing This tech-nology is powerful and open source

bull Data analysisWe implemented an application using Javato manage the data analysis This is because there is nosuggestion and consolidate techniques to conduct the dataanalysis

bull Storage of information These components implement theMongoDB [9] database This database is the state-of-art tech-nology for storing massive data and non-structured infor-mationrsquos We used the MongoDB version 344

bull Consumers Consumers are not defined in this architecturebecause they are external entities Any external entity canconsume the data provided by the architecture

Data Source PublisherSubscriber Consumers

Analysis

Informationrsquos Processing Storage

Figure 2 Implemented architecture

3 ASPECTS OF IMPLEMENTATION ANDEVALUATION

This section presents an empirical evaluation of the proposed archi-tecture The architecture is evaluated through realistic scenariosin which a large amount of data is generated in real time froma portable electroencephalogram (EEG) Section 31 describes theimplementation aspects of the proposed architecture Section 32 de-scribes the adopted evaluation method Finally Section 33 presentsthe results and discussion about the performance of the imple-mented system that adopted the proposed architecture

SBSIrsquo2018 Junho 2018 Caxias do Sul Rio Grande do Sul Brazil M Cadaviz et al

APACHESPARK

DATA SOURCES

MONGODB

ANALYSISAPPLICATION

CONSUMERSAPACHEKAFKA

Legend

A

Component

Provided Interface

Required Interface

Aspectual Connector

Figure 3 Component diagram

31 Implementation AspectsThe publishing and receiving stage implemented with ApacheKafka controls the entire data flux acting as the central nervoussystem of the structure but the available producer and consumerin the prototype work via prompt through the use of batch filesIn addition activities such as starting the Kafka server creating anew information topic or even listing messages also are managedthrough the prompt command lines

The prototype of the Doric architecture was constructed usingJava language Maven3 were used to manage the construction anddependencies of the project User interface was created using Swingjava library using Netbeans IDE The prototype was built to sim-ulate simple interactions with the Kafka cluster the sending ofinformation to the servers and consumers obtaining the analyzedinformation

Figure 4 shows an overview of the prototype It is possible toverify the existence of areas representing the dimensions of DIRTapplications Each of these areas implements some functions relatedto the respective area The list bellow describes these functionsaccording to each dimension

The DORIC components implemented in the prototype were(1) Publish and subscribe This component represented in

Figure 4(1) is related to the functions responsible for (a) changetopics it allows to switch between the existing topics inKafka As previously mentioned a topic is a category inwhich the input data are recorded (b) create topics it allowsthe creation of a new topic

(2) Data source We have used the EPOC Emotiv 4 as datasource This component represented in Figure 4(2) is re-sponsible for the functions related to manage of data sourcesThese functions enable (a) create new message generates asingle and unique message in the selected topic (b) importmessages performs the import of plain text messages fromthe file to the selected topic

3Maven httpsmavenapacheorg4EPOC Emotiv httpswwwemotivcomepoc

Figure 4 An overview of the prototype

(3) Consumer responsible to accommodate functions to man-age Consumers such as (a) last messages displays the lastmessages sent to the selected topic

32 Evaluation methodThe purpose of architecture evaluation is to present some results ob-tained after submitting information to the publishing and receivingcomponent Then the objective is to demonstrate the architectureto handle large-scale and real-time information The architecturewas built on a virtual machine with the following requirements

bull Windows 10 Operating Systembull Intel i7 (27 GHz) CPU with 4 coresbull 4 GB RAM DDR4 2133 MHz of memorybull 30 GB (5400 RPM) of Hard Disk

The scenario where this architecture is evaluated consists inprocessing and analyzing data from a health-care system Thissystem collects mental indicators data from a portable EEG Thenthe test is divided in two steps

The first assessment considers the architecturersquos ability to readlarge files For this several files with information of a wirelesselectroencephalogram in a csv file were submitted to the infra-structure The largest of them has 50 MB of information totalingmore than 50 thousand of lines records and around of 51 millioncharacters The second step tests the concept of large scale For thisseveral threads are being created to process the associated data ofthe producer to the system publish information simultaneously

33 ResultsMeasurements of time used in the experiment were carried onthe upper layers of the prototype thus creating a more realisticenvironment for the end user However this method of measuringjust considering the time spent in the transmission of informationsuch as processing time Several tests of single message submissionwere performed using the prototype which obtained an averageresponse time of 318 ms We did not observe a correlation betweenresponse time and the size of the message the type of contentneither the frequency of the topic structure

DORIC An Architecture for Data-intensive Real-time Applications SBSIrsquo2018 Junho 2018 Caxias do Sul Rio Grande do Sul Brazil

The evaluation of capacity of real-time response was performedwith five different file sizes Table 2 presents the obtained resultsThe import process was repeated fifty times for each file to obtainreliable values The size of the file influences in the processingtime The time increased when larger files were processed Wealso observed the time between sending the last record and thenotification of process completion In this case the average time alsoincreased according to the size of the file Specifically the averagetime increases significantly for 30 MB files Figure 5 illustrates theresults contained on Table2

Table 2 Test results of data provided from direct files

File

Size (MB)

Dispatch Time

Averarge (ms)

Receiving Time

Averarge (ms)

10 34366 12743

20 39682 140025

30 42562 201919

40 90672 222783

50 162845 22626

Next we also evaluated the architecturersquos capability to handlelarge-scale data For this we executed multiple threads to simulatethe role of data sources These threads simultaneously executedand provided the same topic Each thread creates a producer andgenerates a certain number of messages We monitored two aver-age times in this case (1) the average time spent on sending themessages to the server and (2) the average time the infrastructuretakes to signalize that it received the information Figure 5 depictsthe results Table 3 specifies the results obtained

File Size (MB)

Tim

e (

ms)

Figure 5 Receiving time average

The results evidence a gradual increase in the receiving timewhen there are more threads in execution The same occurs in themoment that the number of messages increases This behavior wasexpected because the tests were conducted on a virtual machineThere is a limitation regarding the hardware capability eg there isonly one CPU This implies that a larger number of threads causes

Table 3 Test results of data provided from threads

ThreadsMenssages

Number

Dispatch Time

Averarge (ms)

Receiving Time

Averarge (ms)

2 2 0 45083

4 2 0 52482

8 2 0 67111

16 2 0 68427

32 2 32 75904

1024 2 2592 182025

2 20 0 25002

4 20 0 21881

8 20 0 33174

16 20 0 97486

32 20 632 102732

1024 20 31448 200572

congestion on the processing queue There is more processes wait-ing on the queue because there is no available resource to executeall simultaneously Furthermore this overall hardware resource isshared with the operating system on which the virtual machine isdeployed This also impacts negatively in the average time

Figure 6 shows the results obtained Analyzing the extreme caseon the second test there are 20 messages being sent from 1024data sources simultaneously This is a total of 20480 messagesThese numbers of messages were processed in 200572 ms In thecase of transmitting messages from a single producer we reachedat the number of 106650 messages processed in 784ms Due toresource limitation the increasing of threads impacts negativelyon the performance of messages sent

Tim

e (

ms)

Number of Threads

2 Messages

20 Messages

Figure 6 Representation of receiving average time fromtread

4 RELATEDWORKSThere is a growing concern on which architecture must be used forapplications that process large volumes of data in real time Recent

SBSIrsquo2018 Junho 2018 Caxias do Sul Rio Grande do Sul Brazil M Cadaviz et al

literature has explored this challenge in the last years Howeverlittle has been done to clarify about how different architecturalapproaches might be brought together to address particular facetsof DIRT information systems

By doing so this section explores three architectures for DIRTapplications applied to different domains Section 41 describes anarchitecture for scientific applications Section 42 describes anarchitecture for weather applications Section 43 presents an ar-chitecture for application control Finally Section 44 presents acomparative analysis between the Doric and the architectures dis-cussed in order to identify the similarities and differences betweenthem as well as the research gaps the proposed architecture fulfills

41 Large-Scale and Real Time Processing forScientific Applications

Based on dynamic data-driven application systems (DDDAS) con-cept Cao and Li propose in [5] a framework to optimize the perfor-mance of data processing where data is generated in high volumeabout 10 MB per second almost 1 TB per day Another challengeis the issue of transportation since the equipment is spread overthousands of miles away In this context performance is a criticalfactor since the processing of the collected information must becompleted in 30 minutes

The architecture proposed is based on four layers (1) appli-cation allows the input of parameters and publishes the resultsin a user-friendly way (2) data processing performs additionaldata processing based on monitoring outputs (3) data monitor-ing dynamically accompanies information on a central monitorand (4) instrumentation receives elements collected from one ormore measurement systems and submits them to treatment stepssubsequently storing the assembly in the instrumentation layerAccording to [5] the frameworkrsquos strengths are in data reductionalgorithms which can reduce a block of data from 9063 MB to 29MB thus reducing possible bottlenecks in the transport area

The authors believe that in the near future the data scaleswill grow exponentially In this scenario although the concept ofDDDAS is directed to simulation applications the framework willhave the possibility of revealing its potential for several scientificapplications

42 Real-time Processing for Weather StationsIn China it is common to use autonomous weather stations withmore than 30000 units [16] Data-generating devices and controlterminals are commonly located in different locations such as fieldsislands reservations among others

Using the concept of clientserver all collected data is sent to adata server simultaneously In this system where there is a largedata transmission network that is transmitted with a small amountof information and in large scale making use of sequential transmis-sions it is necessary to implement an efficient real-time processingcapable of receiving and treating the data synchronously

Guangsheng and colleagues in [16] presents a detailed discussionon the design of a system for receiving and processing data fromautonomous meteorological stations in Guangdong province Eachobservation station produces a group of data every 5 minutes ondays of good weather or 1 minute in atypical situations Thus the

model must be auditable and extensible according to the technicalor operational needs of the meteorological stations in addition toavoiding the congestion in the observation data uploaded at criticalmoments For this the premise is that the entire processing anddata storage cycle must be completed within 1 minute or less

This system was deployed in more than 1700 weather stationsfor over a year Running with stability efficiency and reliability Allstations send their packets to the control center at the same timeevery minute The time for receiving processing and uploadingthe response for each cycle was less than 1 minute and the hard-ware requirements are low reducing maintenance and reducingthe complexity of the systems used

43 Real-Time Scheduling for Large ScaleApplication Control

In [15] Waknis and Sztipanovits argue that real-time applicationrequirements can be satisfied by a scheduling scheme appropriateto a model-based programming environment They developed anintelligent process control system which is a model-based systemand acts as a real-time supervisor of large-scale industrial processesable to control various activities monitor different operations andmaking use of the sensor information to diagnose a system failure[15]

The schema is constructed in three layers (1) model buildingtools layer which allows the qualitative and quantitative repre-sentation of the information (2) model interpretation layer wherethe information is stored in a descriptive way which is used formonitoring control and diagnosis besides having the capacity totransform the models into executable computational structures and(3) the environment execution layer which is implemented with themultigraph architecture using directional graphs The nodes of thegraph represent the computational blocks and their connectionsrepresent the flow of data between them Each block has a programsegment that is staggered according to demand and data flow

One notable feature that is offered by this framework is theability to dynamically reconfigure This is not limited only to ad-justments in system parameters allowing for architectural changesat runtime by the inclusion and removal of nodes and connectionsThe structure also presents interfaces for receiving informationfrom the external environment and integration with its consumersin addition it also has an interface implemented by each noderesponsible for the parallel and scalable processing of the inputdata and a similar function that processes the output data

44 Comparative AnalysisThis section presents a comparative analysis of the related worksThis comparison serves as a basis to identify the similarities anddifferences between the Doric and the selected works The com-parison criteria seek to reveal specific characteristics of the relatedworks which can be contrasted with the proposed architectureThe comparison criteria are discussed as follows

bull Data source Studies that seek to offer expandable archi-tectures allowing to support several data sources Modernarchitectures for information systems need to give their pro-tection to multiple types and formats of data

DORIC An Architecture for Data-intensive Real-time Applications SBSIrsquo2018 Junho 2018 Caxias do Sul Rio Grande do Sul Brazil

Table 4 Comparative Analysis

ModularizedData Source Publish and Subscrib Information Processing Analysis Storage Consumers

Expandable Expandable Scalable Expandable Scalable Expandable scalable Expandable Scalable Expandable

Real-time and Large-scale

processing for Scientific Applicationssim - sim + sim + sim + sim + -

Real-time and Large-scale

processing for Weather Stationssim - sim + sim + sim + sim sim +

Real-time Scheduling For

Large-scale Control Applicationssim sim + + sim + sim + sim + +

Proposed Architecture + + + + + + + + + + +

Legend ldquo+rdquo Supports ldquosimrdquo Partially Supports ldquo-rdquo Does not Support

bull PublishSubscribe Studies that are based on the publish-subscribe pattern

bull Information processing Studies that are scalable and ex-pandable for information processing Processing informationis fundamental to any architecture for the development ofmodern information systems

bull Analysis Studies that provide support for data analysisthrough machine learning techniques

bull Storage Studies that are extensible and scalable with regardto the type and volume of data storage

bull Consumers Studies that are flexible to support differenttypes of customers

Table 4 presents the comparative analysis considering thesecriteria It is observed that only the Doric fully meets the definedcriteria highlighting the contribution and the differential of thiswork

It is possible to verify that in spite of the great scalability of theevaluated projects there is a deficiency in the issue of expansionand reuse of the structure for this reason the proposed architec-ture aims not only to satisfy the demands presented but also toprovide a structure divided into modules and reusable making useof microservices and emphasizing the separation between its pro-cessing stages Such an implementation favors the construction ofexpandable reusable and easily understood structures contributingboth to existing systems and to those that will still be built

5 CONCLUSIONThe adoption of DIRT applications increased in recent years How-ever there is a lack of standardization and modularized architec-tures This implies that a specific infrastructure was needed to beprojected to attend a different demand of the applications In otherwords a different construction was made despite they have thesame essential purpose This purpose is to serve as a large-scaleand real-time architecture

To resolve this problem this work proposed the Doric a flexibleand modular architecture for data-intensive and real-time systemsThis architecture was built to accommodate the state-of-the-arttechnologies related to this purpose A prototype was implementedto conduct an empirical evaluation This evaluation demonstratedthat the architecture could process a large volume of data in a shortperiod of time This was possible even with a hardware limitationLastly the future work consists of deploying the architecture in anenvironment with improved hardware capability and evaluate thearchitecture adaptability in more scenarios

6 ACKNOWLEDGMENTSThank you to Unisinos for providing the spaces and resources toconduct this research

REFERENCES[1] Ricardo Alexandre Afonso WM Da Silva GHRP Tomas Kiev Gama Alezy

Oliveira Alexandre Alvaro and Vinicius Cardoso Garcia 2013 Br-SCMM Mod-elo Brasileiro de Maturidade para Cidades Inteligentes Simpoacutesio Brasileiro DeSistemas De Informaccedilatildeo (2013)

[2] Apache 2018 Apache Spark httpssparkapacheorg [online accessed onjanuary 2018]

[3] Apache 2018 Apache Zookeper httpszookeeperapacheorg [onlineaccessed on january 2018]

[4] Luciana Regina Bencke Anderson Luiz Fernandes Perez and Osvaldo da Costa Ar-mendaris 2017 Rodovias Inteligentes uma visatildeo geral sobre as tecnologias em-pregadas no Brasil e no mundo iSys-Revista Brasileira de Sistemas de Informaccedilatildeo10 4 (2017) 80ndash102

[5] Junwei Cao and Junwei Li 2011 Large-scale real-time data-driven scientificapplications In Networking and Distributed Computing (ICNDC) 2011 SecondInternational Conference on IEEE 116ndash121

[6] Dave Evans 2012 The Internet of Things how the next evolution of the internet ischanging everything (april 2011) White Paper by Cisco Internet Business SolutionsGroup (IBSG) (2012)

[7] Apache Kafka 2017 Apache Kafka a distributed system platform Introductionhttpkafkaapacheorgintrohtml [online accessed on january 2018]

[8] Juacutelio Menezes Jr and Cristine Gusmatildeo 2014 InteliMEDndashProposta de Sistema deApoio ao Diagnoacutestico Meacutedico para Dispositivos Moacuteveis iSys-Revista Brasileirade Sistemas de Informaccedilatildeo 6 1 (2014) 44ndash61

[9] Mongo 2018 MongoDB httpswwwmongodbcom [online accessed onjanuary 2018]

[10] Linda Northrop Peter Feiler Richard P Gabriel John Goodenough Rick LingerTom Longstaff Rick Kazman Mark Klein Douglas Schmidt Kevin Sullivan et al2006 Ultra-large-scale systems The software challenge of the future Technical Re-port CARNEGIE-MELLON UNIV PITTSBURGH PA SOFTWARE ENGINEERINGINST

[11] Oracle 2018 Java 8 httpwwworaclecomtechnetworkptjavajavasedownloadsjre8-downloads-2133155html [online accessed on january 2018]

[12] Dan Puiu Payam Barnaghi Ralf Toenjes Daniel Kuumlmper Muhammad IntizarAli Alessandra Mileo Josiane Xavier Parreira Marten Fischer Sefki KolozaliNazli Farajidavar et al 2016 Citypulse Large scale data analytics framework forsmart cities IEEE Access 4 (2016) 1086ndash1108

[13] The Statistics Portal STATISTA 2017 Internet of Things (IoT) number of con-nected devices worldwide from 2015 to 2025 (in billions) httpswwwstatistacomstatistics471264iot-number-of-connected-devices-worldwide

[14] STRATA 2015 Large-Scale Real-Time Applications [Strata + Hadoop WorldConferences]

[15] Prashant Waknis and Janos Sztipanovits 1993 Hard real-time scheduling forlarge scale process control applications In Proceedings of the IEEE Workshop onReal-Time Applications IEEE 71ndash75

[16] Guangsheng Wu Zhenlang Ao Jianyong Li and Qinqiang Zhou 2011 A Real-time Receiving and Distributed Processing System for Large-scale Burst Data In2011 Second International Conference on Networking and Distributed Computing(ICNDC) IEEE 111ndash115

SBSIrsquo2018 Junho 2018 Caxias do Sul Rio Grande do Sul Brazil M Cadaviz et al

APACHESPARK

DATA SOURCES

MONGODB

ANALYSISAPPLICATION

CONSUMERSAPACHEKAFKA

Legend

A

Component

Provided Interface

Required Interface

Aspectual Connector

Figure 3 Component diagram

31 Implementation AspectsThe publishing and receiving stage implemented with ApacheKafka controls the entire data flux acting as the central nervoussystem of the structure but the available producer and consumerin the prototype work via prompt through the use of batch filesIn addition activities such as starting the Kafka server creating anew information topic or even listing messages also are managedthrough the prompt command lines

The prototype of the Doric architecture was constructed usingJava language Maven3 were used to manage the construction anddependencies of the project User interface was created using Swingjava library using Netbeans IDE The prototype was built to sim-ulate simple interactions with the Kafka cluster the sending ofinformation to the servers and consumers obtaining the analyzedinformation

Figure 4 shows an overview of the prototype It is possible toverify the existence of areas representing the dimensions of DIRTapplications Each of these areas implements some functions relatedto the respective area The list bellow describes these functionsaccording to each dimension

The DORIC components implemented in the prototype were(1) Publish and subscribe This component represented in

Figure 4(1) is related to the functions responsible for (a) changetopics it allows to switch between the existing topics inKafka As previously mentioned a topic is a category inwhich the input data are recorded (b) create topics it allowsthe creation of a new topic

(2) Data source We have used the EPOC Emotiv 4 as datasource This component represented in Figure 4(2) is re-sponsible for the functions related to manage of data sourcesThese functions enable (a) create new message generates asingle and unique message in the selected topic (b) importmessages performs the import of plain text messages fromthe file to the selected topic

3Maven httpsmavenapacheorg4EPOC Emotiv httpswwwemotivcomepoc

Figure 4 An overview of the prototype

(3) Consumer responsible to accommodate functions to man-age Consumers such as (a) last messages displays the lastmessages sent to the selected topic

32 Evaluation methodThe purpose of architecture evaluation is to present some results ob-tained after submitting information to the publishing and receivingcomponent Then the objective is to demonstrate the architectureto handle large-scale and real-time information The architecturewas built on a virtual machine with the following requirements

bull Windows 10 Operating Systembull Intel i7 (27 GHz) CPU with 4 coresbull 4 GB RAM DDR4 2133 MHz of memorybull 30 GB (5400 RPM) of Hard Disk

The scenario where this architecture is evaluated consists inprocessing and analyzing data from a health-care system Thissystem collects mental indicators data from a portable EEG Thenthe test is divided in two steps

The first assessment considers the architecturersquos ability to readlarge files For this several files with information of a wirelesselectroencephalogram in a csv file were submitted to the infra-structure The largest of them has 50 MB of information totalingmore than 50 thousand of lines records and around of 51 millioncharacters The second step tests the concept of large scale For thisseveral threads are being created to process the associated data ofthe producer to the system publish information simultaneously

33 ResultsMeasurements of time used in the experiment were carried onthe upper layers of the prototype thus creating a more realisticenvironment for the end user However this method of measuringjust considering the time spent in the transmission of informationsuch as processing time Several tests of single message submissionwere performed using the prototype which obtained an averageresponse time of 318 ms We did not observe a correlation betweenresponse time and the size of the message the type of contentneither the frequency of the topic structure

DORIC An Architecture for Data-intensive Real-time Applications SBSIrsquo2018 Junho 2018 Caxias do Sul Rio Grande do Sul Brazil

The evaluation of capacity of real-time response was performedwith five different file sizes Table 2 presents the obtained resultsThe import process was repeated fifty times for each file to obtainreliable values The size of the file influences in the processingtime The time increased when larger files were processed Wealso observed the time between sending the last record and thenotification of process completion In this case the average time alsoincreased according to the size of the file Specifically the averagetime increases significantly for 30 MB files Figure 5 illustrates theresults contained on Table2

Table 2 Test results of data provided from direct files

File

Size (MB)

Dispatch Time

Averarge (ms)

Receiving Time

Averarge (ms)

10 34366 12743

20 39682 140025

30 42562 201919

40 90672 222783

50 162845 22626

Next we also evaluated the architecturersquos capability to handlelarge-scale data For this we executed multiple threads to simulatethe role of data sources These threads simultaneously executedand provided the same topic Each thread creates a producer andgenerates a certain number of messages We monitored two aver-age times in this case (1) the average time spent on sending themessages to the server and (2) the average time the infrastructuretakes to signalize that it received the information Figure 5 depictsthe results Table 3 specifies the results obtained

File Size (MB)

Tim

e (

ms)

Figure 5 Receiving time average

The results evidence a gradual increase in the receiving timewhen there are more threads in execution The same occurs in themoment that the number of messages increases This behavior wasexpected because the tests were conducted on a virtual machineThere is a limitation regarding the hardware capability eg there isonly one CPU This implies that a larger number of threads causes

Table 3 Test results of data provided from threads

ThreadsMenssages

Number

Dispatch Time

Averarge (ms)

Receiving Time

Averarge (ms)

2 2 0 45083

4 2 0 52482

8 2 0 67111

16 2 0 68427

32 2 32 75904

1024 2 2592 182025

2 20 0 25002

4 20 0 21881

8 20 0 33174

16 20 0 97486

32 20 632 102732

1024 20 31448 200572

congestion on the processing queue There is more processes wait-ing on the queue because there is no available resource to executeall simultaneously Furthermore this overall hardware resource isshared with the operating system on which the virtual machine isdeployed This also impacts negatively in the average time

Figure 6 shows the results obtained Analyzing the extreme caseon the second test there are 20 messages being sent from 1024data sources simultaneously This is a total of 20480 messagesThese numbers of messages were processed in 200572 ms In thecase of transmitting messages from a single producer we reachedat the number of 106650 messages processed in 784ms Due toresource limitation the increasing of threads impacts negativelyon the performance of messages sent

Tim

e (

ms)

Number of Threads

2 Messages

20 Messages

Figure 6 Representation of receiving average time fromtread

4 RELATEDWORKSThere is a growing concern on which architecture must be used forapplications that process large volumes of data in real time Recent

SBSIrsquo2018 Junho 2018 Caxias do Sul Rio Grande do Sul Brazil M Cadaviz et al

literature has explored this challenge in the last years Howeverlittle has been done to clarify about how different architecturalapproaches might be brought together to address particular facetsof DIRT information systems

By doing so this section explores three architectures for DIRTapplications applied to different domains Section 41 describes anarchitecture for scientific applications Section 42 describes anarchitecture for weather applications Section 43 presents an ar-chitecture for application control Finally Section 44 presents acomparative analysis between the Doric and the architectures dis-cussed in order to identify the similarities and differences betweenthem as well as the research gaps the proposed architecture fulfills

41 Large-Scale and Real Time Processing forScientific Applications

Based on dynamic data-driven application systems (DDDAS) con-cept Cao and Li propose in [5] a framework to optimize the perfor-mance of data processing where data is generated in high volumeabout 10 MB per second almost 1 TB per day Another challengeis the issue of transportation since the equipment is spread overthousands of miles away In this context performance is a criticalfactor since the processing of the collected information must becompleted in 30 minutes

The architecture proposed is based on four layers (1) appli-cation allows the input of parameters and publishes the resultsin a user-friendly way (2) data processing performs additionaldata processing based on monitoring outputs (3) data monitor-ing dynamically accompanies information on a central monitorand (4) instrumentation receives elements collected from one ormore measurement systems and submits them to treatment stepssubsequently storing the assembly in the instrumentation layerAccording to [5] the frameworkrsquos strengths are in data reductionalgorithms which can reduce a block of data from 9063 MB to 29MB thus reducing possible bottlenecks in the transport area

The authors believe that in the near future the data scaleswill grow exponentially In this scenario although the concept ofDDDAS is directed to simulation applications the framework willhave the possibility of revealing its potential for several scientificapplications

42 Real-time Processing for Weather StationsIn China it is common to use autonomous weather stations withmore than 30000 units [16] Data-generating devices and controlterminals are commonly located in different locations such as fieldsislands reservations among others

Using the concept of clientserver all collected data is sent to adata server simultaneously In this system where there is a largedata transmission network that is transmitted with a small amountof information and in large scale making use of sequential transmis-sions it is necessary to implement an efficient real-time processingcapable of receiving and treating the data synchronously

Guangsheng and colleagues in [16] presents a detailed discussionon the design of a system for receiving and processing data fromautonomous meteorological stations in Guangdong province Eachobservation station produces a group of data every 5 minutes ondays of good weather or 1 minute in atypical situations Thus the

model must be auditable and extensible according to the technicalor operational needs of the meteorological stations in addition toavoiding the congestion in the observation data uploaded at criticalmoments For this the premise is that the entire processing anddata storage cycle must be completed within 1 minute or less

This system was deployed in more than 1700 weather stationsfor over a year Running with stability efficiency and reliability Allstations send their packets to the control center at the same timeevery minute The time for receiving processing and uploadingthe response for each cycle was less than 1 minute and the hard-ware requirements are low reducing maintenance and reducingthe complexity of the systems used

43 Real-Time Scheduling for Large ScaleApplication Control

In [15] Waknis and Sztipanovits argue that real-time applicationrequirements can be satisfied by a scheduling scheme appropriateto a model-based programming environment They developed anintelligent process control system which is a model-based systemand acts as a real-time supervisor of large-scale industrial processesable to control various activities monitor different operations andmaking use of the sensor information to diagnose a system failure[15]

The schema is constructed in three layers (1) model buildingtools layer which allows the qualitative and quantitative repre-sentation of the information (2) model interpretation layer wherethe information is stored in a descriptive way which is used formonitoring control and diagnosis besides having the capacity totransform the models into executable computational structures and(3) the environment execution layer which is implemented with themultigraph architecture using directional graphs The nodes of thegraph represent the computational blocks and their connectionsrepresent the flow of data between them Each block has a programsegment that is staggered according to demand and data flow

One notable feature that is offered by this framework is theability to dynamically reconfigure This is not limited only to ad-justments in system parameters allowing for architectural changesat runtime by the inclusion and removal of nodes and connectionsThe structure also presents interfaces for receiving informationfrom the external environment and integration with its consumersin addition it also has an interface implemented by each noderesponsible for the parallel and scalable processing of the inputdata and a similar function that processes the output data

44 Comparative AnalysisThis section presents a comparative analysis of the related worksThis comparison serves as a basis to identify the similarities anddifferences between the Doric and the selected works The com-parison criteria seek to reveal specific characteristics of the relatedworks which can be contrasted with the proposed architectureThe comparison criteria are discussed as follows

bull Data source Studies that seek to offer expandable archi-tectures allowing to support several data sources Modernarchitectures for information systems need to give their pro-tection to multiple types and formats of data

DORIC An Architecture for Data-intensive Real-time Applications SBSIrsquo2018 Junho 2018 Caxias do Sul Rio Grande do Sul Brazil

Table 4 Comparative Analysis

ModularizedData Source Publish and Subscrib Information Processing Analysis Storage Consumers

Expandable Expandable Scalable Expandable Scalable Expandable scalable Expandable Scalable Expandable

Real-time and Large-scale

processing for Scientific Applicationssim - sim + sim + sim + sim + -

Real-time and Large-scale

processing for Weather Stationssim - sim + sim + sim + sim sim +

Real-time Scheduling For

Large-scale Control Applicationssim sim + + sim + sim + sim + +

Proposed Architecture + + + + + + + + + + +

Legend ldquo+rdquo Supports ldquosimrdquo Partially Supports ldquo-rdquo Does not Support

bull PublishSubscribe Studies that are based on the publish-subscribe pattern

bull Information processing Studies that are scalable and ex-pandable for information processing Processing informationis fundamental to any architecture for the development ofmodern information systems

bull Analysis Studies that provide support for data analysisthrough machine learning techniques

bull Storage Studies that are extensible and scalable with regardto the type and volume of data storage

bull Consumers Studies that are flexible to support differenttypes of customers

Table 4 presents the comparative analysis considering thesecriteria It is observed that only the Doric fully meets the definedcriteria highlighting the contribution and the differential of thiswork

It is possible to verify that in spite of the great scalability of theevaluated projects there is a deficiency in the issue of expansionand reuse of the structure for this reason the proposed architec-ture aims not only to satisfy the demands presented but also toprovide a structure divided into modules and reusable making useof microservices and emphasizing the separation between its pro-cessing stages Such an implementation favors the construction ofexpandable reusable and easily understood structures contributingboth to existing systems and to those that will still be built

5 CONCLUSIONThe adoption of DIRT applications increased in recent years How-ever there is a lack of standardization and modularized architec-tures This implies that a specific infrastructure was needed to beprojected to attend a different demand of the applications In otherwords a different construction was made despite they have thesame essential purpose This purpose is to serve as a large-scaleand real-time architecture

To resolve this problem this work proposed the Doric a flexibleand modular architecture for data-intensive and real-time systemsThis architecture was built to accommodate the state-of-the-arttechnologies related to this purpose A prototype was implementedto conduct an empirical evaluation This evaluation demonstratedthat the architecture could process a large volume of data in a shortperiod of time This was possible even with a hardware limitationLastly the future work consists of deploying the architecture in anenvironment with improved hardware capability and evaluate thearchitecture adaptability in more scenarios

6 ACKNOWLEDGMENTSThank you to Unisinos for providing the spaces and resources toconduct this research

REFERENCES[1] Ricardo Alexandre Afonso WM Da Silva GHRP Tomas Kiev Gama Alezy

Oliveira Alexandre Alvaro and Vinicius Cardoso Garcia 2013 Br-SCMM Mod-elo Brasileiro de Maturidade para Cidades Inteligentes Simpoacutesio Brasileiro DeSistemas De Informaccedilatildeo (2013)

[2] Apache 2018 Apache Spark httpssparkapacheorg [online accessed onjanuary 2018]

[3] Apache 2018 Apache Zookeper httpszookeeperapacheorg [onlineaccessed on january 2018]

[4] Luciana Regina Bencke Anderson Luiz Fernandes Perez and Osvaldo da Costa Ar-mendaris 2017 Rodovias Inteligentes uma visatildeo geral sobre as tecnologias em-pregadas no Brasil e no mundo iSys-Revista Brasileira de Sistemas de Informaccedilatildeo10 4 (2017) 80ndash102

[5] Junwei Cao and Junwei Li 2011 Large-scale real-time data-driven scientificapplications In Networking and Distributed Computing (ICNDC) 2011 SecondInternational Conference on IEEE 116ndash121

[6] Dave Evans 2012 The Internet of Things how the next evolution of the internet ischanging everything (april 2011) White Paper by Cisco Internet Business SolutionsGroup (IBSG) (2012)

[7] Apache Kafka 2017 Apache Kafka a distributed system platform Introductionhttpkafkaapacheorgintrohtml [online accessed on january 2018]

[8] Juacutelio Menezes Jr and Cristine Gusmatildeo 2014 InteliMEDndashProposta de Sistema deApoio ao Diagnoacutestico Meacutedico para Dispositivos Moacuteveis iSys-Revista Brasileirade Sistemas de Informaccedilatildeo 6 1 (2014) 44ndash61

[9] Mongo 2018 MongoDB httpswwwmongodbcom [online accessed onjanuary 2018]

[10] Linda Northrop Peter Feiler Richard P Gabriel John Goodenough Rick LingerTom Longstaff Rick Kazman Mark Klein Douglas Schmidt Kevin Sullivan et al2006 Ultra-large-scale systems The software challenge of the future Technical Re-port CARNEGIE-MELLON UNIV PITTSBURGH PA SOFTWARE ENGINEERINGINST

[11] Oracle 2018 Java 8 httpwwworaclecomtechnetworkptjavajavasedownloadsjre8-downloads-2133155html [online accessed on january 2018]

[12] Dan Puiu Payam Barnaghi Ralf Toenjes Daniel Kuumlmper Muhammad IntizarAli Alessandra Mileo Josiane Xavier Parreira Marten Fischer Sefki KolozaliNazli Farajidavar et al 2016 Citypulse Large scale data analytics framework forsmart cities IEEE Access 4 (2016) 1086ndash1108

[13] The Statistics Portal STATISTA 2017 Internet of Things (IoT) number of con-nected devices worldwide from 2015 to 2025 (in billions) httpswwwstatistacomstatistics471264iot-number-of-connected-devices-worldwide

[14] STRATA 2015 Large-Scale Real-Time Applications [Strata + Hadoop WorldConferences]

[15] Prashant Waknis and Janos Sztipanovits 1993 Hard real-time scheduling forlarge scale process control applications In Proceedings of the IEEE Workshop onReal-Time Applications IEEE 71ndash75

[16] Guangsheng Wu Zhenlang Ao Jianyong Li and Qinqiang Zhou 2011 A Real-time Receiving and Distributed Processing System for Large-scale Burst Data In2011 Second International Conference on Networking and Distributed Computing(ICNDC) IEEE 111ndash115

DORIC An Architecture for Data-intensive Real-time Applications SBSIrsquo2018 Junho 2018 Caxias do Sul Rio Grande do Sul Brazil

The evaluation of capacity of real-time response was performedwith five different file sizes Table 2 presents the obtained resultsThe import process was repeated fifty times for each file to obtainreliable values The size of the file influences in the processingtime The time increased when larger files were processed Wealso observed the time between sending the last record and thenotification of process completion In this case the average time alsoincreased according to the size of the file Specifically the averagetime increases significantly for 30 MB files Figure 5 illustrates theresults contained on Table2

Table 2 Test results of data provided from direct files

File

Size (MB)

Dispatch Time

Averarge (ms)

Receiving Time

Averarge (ms)

10 34366 12743

20 39682 140025

30 42562 201919

40 90672 222783

50 162845 22626

Next we also evaluated the architecturersquos capability to handlelarge-scale data For this we executed multiple threads to simulatethe role of data sources These threads simultaneously executedand provided the same topic Each thread creates a producer andgenerates a certain number of messages We monitored two aver-age times in this case (1) the average time spent on sending themessages to the server and (2) the average time the infrastructuretakes to signalize that it received the information Figure 5 depictsthe results Table 3 specifies the results obtained

File Size (MB)

Tim

e (

ms)

Figure 5 Receiving time average

The results evidence a gradual increase in the receiving timewhen there are more threads in execution The same occurs in themoment that the number of messages increases This behavior wasexpected because the tests were conducted on a virtual machineThere is a limitation regarding the hardware capability eg there isonly one CPU This implies that a larger number of threads causes

Table 3 Test results of data provided from threads

ThreadsMenssages

Number

Dispatch Time

Averarge (ms)

Receiving Time

Averarge (ms)

2 2 0 45083

4 2 0 52482

8 2 0 67111

16 2 0 68427

32 2 32 75904

1024 2 2592 182025

2 20 0 25002

4 20 0 21881

8 20 0 33174

16 20 0 97486

32 20 632 102732

1024 20 31448 200572

congestion on the processing queue There is more processes wait-ing on the queue because there is no available resource to executeall simultaneously Furthermore this overall hardware resource isshared with the operating system on which the virtual machine isdeployed This also impacts negatively in the average time

Figure 6 shows the results obtained Analyzing the extreme caseon the second test there are 20 messages being sent from 1024data sources simultaneously This is a total of 20480 messagesThese numbers of messages were processed in 200572 ms In thecase of transmitting messages from a single producer we reachedat the number of 106650 messages processed in 784ms Due toresource limitation the increasing of threads impacts negativelyon the performance of messages sent

Tim

e (

ms)

Number of Threads

2 Messages

20 Messages

Figure 6 Representation of receiving average time fromtread

4 RELATEDWORKSThere is a growing concern on which architecture must be used forapplications that process large volumes of data in real time Recent

SBSIrsquo2018 Junho 2018 Caxias do Sul Rio Grande do Sul Brazil M Cadaviz et al

literature has explored this challenge in the last years Howeverlittle has been done to clarify about how different architecturalapproaches might be brought together to address particular facetsof DIRT information systems

By doing so this section explores three architectures for DIRTapplications applied to different domains Section 41 describes anarchitecture for scientific applications Section 42 describes anarchitecture for weather applications Section 43 presents an ar-chitecture for application control Finally Section 44 presents acomparative analysis between the Doric and the architectures dis-cussed in order to identify the similarities and differences betweenthem as well as the research gaps the proposed architecture fulfills

41 Large-Scale and Real Time Processing forScientific Applications

Based on dynamic data-driven application systems (DDDAS) con-cept Cao and Li propose in [5] a framework to optimize the perfor-mance of data processing where data is generated in high volumeabout 10 MB per second almost 1 TB per day Another challengeis the issue of transportation since the equipment is spread overthousands of miles away In this context performance is a criticalfactor since the processing of the collected information must becompleted in 30 minutes

The architecture proposed is based on four layers (1) appli-cation allows the input of parameters and publishes the resultsin a user-friendly way (2) data processing performs additionaldata processing based on monitoring outputs (3) data monitor-ing dynamically accompanies information on a central monitorand (4) instrumentation receives elements collected from one ormore measurement systems and submits them to treatment stepssubsequently storing the assembly in the instrumentation layerAccording to [5] the frameworkrsquos strengths are in data reductionalgorithms which can reduce a block of data from 9063 MB to 29MB thus reducing possible bottlenecks in the transport area

The authors believe that in the near future the data scaleswill grow exponentially In this scenario although the concept ofDDDAS is directed to simulation applications the framework willhave the possibility of revealing its potential for several scientificapplications

42 Real-time Processing for Weather StationsIn China it is common to use autonomous weather stations withmore than 30000 units [16] Data-generating devices and controlterminals are commonly located in different locations such as fieldsislands reservations among others

Using the concept of clientserver all collected data is sent to adata server simultaneously In this system where there is a largedata transmission network that is transmitted with a small amountof information and in large scale making use of sequential transmis-sions it is necessary to implement an efficient real-time processingcapable of receiving and treating the data synchronously

Guangsheng and colleagues in [16] presents a detailed discussionon the design of a system for receiving and processing data fromautonomous meteorological stations in Guangdong province Eachobservation station produces a group of data every 5 minutes ondays of good weather or 1 minute in atypical situations Thus the

model must be auditable and extensible according to the technicalor operational needs of the meteorological stations in addition toavoiding the congestion in the observation data uploaded at criticalmoments For this the premise is that the entire processing anddata storage cycle must be completed within 1 minute or less

This system was deployed in more than 1700 weather stationsfor over a year Running with stability efficiency and reliability Allstations send their packets to the control center at the same timeevery minute The time for receiving processing and uploadingthe response for each cycle was less than 1 minute and the hard-ware requirements are low reducing maintenance and reducingthe complexity of the systems used

43 Real-Time Scheduling for Large ScaleApplication Control

In [15] Waknis and Sztipanovits argue that real-time applicationrequirements can be satisfied by a scheduling scheme appropriateto a model-based programming environment They developed anintelligent process control system which is a model-based systemand acts as a real-time supervisor of large-scale industrial processesable to control various activities monitor different operations andmaking use of the sensor information to diagnose a system failure[15]

The schema is constructed in three layers (1) model buildingtools layer which allows the qualitative and quantitative repre-sentation of the information (2) model interpretation layer wherethe information is stored in a descriptive way which is used formonitoring control and diagnosis besides having the capacity totransform the models into executable computational structures and(3) the environment execution layer which is implemented with themultigraph architecture using directional graphs The nodes of thegraph represent the computational blocks and their connectionsrepresent the flow of data between them Each block has a programsegment that is staggered according to demand and data flow

One notable feature that is offered by this framework is theability to dynamically reconfigure This is not limited only to ad-justments in system parameters allowing for architectural changesat runtime by the inclusion and removal of nodes and connectionsThe structure also presents interfaces for receiving informationfrom the external environment and integration with its consumersin addition it also has an interface implemented by each noderesponsible for the parallel and scalable processing of the inputdata and a similar function that processes the output data

44 Comparative AnalysisThis section presents a comparative analysis of the related worksThis comparison serves as a basis to identify the similarities anddifferences between the Doric and the selected works The com-parison criteria seek to reveal specific characteristics of the relatedworks which can be contrasted with the proposed architectureThe comparison criteria are discussed as follows

bull Data source Studies that seek to offer expandable archi-tectures allowing to support several data sources Modernarchitectures for information systems need to give their pro-tection to multiple types and formats of data

DORIC An Architecture for Data-intensive Real-time Applications SBSIrsquo2018 Junho 2018 Caxias do Sul Rio Grande do Sul Brazil

Table 4 Comparative Analysis

ModularizedData Source Publish and Subscrib Information Processing Analysis Storage Consumers

Expandable Expandable Scalable Expandable Scalable Expandable scalable Expandable Scalable Expandable

Real-time and Large-scale

processing for Scientific Applicationssim - sim + sim + sim + sim + -

Real-time and Large-scale

processing for Weather Stationssim - sim + sim + sim + sim sim +

Real-time Scheduling For

Large-scale Control Applicationssim sim + + sim + sim + sim + +

Proposed Architecture + + + + + + + + + + +

Legend ldquo+rdquo Supports ldquosimrdquo Partially Supports ldquo-rdquo Does not Support

bull PublishSubscribe Studies that are based on the publish-subscribe pattern

bull Information processing Studies that are scalable and ex-pandable for information processing Processing informationis fundamental to any architecture for the development ofmodern information systems

bull Analysis Studies that provide support for data analysisthrough machine learning techniques

bull Storage Studies that are extensible and scalable with regardto the type and volume of data storage

bull Consumers Studies that are flexible to support differenttypes of customers

Table 4 presents the comparative analysis considering thesecriteria It is observed that only the Doric fully meets the definedcriteria highlighting the contribution and the differential of thiswork

It is possible to verify that in spite of the great scalability of theevaluated projects there is a deficiency in the issue of expansionand reuse of the structure for this reason the proposed architec-ture aims not only to satisfy the demands presented but also toprovide a structure divided into modules and reusable making useof microservices and emphasizing the separation between its pro-cessing stages Such an implementation favors the construction ofexpandable reusable and easily understood structures contributingboth to existing systems and to those that will still be built

5 CONCLUSIONThe adoption of DIRT applications increased in recent years How-ever there is a lack of standardization and modularized architec-tures This implies that a specific infrastructure was needed to beprojected to attend a different demand of the applications In otherwords a different construction was made despite they have thesame essential purpose This purpose is to serve as a large-scaleand real-time architecture

To resolve this problem this work proposed the Doric a flexibleand modular architecture for data-intensive and real-time systemsThis architecture was built to accommodate the state-of-the-arttechnologies related to this purpose A prototype was implementedto conduct an empirical evaluation This evaluation demonstratedthat the architecture could process a large volume of data in a shortperiod of time This was possible even with a hardware limitationLastly the future work consists of deploying the architecture in anenvironment with improved hardware capability and evaluate thearchitecture adaptability in more scenarios

6 ACKNOWLEDGMENTSThank you to Unisinos for providing the spaces and resources toconduct this research

REFERENCES[1] Ricardo Alexandre Afonso WM Da Silva GHRP Tomas Kiev Gama Alezy

Oliveira Alexandre Alvaro and Vinicius Cardoso Garcia 2013 Br-SCMM Mod-elo Brasileiro de Maturidade para Cidades Inteligentes Simpoacutesio Brasileiro DeSistemas De Informaccedilatildeo (2013)

[2] Apache 2018 Apache Spark httpssparkapacheorg [online accessed onjanuary 2018]

[3] Apache 2018 Apache Zookeper httpszookeeperapacheorg [onlineaccessed on january 2018]

[4] Luciana Regina Bencke Anderson Luiz Fernandes Perez and Osvaldo da Costa Ar-mendaris 2017 Rodovias Inteligentes uma visatildeo geral sobre as tecnologias em-pregadas no Brasil e no mundo iSys-Revista Brasileira de Sistemas de Informaccedilatildeo10 4 (2017) 80ndash102

[5] Junwei Cao and Junwei Li 2011 Large-scale real-time data-driven scientificapplications In Networking and Distributed Computing (ICNDC) 2011 SecondInternational Conference on IEEE 116ndash121

[6] Dave Evans 2012 The Internet of Things how the next evolution of the internet ischanging everything (april 2011) White Paper by Cisco Internet Business SolutionsGroup (IBSG) (2012)

[7] Apache Kafka 2017 Apache Kafka a distributed system platform Introductionhttpkafkaapacheorgintrohtml [online accessed on january 2018]

[8] Juacutelio Menezes Jr and Cristine Gusmatildeo 2014 InteliMEDndashProposta de Sistema deApoio ao Diagnoacutestico Meacutedico para Dispositivos Moacuteveis iSys-Revista Brasileirade Sistemas de Informaccedilatildeo 6 1 (2014) 44ndash61

[9] Mongo 2018 MongoDB httpswwwmongodbcom [online accessed onjanuary 2018]

[10] Linda Northrop Peter Feiler Richard P Gabriel John Goodenough Rick LingerTom Longstaff Rick Kazman Mark Klein Douglas Schmidt Kevin Sullivan et al2006 Ultra-large-scale systems The software challenge of the future Technical Re-port CARNEGIE-MELLON UNIV PITTSBURGH PA SOFTWARE ENGINEERINGINST

[11] Oracle 2018 Java 8 httpwwworaclecomtechnetworkptjavajavasedownloadsjre8-downloads-2133155html [online accessed on january 2018]

[12] Dan Puiu Payam Barnaghi Ralf Toenjes Daniel Kuumlmper Muhammad IntizarAli Alessandra Mileo Josiane Xavier Parreira Marten Fischer Sefki KolozaliNazli Farajidavar et al 2016 Citypulse Large scale data analytics framework forsmart cities IEEE Access 4 (2016) 1086ndash1108

[13] The Statistics Portal STATISTA 2017 Internet of Things (IoT) number of con-nected devices worldwide from 2015 to 2025 (in billions) httpswwwstatistacomstatistics471264iot-number-of-connected-devices-worldwide

[14] STRATA 2015 Large-Scale Real-Time Applications [Strata + Hadoop WorldConferences]

[15] Prashant Waknis and Janos Sztipanovits 1993 Hard real-time scheduling forlarge scale process control applications In Proceedings of the IEEE Workshop onReal-Time Applications IEEE 71ndash75

[16] Guangsheng Wu Zhenlang Ao Jianyong Li and Qinqiang Zhou 2011 A Real-time Receiving and Distributed Processing System for Large-scale Burst Data In2011 Second International Conference on Networking and Distributed Computing(ICNDC) IEEE 111ndash115

SBSIrsquo2018 Junho 2018 Caxias do Sul Rio Grande do Sul Brazil M Cadaviz et al

literature has explored this challenge in the last years Howeverlittle has been done to clarify about how different architecturalapproaches might be brought together to address particular facetsof DIRT information systems

By doing so this section explores three architectures for DIRTapplications applied to different domains Section 41 describes anarchitecture for scientific applications Section 42 describes anarchitecture for weather applications Section 43 presents an ar-chitecture for application control Finally Section 44 presents acomparative analysis between the Doric and the architectures dis-cussed in order to identify the similarities and differences betweenthem as well as the research gaps the proposed architecture fulfills

41 Large-Scale and Real Time Processing forScientific Applications

Based on dynamic data-driven application systems (DDDAS) con-cept Cao and Li propose in [5] a framework to optimize the perfor-mance of data processing where data is generated in high volumeabout 10 MB per second almost 1 TB per day Another challengeis the issue of transportation since the equipment is spread overthousands of miles away In this context performance is a criticalfactor since the processing of the collected information must becompleted in 30 minutes

The architecture proposed is based on four layers (1) appli-cation allows the input of parameters and publishes the resultsin a user-friendly way (2) data processing performs additionaldata processing based on monitoring outputs (3) data monitor-ing dynamically accompanies information on a central monitorand (4) instrumentation receives elements collected from one ormore measurement systems and submits them to treatment stepssubsequently storing the assembly in the instrumentation layerAccording to [5] the frameworkrsquos strengths are in data reductionalgorithms which can reduce a block of data from 9063 MB to 29MB thus reducing possible bottlenecks in the transport area

The authors believe that in the near future the data scaleswill grow exponentially In this scenario although the concept ofDDDAS is directed to simulation applications the framework willhave the possibility of revealing its potential for several scientificapplications

42 Real-time Processing for Weather StationsIn China it is common to use autonomous weather stations withmore than 30000 units [16] Data-generating devices and controlterminals are commonly located in different locations such as fieldsislands reservations among others

Using the concept of clientserver all collected data is sent to adata server simultaneously In this system where there is a largedata transmission network that is transmitted with a small amountof information and in large scale making use of sequential transmis-sions it is necessary to implement an efficient real-time processingcapable of receiving and treating the data synchronously

Guangsheng and colleagues in [16] presents a detailed discussionon the design of a system for receiving and processing data fromautonomous meteorological stations in Guangdong province Eachobservation station produces a group of data every 5 minutes ondays of good weather or 1 minute in atypical situations Thus the

model must be auditable and extensible according to the technicalor operational needs of the meteorological stations in addition toavoiding the congestion in the observation data uploaded at criticalmoments For this the premise is that the entire processing anddata storage cycle must be completed within 1 minute or less

This system was deployed in more than 1700 weather stationsfor over a year Running with stability efficiency and reliability Allstations send their packets to the control center at the same timeevery minute The time for receiving processing and uploadingthe response for each cycle was less than 1 minute and the hard-ware requirements are low reducing maintenance and reducingthe complexity of the systems used

43 Real-Time Scheduling for Large ScaleApplication Control

In [15] Waknis and Sztipanovits argue that real-time applicationrequirements can be satisfied by a scheduling scheme appropriateto a model-based programming environment They developed anintelligent process control system which is a model-based systemand acts as a real-time supervisor of large-scale industrial processesable to control various activities monitor different operations andmaking use of the sensor information to diagnose a system failure[15]

The schema is constructed in three layers (1) model buildingtools layer which allows the qualitative and quantitative repre-sentation of the information (2) model interpretation layer wherethe information is stored in a descriptive way which is used formonitoring control and diagnosis besides having the capacity totransform the models into executable computational structures and(3) the environment execution layer which is implemented with themultigraph architecture using directional graphs The nodes of thegraph represent the computational blocks and their connectionsrepresent the flow of data between them Each block has a programsegment that is staggered according to demand and data flow

One notable feature that is offered by this framework is theability to dynamically reconfigure This is not limited only to ad-justments in system parameters allowing for architectural changesat runtime by the inclusion and removal of nodes and connectionsThe structure also presents interfaces for receiving informationfrom the external environment and integration with its consumersin addition it also has an interface implemented by each noderesponsible for the parallel and scalable processing of the inputdata and a similar function that processes the output data

44 Comparative AnalysisThis section presents a comparative analysis of the related worksThis comparison serves as a basis to identify the similarities anddifferences between the Doric and the selected works The com-parison criteria seek to reveal specific characteristics of the relatedworks which can be contrasted with the proposed architectureThe comparison criteria are discussed as follows

bull Data source Studies that seek to offer expandable archi-tectures allowing to support several data sources Modernarchitectures for information systems need to give their pro-tection to multiple types and formats of data

DORIC An Architecture for Data-intensive Real-time Applications SBSIrsquo2018 Junho 2018 Caxias do Sul Rio Grande do Sul Brazil

Table 4 Comparative Analysis

ModularizedData Source Publish and Subscrib Information Processing Analysis Storage Consumers

Expandable Expandable Scalable Expandable Scalable Expandable scalable Expandable Scalable Expandable

Real-time and Large-scale

processing for Scientific Applicationssim - sim + sim + sim + sim + -

Real-time and Large-scale

processing for Weather Stationssim - sim + sim + sim + sim sim +

Real-time Scheduling For

Large-scale Control Applicationssim sim + + sim + sim + sim + +

Proposed Architecture + + + + + + + + + + +

Legend ldquo+rdquo Supports ldquosimrdquo Partially Supports ldquo-rdquo Does not Support

bull PublishSubscribe Studies that are based on the publish-subscribe pattern

bull Information processing Studies that are scalable and ex-pandable for information processing Processing informationis fundamental to any architecture for the development ofmodern information systems

bull Analysis Studies that provide support for data analysisthrough machine learning techniques

bull Storage Studies that are extensible and scalable with regardto the type and volume of data storage

bull Consumers Studies that are flexible to support differenttypes of customers

Table 4 presents the comparative analysis considering thesecriteria It is observed that only the Doric fully meets the definedcriteria highlighting the contribution and the differential of thiswork

It is possible to verify that in spite of the great scalability of theevaluated projects there is a deficiency in the issue of expansionand reuse of the structure for this reason the proposed architec-ture aims not only to satisfy the demands presented but also toprovide a structure divided into modules and reusable making useof microservices and emphasizing the separation between its pro-cessing stages Such an implementation favors the construction ofexpandable reusable and easily understood structures contributingboth to existing systems and to those that will still be built

5 CONCLUSIONThe adoption of DIRT applications increased in recent years How-ever there is a lack of standardization and modularized architec-tures This implies that a specific infrastructure was needed to beprojected to attend a different demand of the applications In otherwords a different construction was made despite they have thesame essential purpose This purpose is to serve as a large-scaleand real-time architecture

To resolve this problem this work proposed the Doric a flexibleand modular architecture for data-intensive and real-time systemsThis architecture was built to accommodate the state-of-the-arttechnologies related to this purpose A prototype was implementedto conduct an empirical evaluation This evaluation demonstratedthat the architecture could process a large volume of data in a shortperiod of time This was possible even with a hardware limitationLastly the future work consists of deploying the architecture in anenvironment with improved hardware capability and evaluate thearchitecture adaptability in more scenarios

6 ACKNOWLEDGMENTSThank you to Unisinos for providing the spaces and resources toconduct this research

REFERENCES[1] Ricardo Alexandre Afonso WM Da Silva GHRP Tomas Kiev Gama Alezy

Oliveira Alexandre Alvaro and Vinicius Cardoso Garcia 2013 Br-SCMM Mod-elo Brasileiro de Maturidade para Cidades Inteligentes Simpoacutesio Brasileiro DeSistemas De Informaccedilatildeo (2013)

[2] Apache 2018 Apache Spark httpssparkapacheorg [online accessed onjanuary 2018]

[3] Apache 2018 Apache Zookeper httpszookeeperapacheorg [onlineaccessed on january 2018]

[4] Luciana Regina Bencke Anderson Luiz Fernandes Perez and Osvaldo da Costa Ar-mendaris 2017 Rodovias Inteligentes uma visatildeo geral sobre as tecnologias em-pregadas no Brasil e no mundo iSys-Revista Brasileira de Sistemas de Informaccedilatildeo10 4 (2017) 80ndash102

[5] Junwei Cao and Junwei Li 2011 Large-scale real-time data-driven scientificapplications In Networking and Distributed Computing (ICNDC) 2011 SecondInternational Conference on IEEE 116ndash121

[6] Dave Evans 2012 The Internet of Things how the next evolution of the internet ischanging everything (april 2011) White Paper by Cisco Internet Business SolutionsGroup (IBSG) (2012)

[7] Apache Kafka 2017 Apache Kafka a distributed system platform Introductionhttpkafkaapacheorgintrohtml [online accessed on january 2018]

[8] Juacutelio Menezes Jr and Cristine Gusmatildeo 2014 InteliMEDndashProposta de Sistema deApoio ao Diagnoacutestico Meacutedico para Dispositivos Moacuteveis iSys-Revista Brasileirade Sistemas de Informaccedilatildeo 6 1 (2014) 44ndash61

[9] Mongo 2018 MongoDB httpswwwmongodbcom [online accessed onjanuary 2018]

[10] Linda Northrop Peter Feiler Richard P Gabriel John Goodenough Rick LingerTom Longstaff Rick Kazman Mark Klein Douglas Schmidt Kevin Sullivan et al2006 Ultra-large-scale systems The software challenge of the future Technical Re-port CARNEGIE-MELLON UNIV PITTSBURGH PA SOFTWARE ENGINEERINGINST

[11] Oracle 2018 Java 8 httpwwworaclecomtechnetworkptjavajavasedownloadsjre8-downloads-2133155html [online accessed on january 2018]

[12] Dan Puiu Payam Barnaghi Ralf Toenjes Daniel Kuumlmper Muhammad IntizarAli Alessandra Mileo Josiane Xavier Parreira Marten Fischer Sefki KolozaliNazli Farajidavar et al 2016 Citypulse Large scale data analytics framework forsmart cities IEEE Access 4 (2016) 1086ndash1108

[13] The Statistics Portal STATISTA 2017 Internet of Things (IoT) number of con-nected devices worldwide from 2015 to 2025 (in billions) httpswwwstatistacomstatistics471264iot-number-of-connected-devices-worldwide

[14] STRATA 2015 Large-Scale Real-Time Applications [Strata + Hadoop WorldConferences]

[15] Prashant Waknis and Janos Sztipanovits 1993 Hard real-time scheduling forlarge scale process control applications In Proceedings of the IEEE Workshop onReal-Time Applications IEEE 71ndash75

[16] Guangsheng Wu Zhenlang Ao Jianyong Li and Qinqiang Zhou 2011 A Real-time Receiving and Distributed Processing System for Large-scale Burst Data In2011 Second International Conference on Networking and Distributed Computing(ICNDC) IEEE 111ndash115

DORIC An Architecture for Data-intensive Real-time Applications SBSIrsquo2018 Junho 2018 Caxias do Sul Rio Grande do Sul Brazil

Table 4 Comparative Analysis

ModularizedData Source Publish and Subscrib Information Processing Analysis Storage Consumers

Expandable Expandable Scalable Expandable Scalable Expandable scalable Expandable Scalable Expandable

Real-time and Large-scale

processing for Scientific Applicationssim - sim + sim + sim + sim + -

Real-time and Large-scale

processing for Weather Stationssim - sim + sim + sim + sim sim +

Real-time Scheduling For

Large-scale Control Applicationssim sim + + sim + sim + sim + +

Proposed Architecture + + + + + + + + + + +

Legend ldquo+rdquo Supports ldquosimrdquo Partially Supports ldquo-rdquo Does not Support

bull PublishSubscribe Studies that are based on the publish-subscribe pattern

bull Information processing Studies that are scalable and ex-pandable for information processing Processing informationis fundamental to any architecture for the development ofmodern information systems

bull Analysis Studies that provide support for data analysisthrough machine learning techniques

bull Storage Studies that are extensible and scalable with regardto the type and volume of data storage

bull Consumers Studies that are flexible to support differenttypes of customers

Table 4 presents the comparative analysis considering thesecriteria It is observed that only the Doric fully meets the definedcriteria highlighting the contribution and the differential of thiswork

It is possible to verify that in spite of the great scalability of theevaluated projects there is a deficiency in the issue of expansionand reuse of the structure for this reason the proposed architec-ture aims not only to satisfy the demands presented but also toprovide a structure divided into modules and reusable making useof microservices and emphasizing the separation between its pro-cessing stages Such an implementation favors the construction ofexpandable reusable and easily understood structures contributingboth to existing systems and to those that will still be built

5 CONCLUSIONThe adoption of DIRT applications increased in recent years How-ever there is a lack of standardization and modularized architec-tures This implies that a specific infrastructure was needed to beprojected to attend a different demand of the applications In otherwords a different construction was made despite they have thesame essential purpose This purpose is to serve as a large-scaleand real-time architecture

To resolve this problem this work proposed the Doric a flexibleand modular architecture for data-intensive and real-time systemsThis architecture was built to accommodate the state-of-the-arttechnologies related to this purpose A prototype was implementedto conduct an empirical evaluation This evaluation demonstratedthat the architecture could process a large volume of data in a shortperiod of time This was possible even with a hardware limitationLastly the future work consists of deploying the architecture in anenvironment with improved hardware capability and evaluate thearchitecture adaptability in more scenarios

6 ACKNOWLEDGMENTSThank you to Unisinos for providing the spaces and resources toconduct this research

REFERENCES[1] Ricardo Alexandre Afonso WM Da Silva GHRP Tomas Kiev Gama Alezy

Oliveira Alexandre Alvaro and Vinicius Cardoso Garcia 2013 Br-SCMM Mod-elo Brasileiro de Maturidade para Cidades Inteligentes Simpoacutesio Brasileiro DeSistemas De Informaccedilatildeo (2013)

[2] Apache 2018 Apache Spark httpssparkapacheorg [online accessed onjanuary 2018]

[3] Apache 2018 Apache Zookeper httpszookeeperapacheorg [onlineaccessed on january 2018]

[4] Luciana Regina Bencke Anderson Luiz Fernandes Perez and Osvaldo da Costa Ar-mendaris 2017 Rodovias Inteligentes uma visatildeo geral sobre as tecnologias em-pregadas no Brasil e no mundo iSys-Revista Brasileira de Sistemas de Informaccedilatildeo10 4 (2017) 80ndash102

[5] Junwei Cao and Junwei Li 2011 Large-scale real-time data-driven scientificapplications In Networking and Distributed Computing (ICNDC) 2011 SecondInternational Conference on IEEE 116ndash121

[6] Dave Evans 2012 The Internet of Things how the next evolution of the internet ischanging everything (april 2011) White Paper by Cisco Internet Business SolutionsGroup (IBSG) (2012)

[7] Apache Kafka 2017 Apache Kafka a distributed system platform Introductionhttpkafkaapacheorgintrohtml [online accessed on january 2018]

[8] Juacutelio Menezes Jr and Cristine Gusmatildeo 2014 InteliMEDndashProposta de Sistema deApoio ao Diagnoacutestico Meacutedico para Dispositivos Moacuteveis iSys-Revista Brasileirade Sistemas de Informaccedilatildeo 6 1 (2014) 44ndash61

[9] Mongo 2018 MongoDB httpswwwmongodbcom [online accessed onjanuary 2018]

[10] Linda Northrop Peter Feiler Richard P Gabriel John Goodenough Rick LingerTom Longstaff Rick Kazman Mark Klein Douglas Schmidt Kevin Sullivan et al2006 Ultra-large-scale systems The software challenge of the future Technical Re-port CARNEGIE-MELLON UNIV PITTSBURGH PA SOFTWARE ENGINEERINGINST

[11] Oracle 2018 Java 8 httpwwworaclecomtechnetworkptjavajavasedownloadsjre8-downloads-2133155html [online accessed on january 2018]

[12] Dan Puiu Payam Barnaghi Ralf Toenjes Daniel Kuumlmper Muhammad IntizarAli Alessandra Mileo Josiane Xavier Parreira Marten Fischer Sefki KolozaliNazli Farajidavar et al 2016 Citypulse Large scale data analytics framework forsmart cities IEEE Access 4 (2016) 1086ndash1108

[13] The Statistics Portal STATISTA 2017 Internet of Things (IoT) number of con-nected devices worldwide from 2015 to 2025 (in billions) httpswwwstatistacomstatistics471264iot-number-of-connected-devices-worldwide

[14] STRATA 2015 Large-Scale Real-Time Applications [Strata + Hadoop WorldConferences]

[15] Prashant Waknis and Janos Sztipanovits 1993 Hard real-time scheduling forlarge scale process control applications In Proceedings of the IEEE Workshop onReal-Time Applications IEEE 71ndash75

[16] Guangsheng Wu Zhenlang Ao Jianyong Li and Qinqiang Zhou 2011 A Real-time Receiving and Distributed Processing System for Large-scale Burst Data In2011 Second International Conference on Networking and Distributed Computing(ICNDC) IEEE 111ndash115