kth.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
An Emulation-Based Performance Evaluation Methodology for Edge Computing and Latency Sensitive Applications
KTH, School of Electrical Engineering and Computer Science (EECS), Intelligent systems, Information Science and Engineering.ORCID iD: 0000-0002-3383-2335
2023 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Cloud Computing, with its globally-accessible nature and virtually unlimited scalability, has revolutionized our daily lives since its widespread adoption in the early 2000s. It allows us to access our documents anywhere, keep in touch with friends, back up photos, and even remotely control our appliances. Despite this, Cloud Computing has limitations when it comes to novel appli- cations requiring real-time processing or low-latencies. Applications such as Cyber-Physical Systems (CPSs) or mobile eXtended Reality (XR), which in turn also have great transformative potential, are unable to run on the Cloud. 

Edge Computing is emerging as a potential solution to these limitations by bringing computation closer to the edge of the network, thereby reducing latency and enabling real-time decision making. The combination of Edge Computing and modern mobile network technologies such as 5G offers potential for massive deployments of latency-sensitive applications. However, scaling and understanding these deployments poses important challenges such the optimization of latency through multiple processing steps and trade-offs in wireless system choice, protocols, hardware, and algorithms. Existing approaches have so far been unsuccessful in capturing the complex effects arising from the interplay between network and compute in these systems. 

This dissertation addresses the challenge of performance evaluation of Edge Computing and the applications enabled by this paradigm with two key contributions to literature. First, a methodological approach to experimentally studying the trade-offs between system responsiveness and resource consumption in latency-sensitive applications such as CPSs and XR is introduced. These applications and systems feature characteristics, such as tight interaction with the physical world and the involvement of humans, that make them challenging to study through simulated approaches or analytical modeling. The approach presented in this thesis involves therefore the emulation of the client-side workload while maintaining the real server-side process and network stack to retain the realism of network and compute effects. 

Next, an exploration of the requirements for accuracy in the emulation is presented. This work discusses the extent to which accuracy in the emulation can open new avenues for optimization of these systems. To this end, the first-ever realistic model of human timings for a particular class of Mobile Augmented Reality (MAR) applications is provided. The model is combined with a mathematical framework to study the potential for optimization in Edge Computing applications. 

Results indicate that the methodology introduced in this work offers advantages over existing methods by improving efficiency, repeatability, and replicability. By fully integrating workload components into the emulated software domain, this methodology reduces complexity while still capturing complex effects of network and compute factors that are challenging to model. This approach represents thus an important contribution to literature, as it consists of a comprehensive method for the performance evaluation of Edge environments, encompassing both the application and the infrastructure. Furthermore, results from the exploration into the implications of realism in the emulation suggest that incorporating enhanced realism in client-side emulation can enable the implementation of optimization approaches that would otherwise be infeasible. In particular, this work highlights the significance of considering human behavior and reactions in addition to system-related metrics and performance optimizations in the context of MAR. 

Abstract [sv]

Cloud Computing, med sin globalt tillgängliga natur och praktiskt taget obegränsade skalbarhet, har revolutionerat våra dagliga liv sedan dess allmänna antagande i början av 2000-talet. Det gör det möjligt för oss att få tillgång till våra dokument var som helst, hålla kontakt med vänner, säkerhetskopiera foton, och fjärrstyra våra hushållsapparater.  Trots detta har Cloud Computing begränsningar när det gäller nya applikationer som kräver realtidsbehandling eller låg latens. Applikationer som cyberfysiska system (engelska Cyber-Physical Systems (CPSs)) eller mobilt utvidgad verklighet (engelska eXtended Reality (XR)), som i sin tur också har stor omvandlingspotential, kan inte köras på molnet.

Edge Computing framstår som en potentiell lösning på dessa begräns-ningar genom att föra beräkningar närmare kanten av nätverket, vilket minskar latensen och möjliggör beslutsfattande i realtid. Kombinationen av Edge Computing och moderna mobila nätverksteknologier som 5G erbjuder potential för massiva utrullningar av latenskänsliga applikationer.  Men att skala och förstå dessa utrullningar utgör viktiga utmaningar såsom optimering av latens genom flera bearbetningssteg och avvägningar i val av trådlösa system, protokoll, hårdvara och algoritmer. Befintliga metoder har hittills inte lyckats fånga de komplexa effekter som uppstår från samspelet mellan nätverk och datorer i dessa system. Denna avhandling tar upp utmaningen med prestationsutvärdering av Edge Computing och applikationerna som möjliggörs av detta paradigm med två viktiga bidrag till litteraturen. Först introduceras ett metodologiskt tillvägagångssätt för att experimentellt studera kompromisserna mellan systemrespons och resursförbrukning i latenskänsliga applikationer såsom CPSs och XR som körs på Edge Computing. Dessa applikationer och system har egenskaper, såsom tät interaktion med den fysiska världen och inblandning av människor, som gör dem utmanande att studera genom simulerade tillvägagångssätt eller analytisk modellering. Den metod som presenteras i denna avhandling innebär därför emulering av klientsidans arbetsbelastning samtidigt som den verkliga serversidans process och nätverksstacken bibehålls för att behålla realismen i nätverks- och datoreffekter.

Därefter presenteras en utforskning av kraven på precision i emuleringen. Detta arbete diskuterar i vilken utsträckning precision i emuleringen kan öppna nya vägar för optimering av dessa system. För detta ändamål tillhandahålls den första realistiska modellen av mänskliga tidsbeteenden för en särkild klass av mobil förstärkt verklighet (engelska Mobile Augmented Reality (MAR)). Modellen kombineras med ett matematiskt ramverk för att studera potentialen för optimering i Edge Computing applikationer.

Resultaten indikerar att den metod som introducerats i detta arbete erbjuder fördelar jämfört med befintliga metoder genom att förbättra effektiviteten, repeterbarheten och replikerbarheten. Genom att helt integrera arbetsbelastningskomponenter i den emulerade mjukvarudomänen, minskar denna metodik komplexiteten samtidigt som den fångar komplexa effekter av nätverks- och beräkningsfaktorer som är utmanande att modellera. Detta tillvägagångssätt representerar således ett viktigt bidrag till litteraturen, eftersom det består av en omfattande metod för prestandautvärdering av Edge Computing-miljöer, som omfattar både applikationen och infrastrukturen. Dessutom tyder resultat från utforskningen av implikationerna av precision i emuleringen att inkorporering av förbättrad precision i klientsideemulering kan möjliggöra implementering av optimeringsmetoder som annars skulle vara omöjliga. Särskilt framhåller detta arbete betydelsen av att överväga mänskligt beteende och reaktioner utöver systemrelaterade mätvärden och prestandaoptimeringar i samband med MAR.

Place, publisher, year, edition, pages
Stockholm: KTH Royal Institute of Technology, 2023. , p. xii, 85
Series
TRITA-EECS-AVL ; 2023:50
Keywords [en]
Cloud Computing, Edge Computing, Performance Evaluation, Latency-sensitive Applications, Distributed Systems, Emulation, Extended Reality, Cyber-Physical Systems, Networked Control Systems, Wearable Cognitive Assistance
National Category
Computer Systems
Research subject
Electrical Engineering
Identifiers
URN: urn:nbn:se:kth:diva-327222ISBN: 978-91-8040-613-0 (print)OAI: oai:DiVA.org:kth-327222DiVA, id: diva2:1758413
Public defence
2023-06-15, https://kth-se.zoom.us/j/63144664642, U61, Brinellvägen 26, Stockholm, 13:00 (English)
Opponent
Supervisors
Funder
Swedish Foundation for Strategic Research, ITM17–0246
Note

Plats för disputaion ändrad från Q2, Malvinas väg, KTH Campus, Stockholm till U61, Brinellvägen 26, KTH Campus, Stockholm 

QC 20230523

Available from: 2023-05-23 Created: 2023-05-22 Last updated: 2023-06-19Bibliographically approved
List of papers
1. Demo: Scaling on the Edge – A Benchmarking Suite for Human-in-the-Loop Applications
Open this publication in new window or tab >>Demo: Scaling on the Edge – A Benchmarking Suite for Human-in-the-Loop Applications
2018 (English)In: 2018 IEEE/ACM Symposium on Edge Computing (SEC), IEEE, 2018, p. 323-325, article id 08567677Conference paper, Published paper (Refereed)
Abstract [en]

Benchmarking human-in-the-loop appli-cations running on edge computing in-frastructure is complex given their nature,which heavily depends on the actions takenby the human user. This limits reproducibil-ity as well as feasibility of performance eval-uations. We propose a methodology andpresent a benchmarking suite that can ad-dress these challenges. Our core idea restson recording traces of these applicationswhich are played out in a controlled fashionbased on an underlying model of human be-havior. The traces are exposed to the origi-nal backend compute process of the respec-tive human-in-the-loop application, gener-ating realistic feedback. This allows for anautomated system which greatly simplifiesbenchmarking large scale scenarios.

Place, publisher, year, edition, pages
IEEE, 2018
Keywords
Human-in-the-Loop, Edge Computing, Cognitive Assistance, Performance Evaluation, Scaling of Distributed Systems
National Category
Communication Systems Computer Systems
Research subject
Computer Science; Electrical Engineering
Identifiers
urn:nbn:se:kth:diva-241113 (URN)10.1109/sec.2018.00031 (DOI)000458816000023 ()2-s2.0-85060194661 (Scopus ID)9781538694459 (ISBN)
Conference
3rd ACM/IEEE Symposium on Edge Computing, SEC 2018; Bellevue; United States; 25 October 2018 through 27 October 2018
Note

QC 20190313

Available from: 2019-01-11 Created: 2019-01-11 Last updated: 2024-03-18Bibliographically approved
2. EdgeDroid: An Experimental Approach to Benchmarking Human-in-the-Loop Applications
Open this publication in new window or tab >>EdgeDroid: An Experimental Approach to Benchmarking Human-in-the-Loop Applications
2019 (English)In: Proceedings of the 20th International Workshop on Mobile Computing Systems and Applications (HotMobile '19), Association for Computing Machinery (ACM), 2019Conference paper, Published paper (Refereed)
Abstract [en]

Many emerging mobile applications, including augmented reality (AR) and wearable cognitive assistance (WCA), aim to provide seamless user interaction. However, the complexity of benchmarking these human-in-the-loop applications limits reproducibility and makes performance evaluation difficult. In this paper, we present EdgeDroid, a benchmarking suite designed to reproducibly evaluate these applications.Our core idea rests on recording traces of user interaction, which are then replayed at benchmarking time in a controlled fashion based on an underlying model of human behavior. This allows for an automated system that greatly simplifies benchmarking large scale scenarios and stress testing the application.Our results show the benefits of EdgeDroid as a tool for both system designers and application developers.

Place, publisher, year, edition, pages
Association for Computing Machinery (ACM), 2019
Keywords
Human-in-the-Loop, Edge Computing, Cognitive Assistance, Benchmarking, Cloudlet
National Category
Computer Systems
Research subject
Computer Science
Identifiers
urn:nbn:se:kth:diva-241433 (URN)10.1145/3301293.3302353 (DOI)000473097800017 ()2-s2.0-85062861413 (Scopus ID)978-1-4503-6273-3 (ISBN)
Conference
The 20th International Workshop on Mobile Computing Systems and Applications (HotMobile '19)
Note

QC 20190121

Available from: 2019-01-21 Created: 2019-01-21 Last updated: 2024-03-18Bibliographically approved
3. Impact of delayed response on wearable cognitive assistance
Open this publication in new window or tab >>Impact of delayed response on wearable cognitive assistance
Show others...
2021 (English)In: PLOS ONE, E-ISSN 1932-6203, Vol. 16, no 3, p. e0248690-Article in journal (Refereed) Published
Abstract [en]

Wearable cognitive assistants (WCA) are anticipated to become a widely-used application class, in conjunction with emerging network infrastructures like 5G that incorporate edge computing capabilities. While prototypical studies of such applications exist today, the relationship between infrastructure service provisioning and its implication for WCA usability is largely unexplored despite the relevance that these applications have for future networks. This paper presents an experimental study assessing how WCA users react to varying end-to-end delays induced by the application pipeline or infrastructure. Participants interacted directly with an instrumented task-guidance WCA as delays were introduced into the system in a controllable fashion. System and task state were tracked in real time, and biometric data from wearable sensors on the participants were recorded. Our results show that periods of extended system delay cause users to correspondingly (and substantially) slow down in their guided task execution, an effect that persists for a time after the system returns to a more responsive state. Furthermore, the slow-down in task execution is correlated with a personality trait, neuroticism, associated with intolerance for time delays. We show that our results implicate impaired cognitive planning, as contrasted with resource depletion or emotional arousal, as the reason for slowed user task executions under system delay. The findings have several implications for the design and operation of WCA applications as well as computational and communication infrastructure, and additionally for the development of performance analysis tools for WCA.

Place, publisher, year, edition, pages
Public Library of Science (PLoS), 2021
Keywords
wearable cognitive assistance, delay, latency, augmented reality
National Category
Computer Systems Telecommunications Communication Systems
Identifiers
urn:nbn:se:kth:diva-292036 (URN)10.1371/journal.pone.0248690 (DOI)000632916600038 ()33755667 (PubMedID)2-s2.0-85103205541 (Scopus ID)
Funder
Swedish Foundation for Strategic Research , ITM17-0246
Note

QC 20210324

Available from: 2021-03-23 Created: 2021-03-23 Last updated: 2023-05-23Bibliographically approved
4. CLEAVE: scalable and edge-native benchmarking of networked control systems
Open this publication in new window or tab >>CLEAVE: scalable and edge-native benchmarking of networked control systems
2022 (English)In: EdgeSys '22: Proceedings of the 5th International Workshop on Edge Systems, Analytics and Networking / [ed] Aaron Ding, Volker Hilt, Rennes, France: Association for Computing Machinery (ACM) , 2022, p. 37-42Conference paper, Published paper (Refereed)
Abstract [en]

As the number of cyber-physical systems rises, it becomes increasingly crucial to study Networked Control Systems (NCSs) combining control communication co-design. This nature of NCSs has led to task-specific approaches to research, creating a dearth of generalizable, repeatable, and scalable experimentation. Further, with the advent of edge computing solutions, it is of paramount importance to explore its relevance in such applications. In this work, we present CLEAVE, a novel, completely software-based framework for repeatable and scalable experimentation in edge native NCSs. Our approach is based on the emulation of physical plants communicating over a real network with software-based controllers. CLEAVE is designed and built for the edge, using Python3 and with full compatibility with industry-standard containerization solutions. Although designed for single-loop emulations, the flexibility afforded by the aforementioned characteristics allow our framework to be adapted to a multitude of complex scenarios.

We validate CLEAVE using an initial implementation of an inverted pendulum NCS. Our results showcase the utility of the tool as a repeatable, extensible, and scalable solution to NCS performance evaluation and benchmarking on the Edge.

Place, publisher, year, edition, pages
Rennes, France: Association for Computing Machinery (ACM), 2022
Keywords
Networked Control System, NCS, Edge Computing, Cloud Computing, Benchmarking, Control Systems, Distributed Control Systems, Emulation
National Category
Computer Systems
Research subject
Computer Science
Identifiers
urn:nbn:se:kth:diva-310744 (URN)10.1145/3517206.3526272 (DOI)000927589500007 ()2-s2.0-85128415906 (Scopus ID)
Conference
EdgeSys@EuroSys 2022: Proceedings of the 5th International Workshop on Edge Systems, Analytics and Networking, Rennes, France, April 5-8, 2022
Funder
Swedish Foundation for Strategic Research, ITM17–0246
Note

Part of proceedings: ISBN 978-1-4503-9253-2

QC 20220428

Available from: 2022-04-06 Created: 2022-04-06 Last updated: 2023-05-23Bibliographically approved
5. Ainur: A Framework for Repeatable End-to-End Wireless Edge Computing Testbed Research
Open this publication in new window or tab >>Ainur: A Framework for Repeatable End-to-End Wireless Edge Computing Testbed Research
2022 (English)In: European Wireless Conference, EW 2022, VDE VERLAG GMBH , 2022, p. 139-145Conference paper, Published paper (Refereed)
Abstract [en]

Experimental research on wireless networking in combination with edge and cloud computing has been the subject of explosive interest in the last decade. This development has been driven by the increasing complexity of modern wireless technologies and the extensive softwarization of these through projects such as a Open Radio Access Network (O-RAN). In this context, a number of small- to mid-scale testbeds have emerged, employing a variety of technologies to target a wide array of use-cases and scenarios in the context of novel mobile communication technologies such as 5G and beyond-5G. Little work, however, has yet been devoted to developing a standard framework for wireless testbed automation which is hardwareagnostic and compatible with edge- and cloud-native technologies. Such a solution would simplify the development of new testbeds by completely or partially removing the requirement for custom management and orchestration software. In this paper, we present the first such mostly hardwareagnostic wireless testbed automation framework, Ainur. It is designed to configure, manage, orchestrate, and deploy workloads from an end-to-end perspective. Ainur is built on top of cloudnative technologies such as Docker, and is provided as FOSS to the community through the KTH-EXPECA/Ainur repository on GitHub. We demonstrate the utility of the platform with a series of scenarios, showcasing in particular its flexibility with respect to physical link definition, computation placement, and automation of arbitrarily complex experimental scenarios.

Place, publisher, year, edition, pages
VDE VERLAG GMBH, 2022
Keywords
Edge Computing, Testbed, Automation, Experimental Research
National Category
Computer Systems
Research subject
Computer Science
Identifiers
urn:nbn:se:kth:diva-327217 (URN)2-s2.0-85172030303 (Scopus ID)
Conference
2022 European Wireless Conference, EW 2022, Dresden, Germany, Sep 19 2022 - Sep 21 2022
Funder
Swedish Foundation for Strategic Research, ITM17–0246
Note

Part of ISBN 9781713865698

QC 20230525

Available from: 2023-05-22 Created: 2023-05-22 Last updated: 2023-10-09Bibliographically approved
6. Realistic Modeling of Human Timings for Wearable Cognitive Assistance
Open this publication in new window or tab >>Realistic Modeling of Human Timings for Wearable Cognitive Assistance
Show others...
(English)Manuscript (preprint) (Other academic)
Abstract [en]

Wearable Cognitive Assistance (WCA) applications present a challenge to benchmark and characterize due to their human-in-the-loop nature. Employing user testing to optimize system parameters is generally not feasible, given the scope of the problem and the number of observations needed to detect small but important effects in controlled experiments. Considering the intended mass-scale deployment of WCA applications in the future, there exists a need for tools enabling human-independent benchmarking.We present in this paper the first model for the complete end-to-end emulation of humans in WCA. We build this model through statistical analysis of data collected from previous work in this field, and demonstrate its utility by studying application task durations. Compared to first-order approximations, our model shows a ~36% larger gap between step execution times at high system impairment versus low. We further introduce a novel framework for stochastic optimization of resource consumption-responsiveness tradeoffs in WCA, and show that by combining this framework with our realistic model of human behavior, significant reductions of up to 50% in number processed frame samples and 20% in energy consumption can be achieved with respect to the state-of-the-art. 

Keywords
Edge Computing, Wearable Cognitive Assistance, Modeling, Human Modeling
National Category
Computer Systems
Research subject
Computer Science
Identifiers
urn:nbn:se:kth:diva-327218 (URN)10.48550/arXiv.2212.06100 (DOI)
Funder
Swedish Foundation for Strategic Research, ITM17–0246
Note

QC 20230525

Available from: 2023-05-22 Created: 2023-05-22 Last updated: 2023-05-25Bibliographically approved

Open Access in DiVA

summary(5970 kB)667 downloads
File information
File name FULLTEXT02.pdfFile size 5970 kBChecksum SHA-512
41b80167f20bb8e87fdda525072e96dd4c033b9d0d0a68cb24884da59e342b9e97aa77c8e5c504a25e8bef3d6dfc176e407f0d2dff8cf69a0d592c4a00d4182a
Type fulltextMimetype application/pdf

Authority records

Olguín Muñoz, Manuel Osvaldo Jesús

Search in DiVA

By author/editor
Olguín Muñoz, Manuel Osvaldo Jesús
By organisation
Information Science and Engineering
Computer Systems

Search outside of DiVA

GoogleGoogle Scholar
Total: 668 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 1452 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf