kth.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Towards Trustworthy Machine Learning For Human Activity Recognition
KTH, School of Electrical Engineering and Computer Science (EECS), Computer Science, Software and Computer systems, SCS.ORCID iD: 0000-0001-6780-7755
2024 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Human Activity Recognition presents a multifaceted challenge, encompassing the complexity of human activities, the diversity of sensors used, and the imperative to safeguard user data privacy. Recent advancements in machine learning, deep learning, and sensor technology have opened up new possibilities for human activity recognition. Wearable sensor-based human activity recognition involves collecting time-series data from various sensors, capturing intricate aspects of human activities. The focus of the above activity recognition problem is classifying human activities from the time-series data. Hence, this time-series classification problem demands efficient utilization of temporal properties. Moreover, while accurate prediction is crucial in human activity recognition, the reliability of predictions often goes unnoticed. Ensuring that predictions are reliable involves addressing two issues: calibrating miscalibrated predictions that fail to accurately represent the true likelihood of the data and addressing the challenges around uncertain predictions. Modern deep learning models, used extensively in human activity recognition, often struggle with the above issues. In addition to reliability concerns, machine learning algorithms employed in Human Activity Recognition are also plagued by privacy issues stemming from the utilization of sensitive activity data during model training. While existing techniques such as federated learning can provide some degree of privacy protection in these scenarios, they tend to adhere to a uniform concept of privacy and lack quantifiable privacy metrics that can be effectively conveyed to users and customized to cater to their individual privacy preferences. Hence, in the thesis, we identify the challenges around the effective use of temporal data, reliability, and privacy issues of machine learning models used for wearable sensor-based human activity recognition. To tackle these challenges, we put forth novel solutions, striving to enhance the overall performance and trustworthiness of machine learning models employed in human activity recognition.

Firstly, to improve classification performance, we propose a new temporal ensembling framework that uses data temporality effectively. The framework accommodates various window sizes for time-series data and trains an ensemble of deep-learning models based on that. It enhances classification accuracy and preserves temporal information.

Secondly, we address reliability through calibration and uncertainty estimation. The aforementioned temporal ensembling framework is used for calibration and uncertainty estimation. It provides well-calibrated predictions for human activity recognition and detects out-of-distribution activities, an important task of uncertainty estimation. Furthermore, we apply these methods to real-world scenarios, enhancing the reliability of human activity recognition models.

Thirdly, to address the privacy concern, we introduce a differentially private framework for time-series human activity recognition, quantifying privacy. Additionally, we develop a collaborative federated learning framework, allowing users to define their privacy preferences, advancing privacy preservation in human activity recognition.

These contributions address major challenges and promote improved classification, reliability, and privacy preservation in human activity recognition. It helps us to move towards trustworthy machine learning in human activity recognition, facilitating their usage in realistic and practical scenarios.

Place, publisher, year, edition, pages
Stockholm: KTH Royal Institute of Technology, 2024. , p. xii, 56
Series
TRITA-EECS-AVL ; 2024:12
National Category
Computer Sciences
Research subject
Computer Science; Information and Communication Technology
Identifiers
URN: urn:nbn:se:kth:diva-343130ISBN: 978-91-8040-826-4 (print)OAI: oai:DiVA.org:kth-343130DiVA, id: diva2:1835806
Public defence
2024-03-06, https://kth-se.zoom.us/j/63687967257, Sal C, Kistagången 16, Kista, Stockholm, 13:00 (English)
Opponent
Supervisors
Funder
EU, Horizon 2020, 813162
Note

QC 20240207

Available from: 2024-02-07 Created: 2024-02-07 Last updated: 2024-02-29Bibliographically approved
List of papers
1. Confidence-Calibrated Human Activity Recognition
Open this publication in new window or tab >>Confidence-Calibrated Human Activity Recognition
2021 (English)In: Sensors, E-ISSN 1424-8220, Vol. 21, no 19, p. 6566-6566Article in journal (Refereed) Published
Abstract [en]

Wearable sensors are widely used in activity recognition (AR) tasks with broad applicability in health and well-being, sports, geriatric care, etc. Deep learning (DL) has been at the forefront of progress in activity classification with wearable sensors. However, most state-of-the-art DL models used for AR are trained to discriminate different activity classes at high accuracy, not considering the confidence calibration of predictive output of those models. This results in probabilistic estimates that might not capture the true likelihood and is thus unreliable. In practice, it tends to produce overconfident estimates. In this paper, the problem is addressed by proposing deep time ensembles, a novel ensembling method capable of producing calibrated confidence estimates from neural network architectures. In particular, the method trains an ensemble of network models with temporal sequences extracted by varying the window size over the input time series and averaging the predictive output. The method is evaluated on four different benchmark HAR datasets and three different neural network architectures. Across all the datasets and architectures, our method shows an improvement in calibration by reducing the expected calibration error (ECE)by at least 40%, thereby providing superior likelihood estimates. In addition to providing reliable predictions our method also outperforms the state-of-the-art classification results in the WISDM, UCI HAR, and PAMAP2 datasets and performs as good as the state-of-the-art in the Skoda dataset. 

Place, publisher, year, edition, pages
Basel: MDPI, 2021
Keywords
Electrical and Electronic Engineering, Biochemistry, Instrumentation, Atomic and Molecular Physics, and Optics, Analytical Chemistry
National Category
Computer Sciences
Research subject
Computer Science
Identifiers
urn:nbn:se:kth:diva-303329 (URN)10.3390/s21196566 (DOI)000778245800002 ()34640886 (PubMedID)2-s2.0-85116672506 (Scopus ID)
Note

QC 20220422

Available from: 2021-10-12 Created: 2021-10-12 Last updated: 2024-02-07Bibliographically approved
2. Mixing temporal experts for Human Activity Recognition
Open this publication in new window or tab >>Mixing temporal experts for Human Activity Recognition
2022 (English)In: 34th Workshop of the Swedish Artificial Intelligence Society, SAIS 2022, Institute of Electrical and Electronics Engineers (IEEE) , 2022, p. 11-18Conference paper, Published paper (Refereed)
Abstract [en]

Temporal patterns are encoded within the time-series data, and neural networks, with their unique feature extraction ability, process those patterns to provide a better predictive response. Ensembles of neural networks have proven to be very effective Human Activity Recognition (HAR) tasks with time-series data, e.g., wearable sensors. The combination of predictions coming from the individual models in the ensemble helps boost the overall classification metric through efficient temporal pattern recognition. Currently, the most common strategy for combining the predictions coming from the individual models is simple averaging. However, since each ensemble model learns different temporal patterns of the timseries classification problem, a simple averaging strategy is sub-optimal. This sub-optimality is addressed in this paper through a neural network-based adaptive learning framework. The method's core is training a neural gate that ingests the same input time-series data fed to the other temporal models. The goal of the training process is to adaptively learn scaler values against each temporal model by looking at the input data. These scaler values weigh each temporal model while combining the ensemble. The framework obtains superior predictive performance as compared to the standard ensembling techniques. The framework is evaluated on a benchmark HAR dataset called PAMAP2 [3] with two popular state-of-the-art ensemble architectures namely DTE [1] and LSTM-ensemble [2]. In both cases, the classification performance of the framework in HAR tasks surpasses the state-of-the-art models.

Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers (IEEE), 2022
National Category
Other Engineering and Technologies
Identifiers
urn:nbn:se:kth:diva-319433 (URN)10.1109/SAIS55783.2022.9833028 (DOI)000855561800002 ()2-s2.0-85136111184 (Scopus ID)
Conference
34th Workshop of the Swedish Artificial Intelligence Society, SAIS 2022, Stockholm, 13 June 2022, through 14 June 2022
Note

QC 20220929

Part of proceedings: ISBN 978-1-6654-7126-8

Available from: 2022-09-29 Created: 2022-09-29 Last updated: 2024-04-12Bibliographically approved
3. Out-of-distribution in Human Activity Recognition
Open this publication in new window or tab >>Out-of-distribution in Human Activity Recognition
2022 (English)In: 34th Workshop of the Swedish Artificial Intelligence Society, SAIS 2022, Institute of Electrical and Electronics Engineers (IEEE) , 2022, p. 1-10Conference paper, Published paper (Refereed)
Abstract [en]

With the growing interest of the research community in making deep learning (DL) robust and reliable, detecting out-of-distribution (OOD) data has become critical. Detecting OOD inputs during test/prediction allows the model to account for discriminative features unknown to the model. This capability increases the model's reliability since this model provides a class prediction solely at incoming data similar to the training one. OOD detection is well established in computer vision problems. However, it remains relatively under-explored in other domains such as time series (i.e., Human Activity Recognition (HAR)). Since uncertainty has been a critical driver for OOD in vision-based models, the same component has proven effective in time-series applications. We plan to address the OOD detection problem in HAR with time-series data in this work. To test the capability of the proposed method, we define different types of OOD for HAR that arise from realistic scenarios. We apply an ensemble-based temporal learning framework that incorporates uncertainty and detects OOD for the defined HAR workloads. In particular, we extract OODs from popular benchmark HAR datasets and use the framework to separate those OODs from the indistribution (ID) data. Across all the datasets, the ensemble framework outperformed the traditional deep-learning method (our baseline) on the OOD detection task.

Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers (IEEE), 2022
National Category
Other Engineering and Technologies
Identifiers
urn:nbn:se:kth:diva-319437 (URN)10.1109/SAIS55783.2022.9833052 (DOI)000855561800001 ()2-s2.0-85136092539 (Scopus ID)
Conference
34th Workshop of the Swedish Artificial Intelligence Society, SAIS 2022, Stockholm, 13 June 2022, through 14 June 2022
Note

Part of proceedings: ISBN 978-1-6654-7126-8

QC 20220929

Available from: 2022-09-29 Created: 2022-09-29 Last updated: 2024-04-12Bibliographically approved
4. Classifying falls using out-of-distribution detection in human activity recognition
Open this publication in new window or tab >>Classifying falls using out-of-distribution detection in human activity recognition
2023 (English)In: AI Communications, ISSN 0921-7126, E-ISSN 1875-8452, Vol. 36, no 4, p. 251-267Article in journal (Refereed) Published
Abstract [en]

As the research community focuses on improving the reliability of deep learning, identifying out-of-distribution (OOD) data has become crucial. Detecting OOD inputs during test/prediction allows the model to account for discriminative features unknown to the model. This capability increases the model's reliability since this model provides a class prediction solely at incoming data similar to the training one. Although OOD detection is well-established in computer vision, it is relatively unexplored in other areas, like time series-based human activity recognition (HAR). Since uncertainty has been a critical driver for OOD in vision-based models, the same component has proven effective in time-series applications. In this work, we propose an ensemble-based temporal learning framework to address the OOD detection problem in HAR with time-series data. First, we define different types of OOD for HAR that arise from realistic scenarios. Then we apply our ensemble-based temporal learning framework incorporating uncertainty to detect OODs for the defined HAR workloads. This particular formulation also allows a novel approach to fall detection. We train our model on non-fall activities and detect falls as OOD. Our method shows state-of-The-Art performance in a fall detection task using much lesser data. Furthermore, the ensemble framework outperformed the traditional deep-learning method (our baseline) on the OOD detection task across all the other chosen datasets.

Place, publisher, year, edition, pages
IOS Press, 2023
Keywords
deep learning, human activity recognition, Out-of-distribution detection, time-series classification, uncertainty estimation
National Category
Computer Sciences Bioinformatics (Computational Biology)
Identifiers
urn:nbn:se:kth:diva-339522 (URN)10.3233/AIC-220205 (DOI)001087274200001 ()2-s2.0-85175210057 (Scopus ID)
Note

QC 20231114

Available from: 2023-11-14 Created: 2023-11-14 Last updated: 2024-04-11Bibliographically approved
5. Temporal Differential Privacy for Human Activity Recognition
Open this publication in new window or tab >>Temporal Differential Privacy for Human Activity Recognition
2023 (English)Conference paper, Published paper (Refereed)
Abstract [en]

Differential privacy (DP) is a method to protect individual privacy when the data is used for downstream analytical tasks. The core ability of DP to quantity privacy numerically separates it from other privacy-preserving methods. In human activity recognition (HAR), differential privacy can protect users’ privacy who contribute their data to train machine learning algorithms. While some methods are developed for privacy protection in such cases, no method quantifies privacy and seamlessly integrates into machine learning frameworks like DP. The paper proposes a DP framework called TEMPDIFF (short for temporal differential privacy), which guarantees privacy preserving human activity recognition for wearable time-series data with competitive classification performance and works with any machine-learning/deep-learning methods. TEMPDIFF capitalizes on the temporal characteristics of wearable sensor data to improve the modelling task, which enhances the privacy-utility tradeoff. TEMPDIFF uses ensembling and a novel temporal partitioning algorithm for time-series data to ensure optimal training of ensemble models. In TEMPDIFF, consensus through ensembling and the addition of controlled Laplacian noise obscures sensitive information used to train the models, guaranteeing strict levels of differential privacy. The proposed method is evaluated on two popular HAR datasets. It outperforms the classification accuracy and privacy budget for both datasets compared to the state-of-the-art approaches.

Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers (IEEE), 2023
Keywords
Differential Privacy, Machine Learning, Human Activity Recognition
National Category
Engineering and Technology
Research subject
Computer Science
Identifiers
urn:nbn:se:kth:diva-339761 (URN)10.1109/DSAA60987.2023.10302475 (DOI)
Conference
2023 IEEE 10th International Conference on Data Science and Advanced Analytics (DSAA), Thessaloniki, Greece, 9 - 13 October 2023
Funder
EU, Horizon 2020, 813162
Note

Part of ISBN 979-8-3503-4503-2

QC 20231117

Available from: 2023-11-17 Created: 2023-11-17 Last updated: 2024-02-07Bibliographically approved
6. Private, Fair and Secure Collaborative Learning Framework for Human Activity Recognition
Open this publication in new window or tab >>Private, Fair and Secure Collaborative Learning Framework for Human Activity Recognition
Show others...
2023 (English)In: UbiComp/ISWC '23 Adjunct: Adjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing, Cancun: Association for Computing Machinery (ACM) , 2023, p. 352-358Conference paper, Published paper (Refereed)
Abstract [en]

Federated learning (FL), a decentralized machine learning technique, enhances privacy by enabling multiple devices to collaboratively train a model without transferring data to a central server. FL is used in Human Activity Recognition (HAR) problems, where multiple users generating private wearable data share models with a server to learn a useful global model. However, FL may compromise data privacy through model information sharing during training. Moreover, it adheres to a one-size-fits-all approach toward data privacy, potentially neglecting varied user preferences in collaborative scenarios such as HAR. In response to these challenges, this paper presents a collaborative learning framework integrating differential privacy (DP) and FL, thus providing a tailored approach to privacy protection. While some existing works integrate DP and FL, they do not allow clients to have different privacy preferences. In this work, we introduce a framework that allows different clients to have different privacy preferences and hence more flexibility in terms of privacy. In our framework, DP adds individualized noise to individual clients’ gradient updates for privacy. However, such noised updates can also be interpreted as an attack on the FL system. Defending against these attacks might result in excluding honest private clients altogether from training, posing a fairness concern. On the other hand, not having any defensive measures might allow malicious users to attack the system, posing a security issue. Thus, to address security and fairness, our framework incorporates a client selection strategy that protects the global model from malicious clients and provides fair model access to honest private clients. We have demonstrated the effectiveness of our system on a HAR dataset and provided insights into our framework’s privacy, utility, and fairness.

Place, publisher, year, edition, pages
Cancun: Association for Computing Machinery (ACM), 2023
Keywords
Privacy, Security, Machine Learning, Federated Learning
National Category
Engineering and Technology
Research subject
Computer Science
Identifiers
urn:nbn:se:kth:diva-339762 (URN)10.1145/3594739.3610675 (DOI)2-s2.0-85175448979 (Scopus ID)
Conference
2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2023 ACM International Symposium on Wearable Computing, UbiComp/ISWC 2023, Cancun, Quintana Roo, 8 October 2023
Note

Part of ISBN 9798400702006

QC 20231117

Available from: 2023-11-17 Created: 2023-11-17 Last updated: 2024-02-07Bibliographically approved
7. Kavach: A personalized secure and private decentralized learning setup for Human Activity Recognition
Open this publication in new window or tab >>Kavach: A personalized secure and private decentralized learning setup for Human Activity Recognition
Show others...
(English)Manuscript (preprint) (Other academic)
Abstract [en]

Federated learning (FL) stands as a crucial method in preserving the data privacy of individuals who actively contribute to the machine learning task of Human Activity Recognition (HAR) through wearable devices.Although FL provides a degree of privacy protection, it's crucial to acknowledge that it may not always guarantee robust models and data privacy in specific scenarios. Moreover, FL typically ensures privacy by keeping data localized within user infrastructure, offering a uniform perspective on data privacy.To tackle these challenges, researchers are increasingly focusing on integrating privacy-preserving techniques like Differential Privacy (DP), Homomorphic Encryption (HE), and Trusted Execution Environments (TEE) into the FL framework. However, integrating these techniques directly can introduce their own set of challenges.For instance, DP, while effective in preserving privacy, can disrupt the learning process if excessive noise is added, which is particularly challenging in FL due to data heterogeneity. HE offers strong privacy guarantees but is constrained by computational complexity, and TEEs face scalability issues in practical implementations.To strike a balance between privacy and utility, we introduce the \emph{Kavach} framework designed to seamlessly integrate specific privacy-preserving methods while accommodating diverse privacy preferences. In this framework, privacy is recognized as a non-uniform concept, where privacy does not arise only by withholding data but also by incorporating different privacy preferences and privacy-preserving methods. With respect to the non-uniformity, the quantification of privacy by users plays a crucial role.Kavach leverages DP's unique ability to quantify privacy and tailor privacy budgets to individual user needs. Additionally, the framework allows integration of other privacy methods like encryption or TEEs on a limited scale, reserving them for specific requirements.The primary goal of \textit{Kavach} is to provide a personalized privacy experience for both clients and system designers. It offers flexibility, enabling users to choose their preferred privacy settings and methods, ensuring a customized and effective approach to privacy preservation within collaborative learning.In this work, we present different variants of the \textit{Kavach} framework and showcase experiments on benchmark Human Activity Recognition (HAR) datasets, as well as in the computer vision domain. These experiments highlight the privacy-utility tradeoff and classification performance, demonstrating the framework's effectiveness in achieving personalized privacy while maintaining utility.

Keywords
Machine Learning, Differential Privacy, Federated Learning
National Category
Computer Sciences
Identifiers
urn:nbn:se:kth:diva-342079 (URN)
Funder
EU, Horizon 2020, 813162
Note

QC 20240115

Available from: 2024-01-11 Created: 2024-01-11 Last updated: 2024-02-07Bibliographically approved

Open Access in DiVA

fulltext(8955 kB)127 downloads
File information
File name FULLTEXT01.pdfFile size 8955 kBChecksum SHA-512
819c422b8769bf5489282bc00dd9adf00ffe3d7c09e2a8a085b2266f4e857bf6c872c1cb65c53c5a541c423e7ba747df5d51e8ce9bdc34ccfa460d2addf601d4
Type fulltextMimetype application/pdf

Authority records

Roy, Debaditya

Search in DiVA

By author/editor
Roy, Debaditya
By organisation
Software and Computer systems, SCS
Computer Sciences

Search outside of DiVA

GoogleGoogle Scholar
Total: 127 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 1280 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf