kth.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Scalable Artificial Intelligence for Earth Observation Data Using Hopsworks
KTH, School of Electrical Engineering and Computer Science (EECS), Computer Science, Software and Computer systems, SCS.ORCID iD: 0000-0002-2014-2749
Logical Clocks AB, S-11872 Stockholm, Sweden..
KTH, School of Electrical Engineering and Computer Science (EECS), Computer Science, Software and Computer systems, SCS.ORCID iD: 0000-0001-7236-4637
KTH, School of Electrical Engineering and Computer Science (EECS), Computer Science, Software and Computer systems, SCS.
Show others and affiliations
2022 (English)In: Remote Sensing, E-ISSN 2072-4292, Vol. 14, no 8, article id 1889Article in journal (Refereed) Published
Abstract [en]

This paper introduces the Hopsworks platform to the entire Earth Observation (EO) data community and the Copernicus programme. Hopsworks is a scalable data-intensive open-source Artificial Intelligence (AI) platform that was jointly developed by Logical Clocks and the KTH Royal Institute of Technology for building end-to-end Machine Learning (ML)/Deep Learning (DL) pipelines for EO data. It provides the full stack of services needed to manage the entire life cycle of data in ML. In particular, Hopsworks supports the development of horizontally scalable DL applications in notebooks and the operation of workflows to support those applications, including parallel data processing, model training, and model deployment at scale. To the best of our knowledge, this is the first work that demonstrates the services and features of the Hopsworks platform, which provide users with the means to build scalable end-to-end ML/DL pipelines for EO data, as well as support for the discovery and search for EO metadata. This paper serves as a demonstration and walkthrough of the stages of building a production-level model that includes data ingestion, data preparation, feature extraction, model training, model serving, and monitoring. To this end, we provide a practical example that demonstrates the aforementioned stages with real-world EO data and includes source code that implements the functionality of the platform. We also perform an experimental evaluation of two frameworks built on top of Hopsworks, namely Maggy and AutoAblation. We show that using Maggy for hyperparameter tuning results in roughly half the wall-clock time required to execute the same number of hyperparameter tuning trials using Spark while providing linear scalability as more workers are added. Furthermore, we demonstrate how AutoAblation facilitates the definition of ablation studies and enables the asynchronous parallel execution of ablation trials.

Place, publisher, year, edition, pages
MDPI AG , 2022. Vol. 14, no 8, article id 1889
Keywords [en]
Hopsworks, Copernicus, Earth Observation, machine learning, deep learning, artificial intelligence, model serving, big data, ablation studies, Maggy, ExtremeEarth
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:kth:diva-311886DOI: 10.3390/rs14081889ISI: 000787403900001Scopus ID: 2-s2.0-85129027995OAI: oai:DiVA.org:kth-311886DiVA, id: diva2:1656656
Note

QC 20220506

Available from: 2022-05-06 Created: 2022-05-06 Last updated: 2023-08-28Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records

Hagos, Desta HaileselassieSheikholeslami, SinaWang, TianzeVlassov, VladimirPayberah, Amir HosseinDowling, Jim

Search in DiVA

By author/editor
Hagos, Desta HaileselassieSheikholeslami, SinaWang, TianzeVlassov, VladimirPayberah, Amir HosseinDowling, Jim
By organisation
Software and Computer systems, SCS
In the same journal
Remote Sensing
Computer Sciences

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 141 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf