kth.sePublications KTH
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
A Unified Framework for Real-Time Failure Handling in Robotics Using Vision-Language Models, Reactive Planner and Behavior Trees
Lund University, Lund, Sweden.
Lund University, Lund, Sweden.
KTH, School of Electrical Engineering and Computer Science (EECS), Robotics, Perception and Learning. ABB Robotics, Västerås, Sweden.ORCID iD: 0000-0003-0312-8811
Lund University, Lund, Sweden.
Show others and affiliations
2025 (English)In: 2025 IEEE 21St International Conference On Automation Science And Engineering, Case, Institute of Electrical and Electronics Engineers (IEEE) , 2025, p. 887-894Conference paper, Published paper (Refereed)
Abstract [en]

Robotic systems often face execution failures due to unexpected obstacles, sensor errors, or environmental changes. Traditional failure recovery methods rely on predefined strategies or human intervention, making them less adaptable. This paper presents a unified failure recovery framework that combines Vision-Language Models (VLMs), a reactive planner, and Behavior Trees (BTs) to enable real-time failure handling. Our approach includes pre-execution verification, which checks for potential failures before execution, and reactive failure handling, which detects and corrects failures during execution by verifying existing BT conditions, adding missing preconditions and, when necessary, generating new skills. The framework uses a scene graph for structured environmental perception and an execution history for continuous monitoring, enabling context-aware and adaptive failure handling. We evaluate our framework through real-world experiments with an ABB YuMi robot on tasks like peg insertion, object sorting, and drawer placement, as well as in AI2-THOR simulator. Compared to using pre-execution and reactive methods separately, our approach achieves higher task success rates and greater adaptability. Ablation studies highlight the importance of VLM-based reasoning, structured scene representation, and execution history tracking for effective failure recovery in robotics.

Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers (IEEE) , 2025. p. 887-894
Series
IEEE International Conference on Automation Science and Engineering, ISSN 2161-8070
National Category
Robotics and automation
Identifiers
URN: urn:nbn:se:kth:diva-382135DOI: 10.1109/CASE58245.2025.11164021ISI: 001701272600099Scopus ID: 2-s2.0-105018322736ISBN: 979-8-3315-2246-9 (print)OAI: oai:DiVA.org:kth-382135DiVA, id: diva2:2062482
Conference
21st International Conference on Automation Science and Engineering-CASE-Annual, AUG 17-21, 2025, Los Angeles, CA, USA
Note

QC 20260714

Available from: 2026-05-26 Created: 2026-05-26 Last updated: 2026-07-14Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records

Styrud, Jonathan

Search in DiVA

By author/editor
Styrud, Jonathan
By organisation
Robotics, Perception and Learning
Robotics and automation

Search outside of DiVA

GoogleGoogle Scholar

doi
isbn
urn-nbn

Altmetric score

doi
isbn
urn-nbn
Total: 6 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf