Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Exploration in Structured Reinforcement Learning
KTH, School of Electrical Engineering and Computer Science (EECS), Automatic Control.
KTH, School of Electrical Engineering and Computer Science (EECS), Automatic Control.
KTH, School of Electrical Engineering and Computer Science (EECS), Automatic Control.
2018 (English)In: Advances In Neural Information Processing Systems 31 (NIPS 2018) / [ed] Bengio, S Wallach, H Larochelle, H Grauman, K CesaBianchi, N Garnett, R, Neural Information Processing Systems (NIPS) , 2018, Vol. 31Conference paper, Published paper (Refereed)
Abstract [en]

We address reinforcement learning problems with finite state and action spaces where the underlying MDP has some known structure that could be potentially exploited to minimize the exploration rates of suboptimal (state, action) pairs. For any arbitrary structure, we derive problem-specific regret lower bounds satisfied by any learning algorithm. These lower bounds are made explicit for unstructured MDPs and for those whose transition probabilities and average reward functions are Lipschitz continuous w.r.t. the state and action. For Lipschitz MDPs, the bounds are shown not to scale with the sizes S and A of the state and action spaces, i.e., they are smaller than c log T where T is the time horizon and the constant c only depends on the Lipschitz structure, the span of the bias function, and the minimal action sub-optimality gap. This contrasts with unstructured MDPs where the regret lower bound typically scales as SA log T. We devise DEL (Directed Exploration Learning), an algorithm that matches our regret lower bounds. We further simplify the algorithm for Lipschitz MDPs, and show that the simplified version is still able to efficiently exploit the structure.

Place, publisher, year, edition, pages
Neural Information Processing Systems (NIPS) , 2018. Vol. 31
Series
Advances in Neural Information Processing Systems, ISSN 1049-5258 ; 31
National Category
Control Engineering
Identifiers
URN: urn:nbn:se:kth:diva-249916ISI: 000461852003043OAI: oai:DiVA.org:kth-249916DiVA, id: diva2:1306838
Conference
32nd Conference on Neural Information Processing Systems (NIPS), DEC 02-08, 2018, Montreal, CANADA
Note

QC 20190425

Available from: 2019-04-25 Created: 2019-04-25 Last updated: 2019-04-25Bibliographically approved

Open Access in DiVA

No full text in DiVA

Authority records BETA

Ok, JungseulProutiere, AlexandreTranos, Damianos

Search in DiVA

By author/editor
Ok, JungseulProutiere, AlexandreTranos, Damianos
By organisation
Automatic Control
Control Engineering

Search outside of DiVA

GoogleGoogle Scholar

urn-nbn

Altmetric score

urn-nbn
Total: 15 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf