skip to main content
10.1145/1390156.1390162acmotherconferencesArticle/Chapter ViewAbstractPublication PagesicmlConference Proceedingsconference-collections
research-article

Learning all optimal policies with multiple criteria

Published: 05 July 2008 Publication History

Abstract

We describe an algorithm for learning in the presence of multiple criteria. Our technique generalizes previous approaches in that it can learn optimal policies for all linear preference assignments over the multiple reward criteria at once. The algorithm can be viewed as an extension to standard reinforcement learning for MDPs where instead of repeatedly backing up maximal expected rewards, we back up the set of expected rewards that are maximal for some set of linear preferences (given by a weight vector, w). We present the algorithm along with a proof of correctness showing that our solution gives the optimal policy for any linear preference function. The solution reduces to the standard value iteration algorithm for a specific weight vector, w.

References

[1]
Abeel, P., & Ng, A. (2004). Apprentice learning via inverse reinforcement learning. Proc. ICML-04.
[2]
Ainslie, G. (2001). Breakdown of will. Cambridge, Massachusetts: Cambridge University Press.
[3]
Bellman, R. E. (1957). Dynamic programming. Princeton: Princeton University Press.
[4]
Clarkson, K. L., & Shor, P. W. (1989). Applications of random sampling in computational geometry, II. Discrete and Computational Geometry, 4, 387--421.
[5]
Feinberg, E., & Schwartz, A. (1995). Constrained markov decision models with weighted discounted rewards. Mathematics of Operations Research, 20, 302--320.
[6]
Gabor, Z., Kalmar, Z., & Szepesvari, C. (1998). Multi-criteria reinforcement learning. Proc. ICML-98.
[7]
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998). Planning and acting in partially observable stochastic domains. Artificial Intelligence.
[8]
Mannor, S., & Shimkin, N. (2004). A geometric approach to multi-criterion reinforcement learning. Journal of Machine Learning Research, 325--360.
[9]
Natarajan, S., & Tadepalli, P. (2005). Dynamic preferences in mult-criteria reinforcement learning. Proc. ICML-05. Bonn, Germany.
[10]
Ng, A., & Russell, S. (2000). Algorithms for inverse reinforcement learning. Proc. ICML-00.
[11]
Russell, S., & Zimdars, A. (2003). Q-decomposition for reinforcement learning agents. Proc. ICML-03. Washington, DC.
[12]
Sutton, R., & Barto, A. (1998). Reinforcement learning: An introduction. Cambridge, Massachusetts: The MIT Press.

Cited By

View all
  • (2024)Finite-time convergence and sample complexity of actor-critic multi-objective reinforcement learningProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694632(61913-61933)Online publication date: 21-Jul-2024
  • (2024)Rewards-in-contextProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694392(56276-56297)Online publication date: 21-Jul-2024
  • (2024)Estimating Objective Weights of Pareto-Optimal Policies for Multi-Objective Sequential Decision-MakingJournal of Advanced Computational Intelligence and Intelligent Informatics10.20965/jaciii.2024.p039328:2(393-402)Online publication date: 20-Mar-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
ICML '08: Proceedings of the 25th international conference on Machine learning
July 2008
1310 pages
ISBN:9781605582054
DOI:10.1145/1390156
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

  • Pascal
  • University of Helsinki
  • Xerox
  • Federation of Finnish Learned Societies
  • Google Inc.
  • NSF
  • Machine Learning Journal/Springer
  • Microsoft Research: Microsoft Research
  • Intel: Intel
  • Yahoo!
  • Helsinki Institute for Information Technology
  • IBM: IBM

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 July 2008

Permissions

Request permissions for this article.

Check for updates

Qualifiers

  • Research-article

Conference

ICML '08
Sponsor:
  • Microsoft Research
  • Intel
  • IBM

Acceptance Rates

Overall Acceptance Rate 140 of 548 submissions, 26%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)116
  • Downloads (Last 6 weeks)12
Reflects downloads up to 07 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Finite-time convergence and sample complexity of actor-critic multi-objective reinforcement learningProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694632(61913-61933)Online publication date: 21-Jul-2024
  • (2024)Rewards-in-contextProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694392(56276-56297)Online publication date: 21-Jul-2024
  • (2024)Estimating Objective Weights of Pareto-Optimal Policies for Multi-Objective Sequential Decision-MakingJournal of Advanced Computational Intelligence and Intelligent Informatics10.20965/jaciii.2024.p039328:2(393-402)Online publication date: 20-Mar-2024
  • (2024)Graph Convolutional Network Based Multi-Objective Meta-Deep Q-Learning for Eco-RoutingIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2023.334803425:7(7323-7338)Online publication date: Jul-2024
  • (2024)On Minimizing Total Discounted Cost in MDPs Subject to Reachability ConstraintsIEEE Transactions on Automatic Control10.1109/TAC.2024.338483469:9(6466-6473)Online publication date: Sep-2024
  • (2024)Pruning the Way to Reliable Policies: A Multi-Objective Deep Q-Learning Approach to Critical CareIEEE Journal of Biomedical and Health Informatics10.1109/JBHI.2024.341511528:10(6268-6279)Online publication date: Oct-2024
  • (2024)Interactive Reward Tuning: Interactive Visualization for Preference Elicitation2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)10.1109/IROS58592.2024.10801540(9254-9261)Online publication date: 14-Oct-2024
  • (2024)Generalizing knowledge enabled fast-adaptive optimization for advanced machining systems2024 IEEE 20th International Conference on Automation Science and Engineering (CASE)10.1109/CASE59546.2024.10711708(1650-1655)Online publication date: 28-Aug-2024
  • (2024)Reward Shaping in Reinforcement Learning of Multi-Objective Safety Critical Systems2024 20th CSI International Symposium on Artificial Intelligence and Signal Processing (AISP)10.1109/AISP61396.2024.10475272(1-6)Online publication date: 21-Feb-2024
  • (2024)Multiobjective tree-based reinforcement learning for estimating tolerant dynamic treatment regimesBiometrics10.1093/biomtc/ujad01780:1Online publication date: 14-Feb-2024
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media