بهبود سرعت آموزش در مسائل یادگیری تقویتی مبتنی بر انتقال دانش عصبی‌فازی

نوع مقاله : علمی-پژوهشی

نویسندگان

دانشکده مهندسی کامپیوتر - دانشگاه علم و هنر

چکیده

این مقاله به موضوع انتقال یادگیری در محیط­هایی که بعضی از ویژگی­های آن مشترک است می­پردازد. چالش اصلی در این مبحث، نحوه انتقال دانش به­دست­آمده از محیط مبدأ به محیط مقصد است. در ایده ارائه­شده با در نظر­گرفتن ویژگی­های مشترک در فضای عامل بین دو محیط، ابتدا مقدار ارزش - عمل در محیط مبدأ به­دست می­آید، سپس از یک شبکه عصبی- فازی برای تقریب مقدار تابع ارزش - عمل بهره برده می‏شود. در محیط مقصد، مقدار ارزش - عمل از ترکیب مقدار پیش­بینی شبکه عصبی - فازی و مقدار به­دست­آمده در خود آن محیط استفاده می­شود. به‌عبارت دیگر با توجه به آموزش انجام‌شده در محیط مبدأ، مقادیر ارزش - عمل در محیط مقصد از ترکیب مقادیر ارزش - عمل تقریب­زده­شده توسط شبکه عصبی - فازی و مقدار به­دست آمده از الگوریتم یادگیری در آن محیط به­دست می­آید. شایان ذکر است که از الگوریتم یادگیری Q در محیط استفاده‌شده است. نتایج حاصل از ایده ارائه‌شده، حاکی از افزایش چشمگیر سرعت یادگیری می­باشد.

کلیدواژه‌ها

عنوان مقاله [English]

Improving the learning speed in reinforcement learning issues based on the transfer learning of neuro-fuzzy knowledge

نویسندگان [English]

  • F. Saadatjoo
  • E. Ghandehari
Computer Engineering Department, Science and Art University, Yazd, Iran
چکیده [English]

This paper to the topic of transfer learning in environments that share some of its features. The main challenge in this topic is how to transfer knowledge from the source environment to the target environment. In the presented idea, taking into account the common features in the operating space between the two environments, the value of the operation in the source environment first is obtained and then it uses a neuro -fuzzy network to approximate the value of the value function of the operation. In the target environment, the value of the mode of operation is used to combine the predictive value of the neuro - fuzzy network and the amount received in the environment itself. In other words, according to the training carried out in the source environment, value-action values ​​in the target environment are derived from the combination of value-action values ​​approximated by the neuro - fuzzy network and the amount obtained from the learning algorithm in that environment. It is worth noting that the learning algorithm Q is used in the environment. The results of the proposed idea indicate a significant increase in learning speed.

کلیدواژه‌ها [English]

  • Reinforcement learning
  • transfer knowledge
  • common features
  • neuro - fuzzy network

مراجع

[1]      W.Böhmer, J. T.Springenberg, J.Boedecker, M.Riedmiller and K.Obermayer, Autonomous learning of state representations for control: An emerging field aims to autonomously learn state representations for reinforcement learning agents from their real-world sensor observations, KI-Künstliche Intelligenz, vol. 29, no. 4, pp. 353-362, 2015.
[2]      J. Kober, J. A. Bagnell and J. Peters, Reinforcement learning in robotics: A survey, International Journal of Robotics Research, vol. 32, no. 11, pp. 1238-1274, 2013.
[3]      T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y.Tassa and D.Wierstra, Continuous control with deep reinforcement learning,arXiv preprint arXiv:1509.02971, 2015.
 
[4]      عادل اکبری مجد، حسین شایقی، حمید محمد نژاد، عبدااله یونسی، کنترل­کننده مقاوم تطبیقی بار فرکانس مبتنی بر یادگیری تقویتی برای یک سیستم قدرت به هم پیوسته شاملSMES، مجله مهندسی برق دانشگاه تبریز، جلد 47، شماره 2، 1396.
[5]      Y. J. Liu, L. Tang, S. Tong, C. P. Chen and D. J. Li, Reinforcement learning design-based adaptive tracking control with less learning parameters for nonlinear discrete-time MIMO systems, IEEE Transactions on Neural Networks and Learning Systems, vol. 26, no.1, pp. 165-176, 2015.
[6]      H. B.Ammar, E., Eaton, J. M., Luna and P.Ruvolo, Autonomous cross-domain knowledge transfer in lifelong policy gradient reinforcement learning, International Joint Conference on Artificial Intelligence, pp. 3345-3351, 2015.
[7]      A. Fachantidis, I. Partalas, G. Tsoumakas and I. Vlahavas, Transferring task models in reinforcement learning agents, Neurocomputing, vol. 107, pp. 23-32, 2013.
[8]      M.Ghavamzadeh S.Mannor, J.Pineau and A. Tamar, Bayesian reinforcement learning: A survey, Foundations and Trends® in Machine Learning, vol. 8, no. 5-6, pp. 359-483, 2015.
[9]      A. Gupta, C. Devin, Y. Liu, P. Abbeel and S. Levine, Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning, arXiv preprint arXiv: 1703.02949, 2017.
[10]      O. Mohammed, G.Bailly and D.Pellier, Acquiring Human-Robot Interaction skills with Transfer Learning Techniques, Proceedings of the Companion on Human-Robot Interaction, pp. 359-360, 2017.
[11]      F. L.da Silva and A. H. R. Costa, Accelerating Multiagent Reinforcement Learning through Transfer Learning, Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pp. 5034-5035, 2017.
[12]      M. N. Ahmadabadi and M. Asadpour, Expertness based cooperative Q-learning, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), vol. 32, no. 1, pp. 66-76, 2002.
[13]      K. Ito, A. Gofuku, Y. Imoto, and M. Takeshita, A study of reinforcement learning with knowledge sharing for distributed autonomous system,Proceedings of the International Symposium on Computational Intelligence in Robotics and Automation, pp. 16-20, 2003.
[14]      R. A. Bianchi, L. A. Celiberto, P. E. Santos, J. P.Matsuura and R. L. deMantaras, Transferring knowledge as heuristics in reinforcement learning: A case-based approach, Artificial Intelligence, vol. 226, pp.102-121, 2015.
[15]      Y. Hou, Y. S.Ong, L. Feng and J. M. Zurada, An Evolutionary Transfer Reinforcement Learning Framework for Multi-Agent System, IEEE Transactions on Evolutionary Computation, vol. 21, no. 4, pp. 601-615, 2017.
[16]      Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever and P. Abbeel, RL2: Fast Reinforcement Learning via Slow Reinforcement Learning. arXiv preprint arXiv:1611.02779, 2016.
[17]      P. Tommasino, D. Caligiore, M. Mirolli and G. Baldassarre, A Reinforcement Learning Architecture that Transfers Knowledge between Skills when Solving Multiple Tasks, IEEETransactions on Cognitive and Developmental Systems, 2016.
[18]      G. F. Wang, Z. Fang, P. Li and B. Li, Transferring knowledge from human-demonstration trajectories to reinforcement learning, Transactions of the Institute of Measurement and Control, vol. 40, no.1, pp. 94-101, 2018.
[19]      R. Glatt, F. L. da Silva and A. H. R. Costa, Towards Knowledge Transfer in Deep Reinforcement Learning, 5th Brazilian Conference on Intelligent Systems, pp. 91-96, 2016.
[20]      L. Zhou, P. Yang, C. Chen, Y. Gao, Multiagent reinforcement learning with sparse interactions by negotiation and knowledge transfer, IEEE transactions on cybernetics, vol. 47, no. 5, pp. 1238-1250, 2017.
[21]      T. Takano, H. Takase, H. Kawanaka and S. Tsuruoka, Preferential exploration method of transfer learning for reinforcement learning in same transition model, 6th International Conference on Soft Computing and Intelligent Systems and 13th International Symposium on Advanced Intelligent Systems, pp. 2099-2103, 2012.
[22]      G. Konidaris, I. Scheidwasser and A. Barto, Transfer in reinforcement learning via shared features, Journal of Machine Learning Research, pp. 1331-1371, 2012.
[23]      B. Banerjee and P. Stone, General Game Learning Using Knowledge Transfer. IJCAI, pp. 672-677. 2007.
[24]      E. Ferrante, A. Lazaric, and M. Restelli, Transfer of task representation in reinforcement learning using policy-based proto-value functions, Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems, Vol. 3, pp. 1329-1332, 2008.
[25]      A. Lazaric, Knowledge transfer in reinforcement learning, PhD thesis, Politecnico di Milano, 2008.
[26]      سیده ملیحه اخلاقی هاشمی‌پور، انتقال دانش در مسائل یادگیری تقویتی با ویژگی‌های مشترک، پایان نامه کارشناسی ارشد، دانشکده برق و کامپیوتر، دانشگاه یزد، 1394.
 
[27]      R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, Cambridge, MIT Press, 1998.
[28]      G. Yen and T. Hickey, Reinforcement learning algorithms for robotic navigation in dynamic environment, ISI Transaction, vol. 43,no. 2, pp. 217-230, 2004.
[29]      A. Epshteyn and G. Dejong, Qualitative Reinforcement Learning, Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, 2006.
[30]      J. Asmuth, M. L. Littman and R. Zinkov, Potential-based shaping in model based reinforcement learning, Proceedings of the 23rd AAAI conference on Artificial intelligence, pp. 604-609, 2008.
[31]      H. Van Hasselt, A. Guez and D. Silver, Deep Reinforcement Learning with Double Q-Learning, Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pp. 1-7, 2016.
  [32]      Q. Wang, L. Ruan and L. Si, Adaptive Knowledge Transfer for Multiple Instance Learning in Image Classification, Proceedings of the 28rd AAAI Conference on Artificial Intelligence, pp. 1334-1340, 2014.
 
   [33]      حسین مرادی فراهانی، جواد عسگری، طراحی کنترل­کننده عصبی- فازی نوع-2، مجله مهندسی برق دانشگاه تبریز، جلد 43، شماره 1، 1392.
[34]      A. Belaout, F. Krim, A. Mellit, B. Talbi and A. Arabi, Multiclass adaptive neuro-fuzzy classifier and feature selection techniques for photovoltaic array fault detection and classification, Renewable Energy, vol. 127, pp. 548-558, 2018.
[35]      A. Z. Kamil, S. Rustamov, M. A. Clements and E. Mustafayev, Adaptive Neuro-Fuzzy Inference System for Classification of Texts, Recent Developments and the New Direction in Soft-Computing Foundations and Applications, pp. 63-70, 2018.
[36]      S. V. R. Termeh, A. Kornejady, H. R. Pourghasemi and S. Keesstra, Flood susceptibility mapping using novel ensembles of adaptive neuro fuzzy inference system and metaheuristic algorithms, Science of the Total Environment, vol. 615, pp. 438-451, 2018.
[37]      D. Wang, T. He, Z. Li, L. Cao, N. Dey, A. S. Ashour, ... and F. Shi, Image feature-based affective retrieval employing improved parameter and structure identification of adaptive neuro-fuzzy inference system, Neural Computing and Applications, vol. 29, no. 4, pp. 1087-1102, 2018.

فایل ها

هم رسانی

ارجاع به این مقاله

آمار