Vision Transformer-based Meta Loss Landscape Exploration with Actor-Critic Method

• Item type: SIG Technical Reports(1)
• 公開日: 2024-07-15
• タイトル: Vision Transformer-based Meta Loss Landscape Exploration with Actor-Critic Method
• タイトル / 言語: en
• タイトル: Vision Transformer-based Meta Loss Landscape Exploration with Actor-Critic Method
• 言語: eng
• 資源タイプ / 資源タイプ識別子: http://purl.org/coar/resource_type/c_18gh
• 資源タイプ: technical report
• 著者所属: Grad. School of Info. Science and Technology Hokkaido University
• 著者所属: Grad. School of Info. Science and Technology Hokkaido University
• 著者所属: Grad. School of Info. Science and Technology Hokkaido University
• 著者所属: Grad. School of Info. Science and Technology Hokkaido University
• 著者所属: RIKEN Center for Computational Science
• 著者所属: Information Initiative Center Hokkaido University
• 著者所属(英): en
• 著者所属(英): Grad. School of Info. Science and Technology Hokkaido University
• 著者所属(英): en
• 著者所属(英): Grad. School of Info. Science and Technology Hokkaido University
• 著者所属(英): en
• 著者所属(英): Grad. School of Info. Science and Technology Hokkaido University
• 著者所属(英): en
• 著者所属(英): Grad. School of Info. Science and Technology Hokkaido University
• 著者所属(英): en
• 著者所属(英): RIKEN Center for Computational Science
• 著者所属(英): en
• 著者所属(英): Information Initiative Center Hokkaido University
• 著者名: Enzhi, Zhang; Xuanyi, Wu; Rui, Zhong; Xingbang, Du; Mohamed, Wahib; Masaharu, Munetomo
• 著者名(英): Enzhi, Zhang; Xuanyi, Wu; Rui, Zhong; Xingbang, Du; Mohamed, Wahib; Masaharu, Munetomo
• 論文抄録 / 内容記述タイプ: Other
• 論文抄録 / 内容記述: Detecting and mitigating overfitting in deep neural networks remains a critical challenge in modern machine learning. This paper investigates innovative approaches to address these challenges, particularly focusing on vision transformer-based models. By leveraging meta-learning techniques and reinforcement learning frameworks, we introduce Transformer-based Loss Landscape Exploration (TLLE), which utilizes the validation loss landscape to guide gradient descent optimization. Unlike conventional methods, TLLE employs the Actor-Critic algorithm to learn the mapping from model weights to future values, facilitating efficient sample collection and precise value predictions. Experimental results demonstrate the superior performance of TLLE-enhanced transformer models in image classification and segmentation tasks, showcasing the efficacy of our approach in optimizing deep learning models for image analysis.
• 論文抄録(英) / 内容記述タイプ: Other
• 論文抄録(英) / 内容記述: Detecting and mitigating overfitting in deep neural networks remains a critical challenge in modern machine learning. This paper investigates innovative approaches to address these challenges, particularly focusing on vision transformer-based models. By leveraging meta-learning techniques and reinforcement learning frameworks, we introduce Transformer-based Loss Landscape Exploration (TLLE), which utilizes the validation loss landscape to guide gradient descent optimization. Unlike conventional methods, TLLE employs the Actor-Critic algorithm to learn the mapping from model weights to future values, facilitating efficient sample collection and precise value predictions. Experimental results demonstrate the superior performance of TLLE-enhanced transformer models in image classification and segmentation tasks, showcasing the efficacy of our approach in optimizing deep learning models for image analysis.
• 書誌レコードID / 収録物識別子タイプ: NCID
• 書誌レコードID / 収録物識別子: AN10505667
• 書誌情報: 研究報告数理モデル化と問題解決（MPS）; 巻 2024-MPS-149, 号 12, p. 1-5, 発行日 2024-07-15
• ISSN / 収録物識別子タイプ: ISSN
• ISSN / 収録物識別子: 2188-8833
• Notice: SIG Technical Reports are nonrefereed and hence may later appear in any journals, conferences, symposia, etc.
• 出版者 / 言語: ja
• 出版者: 情報処理学会