Analysis of Conventional Dropout and its Application to Group Dropout

• Item type: Trans(1)
• 公開日: 2017-07-19
• タイトル: Analysis of Conventional Dropout and its Application to Group Dropout
• タイトル / 言語: en
• タイトル: Analysis of Conventional Dropout and its Application to Group Dropout
• 言語: eng
• キーワード / 主題Scheme: Other
• キーワード / 主題: [オリジナル論文] dropout, over-fitting, ensemble learning, online learning, soft-committee machine
• 資源タイプ / 資源タイプ識別子: http://purl.org/coar/resource_type/c_6501
• 資源タイプ: journal article
• 著者所属: College of Industrial Technology, Nihon University
• 著者所属: Graduate School of Industrial Technology, Nihon University
• 著者所属: Graduate School of Informatics and Engineering, The University of Electro-Communications
• 著者所属: Graduate School of Industrial Technology, Nihon University
• 著者所属: Graduate School of Informatics and Engineering, The University of Electro-Communications
• 著者所属(英): en
• 著者所属(英): College of Industrial Technology, Nihon University
• 著者所属(英): en
• 著者所属(英): Graduate School of Industrial Technology, Nihon University
• 著者所属(英): en
• 著者所属(英): Graduate School of Informatics and Engineering, The University of Electro-Communications
• 著者所属(英): en
• 著者所属(英): Graduate School of Industrial Technology, Nihon University
• 著者所属(英): en
• 著者所属(英): Graduate School of Informatics and Engineering, The University of Electro-Communications
• 著者名: Kazuyuki, Hara; Daisuke, Saitoh; Satoshi, Suzuki; Takumi, Kondou; Hayaru, Shouno
• 著者名(英): Kazuyuki, Hara; Daisuke, Saitoh; Satoshi, Suzuki; Takumi, Kondou; Hayaru, Shouno
• 論文抄録 / 内容記述タイプ: Other
• 論文抄録 / 内容記述: Deep learning is a state-of-the-art learning method that is used in fields such as visual object recognition and speech recognition. It uses very deep layers and a huge number of units and connections, so overfitting is a serious problem. The dropout method is used to address this problem. Dropout is a regularizer that neglects randomly selected inputs and hidden units during the learning process with probability q; after learning, the neglected inputs and hidden units are combined with the learned network to express the final output. Wager et al. pointed out that conventional dropout is an adaptive L2 regularizer, so we compared the learning behavior of conventional dropout with that of stochastic gradient descent with the L2 regularizer. We found that combining the neglected hidden units with the learned network can be regarded as ensemble learning, so we analyzed, on the basis of on-line learning, conventional dropout learning from the viewpoint of ensemble learning. Next we compared conventional dropout and ensemble learning from two additional viewpoints and confirmed that conventional dropout can be regarded as ensemble learning that divides a student network into two sub-networks. On the basis of this finding, we developed a novel dropout method that divides the network into more than two sub-networks. Computer simulation demonstrated that this method enhances the benefit of ensemble learning.
• 論文抄録(英) / 内容記述タイプ: Other
• 論文抄録(英) / 内容記述: Deep learning is a state-of-the-art learning method that is used in fields such as visual object recognition and speech recognition. It uses very deep layers and a huge number of units and connections, so overfitting is a serious problem. The dropout method is used to address this problem. Dropout is a regularizer that neglects randomly selected inputs and hidden units during the learning process with probability q; after learning, the neglected inputs and hidden units are combined with the learned network to express the final output. Wager et al. pointed out that conventional dropout is an adaptive L2 regularizer, so we compared the learning behavior of conventional dropout with that of stochastic gradient descent with the L2 regularizer. We found that combining the neglected hidden units with the learned network can be regarded as ensemble learning, so we analyzed, on the basis of on-line learning, conventional dropout learning from the viewpoint of ensemble learning. Next we compared conventional dropout and ensemble learning from two additional viewpoints and confirmed that conventional dropout can be regarded as ensemble learning that divides a student network into two sub-networks. On the basis of this finding, we developed a novel dropout method that divides the network into more than two sub-networks. Computer simulation demonstrated that this method enhances the benefit of ensemble learning.
• 書誌レコードID / 収録物識別子タイプ: NCID
• 書誌レコードID / 収録物識別子: AA11464803
• 書誌情報: 情報処理学会論文誌数理モデル化と応用（TOM）; 巻 10, 号 2, p. 25-32, 発行日 2017-07-19
• ISSN / 収録物識別子タイプ: ISSN
• ISSN / 収録物識別子: 1882-7780
• 出版者 / 言語: ja
• 出版者: 情報処理学会