TitleKNAS: Green Neural Architecture Search
AuthorsXu, Jingjing
Zhao, Liang
Lin, Junyang
Gao, Rundong
Sun, Xu
Yang, Hongxia
AffiliationPeking Univ, Sch EECS, MOE Key Lab Computat Linguist, Beijing, Peoples R China
Peking Univ, Ctr Data Sci, Beijing, Peoples R China
Alibaba Grp, Hangzhou, Peoples R China
Issue Date2021
PublisherINTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 139
AbstractMany existing neural architecture search (NAS) solutions rely on downstream training for architecture evaluation, which takes enormous computations. Considering that these computations bring a large carbon footprint, this paper aims to explore a green (namely environmental-friendly) NAS solution that evaluates architectures without training. Intuitively, gradients, induced by the architecture itself, directly decide the convergence and generalization results. It motivates us to propose the gradient kernel hypothesis: Gradients can be used as a coarse-grained proxy of downstream training to evaluate random-initialized networks. To support the hypothesis, we conduct a theoretical analysis and find a practical gradient kernel that has good correlations with training loss and validation performance. According to this hypothesis, we propose a new kernel based architecture search approach KNAS. Experiments show that KNAS achieves competitive results with orders of magnitude faster than "train-then-test" paradigms on image classification tasks. Furthermore, the extremely low search cost enables its wide applications. The searched network also outperforms strong baseline RoBERTA-large on two text classification tasks. Codes are available at https://github.com/Jingjing -NLP/KNAS.
URIhttp://hdl.handle.net/20.500.11897/641458
ISSN2640-3498
IndexedCPCI-S(ISTP)
Appears in Collections:信息科学技术学院
计算语言学教育部重点实验室
其他研究院

Files in This Work
There are no files associated with this item.

Web of Science®



Checked on Last Week

百度学术™



Checked on Current Time




License: See PKU IR operational policies.