TY - CHAP
T1 - Exploring Transformers as Compact, Data-Efficient Language Models
AU - Fields, Clayton
AU - Kennington, Casey
N1 - Fields, Clayton and Kennington, Casey. (2023). "Exploring Transformers as Compact, Data-Efficient Language Models". In J. Jiang, D. Reitter, and S. Deng (Eds.), Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL) (pp. 521-531). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.conll-1.35
PY - 2023/1/1
Y1 - 2023/1/1
N2 - Large scale transformer models, trained with massive datasets have become the standard in natural language processing. The huge size of most transformers make research with these models impossible for those with limited computational resources. Additionally, the enormous pretraining data requirements of transformers exclude pretraining them with many smaller datasets that might provide enlightening results. In this study, we show that transformers can be significantly reduced in size, with as few as 5.7 million parameters, and still retain most of their downstream capability. Further we show that transformer models can retain comparable results when trained on human-scale datasets, as few as 5 million words of pretraining data. Overall, the results of our study suggest transformers function well as compact, data efficient language models and that complex model compression methods, such as model distillation are not necessarily superior to pretraining reduced size transformer models from scratch.
AB - Large scale transformer models, trained with massive datasets have become the standard in natural language processing. The huge size of most transformers make research with these models impossible for those with limited computational resources. Additionally, the enormous pretraining data requirements of transformers exclude pretraining them with many smaller datasets that might provide enlightening results. In this study, we show that transformers can be significantly reduced in size, with as few as 5.7 million parameters, and still retain most of their downstream capability. Further we show that transformer models can retain comparable results when trained on human-scale datasets, as few as 5 million words of pretraining data. Overall, the results of our study suggest transformers function well as compact, data efficient language models and that complex model compression methods, such as model distillation are not necessarily superior to pretraining reduced size transformer models from scratch.
UR - https://scholarworks.boisestate.edu/cs_facpubs/392
M3 - Chapter
BT - Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL)
ER -