Grid Search _ 300M pure data sets

Results of [1B, 3B, 7B] models fully fine-tuned on 300M tokens of [elsevier, wikipedia and pubmed abstracts]

— = Random performance

Possibly STEM/college chem results are not meaningful, but Lambada is interesting

Screenshot 2023-05-24 at 12.01.47.png

Screenshot 2023-05-24 at 12.01.59.png

Screenshot 2023-05-24 at 12.02.26.png

Screenshot 2023-05-24 at 12.02.13.png

Screenshot 2023-05-24 at 12.02.39.png

Screenshot 2023-05-24 at 12.02.53.png

Screenshot 2023-05-24 at 12.03.08.png

Screenshot 2023-05-24 at 12.03.25.png

Screenshot 2023-05-24 at 12.03.43.png