Overview

<aside> 📊 Highlights

• Strong performance on Hendryck’s college biology 📈
• Mild degradation of Lambada, especially close to loss spike regions 📉
• Nothing significant of note on other benchmark tasks 🫳

All evaluation below is zero-shot.

</aside>

Context

This model is currently being finetuned from a pretrained Pythia-1b checkpoint on 250B tokens ⇒ 70k optimiser steps of EuroPMC biomedical / scientific text. The model suffered from loss spikes at ~ steps 10500, 14000, 20000 and 21000. The loss curve can be seen below with each bar in the subsequent evaluation charts being a saved checkpoint at 5k intervals. See more details here.

Results

A float in the legend denotes a checkpoint which has been finetuned for that many optimiser steps.

Good

• We can see promising results on the hendryck’s college biology benchmark where we outperform a random baseline and Pythia pretrained model significantly (relatively equivalent).
• We also see the performance improve as we train for more steps.
• Finally, this benchmark is relevant to our training corpus and suggests we are potentially capable of re-using our EuroPMC knowledge in this new context.

Bad

We did drop in Lambada performance throughout training and the model seems to almost get 0% during the period of a loss spike (red bar at 20k optimiser steps).

Ugly Indifferent

There were slight drops in performance or maintaining equivalent performance across most other benchmarks.