Do Language Models Know How Heavy an Elephant Is?
Humans have a pretty good sense of scale, or reasonable ranges of these numeric attributes, of different objects, but do pre-trained language representations? Although pre-trained Language Models (LMs) like BERT have shown a remarkable ability to learn all kinds of knowledge, including factual knowledge, it remains unclear whether their representations can capture these types of numeric attributes from text alone without explicit training data. In our recent paper, we measure the amount of scale information that is captured in several kinds of pre-trained text representations and show that...
Source: stanford.edu