FB15K: Understanding the Dataset

This investigation started while I was writing my master’s thesis.

Papers about Knowledge Graph Embedding and Knowledge Graph Completion repeatedly mentioned FB15K and FB15K-237. They explained models, loss functions, and evaluation scores in detail, but often said little about the dataset itself.

Before comparing model scores, I wanted to see what researchers were actually using:

• What kind of knowledge does the dataset contain?
• Why are there two versions?
• What does a good score on the benchmark mean?
• How close is it to a knowledge graph used in practice?

This article looks at the dataset itself. In the next article, I load FB15K-237 into Neo4j and explore its structure with Cypher.

From Freebase to FB15K

FB15K was extracted from Freebase, a large collaborative knowledge base originally developed by Metaweb and later operated by Google.

Freebase represented people, places, organizations, films, books, music, and many other subjects as entities connected by relationships.

A fact can be expressed as a triple:

Entity -- Relation --> Entity

For example:

Person -- profession --> Politician

Freebase is no longer available as an online service, but Google still provides its final data dumps and Freebase-to-Wikidata mappings for research and historical use.

• Freebase data dumps

What is FB15K?

FB15K is a subset of Freebase created for research on multi-relational data and knowledge graph completion. It was introduced with the TransE model in 2013.

The name is straightforward: FB comes from Freebase, and 15K indicates that the dataset contains about 15,000 entities.

Item	Count
Entities	14,951
Relations	1,345
Triples	592,213

The dataset is much smaller than the original Freebase knowledge base. That makes it practical for repeated machine learning experiments and comparisons between models.

Why was FB15K-237 created?

FB15K became a popular benchmark, but researchers later identified an important weakness.

Some relations in the training data were inverse or near-equivalent forms of relations found in the test data.

For example:

Person -- born_in --> City

and:

City -- people_born_here --> Person

When one direction is already present in the training data, predicting the reverse direction can become too easy. A model may receive a high score by learning a simple inverse pattern rather than learning more general graph structure.

FB15K-237 was created to reduce this form of test leakage by removing many inverse and equivalent relations. Its construction also removed validation and test triples whose entity pairs were already directly connected in the training data.

Item	Count
Entities	14,541
Relations	237
Training triples	272,115
Validation triples	17,535
Test triples	20,466
Total triples	310,116

These are the counts reported for the official FB15K-237 release. If a local count differs, check whether the files were modified or whether the import skipped or duplicated rows.

Japanese note:
FB15K-237 は、単にリレーション数を減らした版ではありません。訓練データからテストデータを容易に推測できる逆関係や同等関係を減らし、評価時の情報漏洩を抑えるために作られました。

What do the files contain?

A knowledge-base record in FB15K-237 looks like this:

/m/0grwj    /people/person/profession    /m/05sxg2

The three tab-separated values are:

subject MID    relation    object MID

The first and third values are Freebase MIDs, or Machine IDs. The middle value is the relationship between them.

This maps naturally to a graph:

(Entity)-[Relation]->(Entity)

However, the main triple files do not contain readable names or descriptions for the entities. Looking only at the record above, we cannot easily tell what /m/0grwj or /m/05sxg2 represents.

That is not an import error or an incomplete download. It reflects the purpose of the benchmark.

A machine learning model can treat each MID as an identifier and learn from the surrounding graph structure. It does not need to know whether the entity is Tokyo, a film, or a person.

The benchmark has three splits

The knowledge-base triples are divided into three files:

train.txt
valid.txt
test.txt

The training set is used to learn the model. The validation set is used to tune settings and compare model variants. The test set is used for the final evaluation.

This separation is one reason the dataset is convenient for research: different models can be evaluated under a common setup.

Text data is included too

The Microsoft FB15K-237 package also includes textual mention files derived from ClueWeb12:

text_cvsc.txt
text_emnlp.txt

These records contain two MIDs, a lexicalized dependency path between them, and an occurrence count. They were used in research combining text with knowledge-base representations.

They are different from the knowledge-base triples in train.txt, valid.txt, and test.txt. When people refer to FB15K-237 for link prediction, they often mean these three triple files.

What is Knowledge Graph Completion?

A real knowledge graph is rarely complete. Relationships may be missing because source data is incomplete, outdated, or collected from different systems.

Knowledge Graph Completion, often shortened to KGC, predicts these missing facts.

A model may receive a partial triple such as:

Person A -- profession --> ?

It then ranks possible entities for the missing object.

The same task can be performed with a missing subject:

? -- profession --> Actor

This is also commonly called link prediction.

FB15K and FB15K-237 are benchmark datasets for evaluating this prediction task. They are not complete knowledge management systems.

How are models evaluated?

Knowledge graph completion models rank candidate entities for the missing part of a triple.

Common metrics include:

Metric	Meaning
MRR	Mean Reciprocal Rank
Hits@1	Correct entity ranked first
Hits@3	Correct entity ranked in the top 3
Hits@10	Correct entity ranked in the top 10

If the correct answer is ranked first, its reciprocal rank is 1. If it is ranked second, the reciprocal rank is 0.5. MRR is the average reciprocal rank across the test queries.

Many papers also use filtered evaluation. Other known correct triples are removed from the candidate errors, so a model is not penalized for ranking another valid answer highly.

These metrics are useful, but they measure a specific task: how well a model ranks missing entities in this benchmark.

Why is FB15K-237 used so often?

FB15K-237 remains useful for several practical reasons:

• Many papers already report results on it, making comparisons easier.
• It is small enough to run without very large infrastructure.
• Training, validation, and test data are already separated.
• The research community is familiar with its format and metrics.

A benchmark becomes more useful when many people understand it and publish comparable results.

A benchmark is not a production knowledge graph

This was the most important distinction for me.

FB15K-237 provides	A production knowledge graph may also need
Entity identifiers	Readable names and descriptions
Relation names	Business definitions and ontology
Graph topology	Source and provenance
Fixed benchmark splits	Update and deletion rules
Link-prediction targets	Valid time and change history
Comparable evaluation data	Confidence, uncertainty, and access control

FB15K-237 deliberately isolates a narrower problem: predicting missing links.

That does not make it a poor dataset. It means that its scores must be interpreted within its intended purpose.

A good MRR or Hits@10 score does not automatically show that a system can manage enterprise knowledge, explain where facts came from, resolve real-world identities, or handle knowledge that changes over time.

Japanese note:
FB15K-237 は、実務の知識グラフを小さくしたものではなく、リンク予測モデルを比較するための研究用ベンチマークです。目的が異なるため、名称、出典、時間、信頼度など、実務で重要な情報の多くは対象外です。

What I learned

Before opening the files, I imagined a small but rich knowledge graph.

The actual benchmark was simpler:

• entity IDs
• relation names
• triples divided into training, validation, and test sets

That simplicity is intentional. It allows researchers to focus on graph structure and link prediction.

Looking at the data also changed how I read research papers. MRR and Hits@10 are no longer just numbers in a table; they describe performance on a particular dataset with specific assumptions and limitations.

The name FB15K appears in many papers. Understanding the files behind that name makes those papers much easier to evaluate.

Next: loading FB15K-237 into Neo4j

In the next article, I:

• download the Microsoft package
• inspect the triple files
• load the data into Neo4j
• preserve the training, validation, and test splits
• explore the graph with Cypher
• examine what the MID-only graph reveals

Sometimes the fastest way to understand a benchmark is to load it and look at what is—and is not—there.

References

• Antoine Bordes et al., Translating Embeddings for Modeling Multi-relational Data, 2013
• Kristina Toutanova and Danqi Chen, Observed versus Latent Features for Knowledge Base and Text Inference, 2015
• Kristina Toutanova et al., Representing Text for Joint Embedding of Text and Knowledge Bases, 2015
• FB15K-237 Knowledge Base Completion Dataset — Microsoft
• Freebase data dumps — Google