COFAR

The COFAR (COmmonsense and FActual Reasoning) dataset is a collection of images and text queries specifically designed to challenge and evaluate image search models that aim to go beyond simple visual matching. It focuses on the ability of these models to perform commonsense and factual reasoning, a capability currently lacking in most existing image search technology.

Key Features of COFAR:

• Named Visual Entities as a Gateway to Knowledge: The dataset consists of images containing prominent named visual entities, such as:
  • Business Brands: Examples include "Rolex", "Starbucks", and "KFC".
  • Celebrities: Examples include "Lionel Messi" and "J.K. Rowling".
  • Landmarks: Examples include "Eiffel Tower", "Taj Mahal", and "Parthenon".
• Complex Queries Requiring Reasoning: Instead of simple object-based queries, COFAR uses carefully crafted textual queries that require a deeper understanding of the image content. These queries incorporate two main elements:
  • Factual Knowledge: This refers to specific facts about the named entities present in the image, often derived from external knowledge sources like Wikipedia. For instance, a query might state that "Lionel Messi is the captain of the Argentina national football team".
  • Commonsense Reasoning: Queries also involve commonsense assumptions about the scene, activities, and relationships depicted in the image. An example is "a queue of customers patiently waiting to buy ice cream", where the model needs to infer that people standing in line are likely "customers" who are "waiting".
• Structured for Evaluation: COFAR is organised into distinct sets to facilitate model training and evaluation:
  • Training Set: This subset contains images and corresponding queries used to train image search models, allowing them to learn the relationships between visual content, textual descriptions, and external knowledge.
  • Gallery Sets: These sets contain images with entities unseen during training. Models are evaluated on their ability to retrieve relevant images from these galleries given a query, testing their generalisation to new, unseen entities.

What Makes COFAR Unique?

• Focus on Reasoning over Visual Recognition: COFAR shifts the emphasis from purely visual aspects of an image to the reasoning required to understand the query. This means that models cannot rely solely on detecting objects or scenes; they need to incorporate external knowledge and make commonsense inferences.
• Diversity and Real-World Applicability: The dataset covers a diverse range of named entities and scenarios, making it more representative of real-world image search tasks. The inclusion of business brands, celebrities, and landmarks makes COFAR relevant to various practical applications, like e-commerce, news search, and tourism.
• Promoting Advanced Image Search Technology: COFAR aims to stimulate the development of more sophisticated image search engines capable of understanding user intent and context. By providing a challenging benchmark, COFAR encourages researchers to explore new techniques and architectures that go beyond the limitations of existing methods.

Limitations of COFAR:

The sources also acknowledge some limitations of the COFAR dataset:

• Single Entity Focus: Each image in COFAR contains only one prominent named visual entity. This might not accurately reflect real-world scenarios where multiple entities often co-exist within an image.
• Dataset Size: While COFAR is larger than some previous datasets in this area, it is still limited in size compared to datasets used for training large vision-language models. This could potentially limit the generalisability of models trained on COFAR.

Overall, COFAR is a valuable resource for researchers working on image search that involves complex reasoning. It presents a challenging benchmark and encourages the development of novel techniques that can leverage external knowledge and commonsense understanding to improve image search accuracy and relevance.