K-SEED

We introduce K-SEED, a Korean adaptation of the SEED-Bench [1] designed for evaluating vision-language models. By translating the first 20 percent of the test subset of SEED-Bench into Korean, and carefully reviewing its naturalness through human inspection, we developed a novel robust evaluation benchmark specifically for Korean language. K-SEED consists of questions across 12 evaluation dimensions, such as scene understanding, instance identity, and instance attribute, allowing a thorough evaluation of model performance in Korean.

For more details, Please refer to the VARCO-VISION technical report.

• Technical Report: VARCO-VISION: Expanding Frontiers in Korean Vision-Language Models
• Blog(Korean): VARCO-VISION Technical Report Summary
• Huggingface Version Model: NCSOFT/VARCO-VISION-14B-HF

Image	SEED-Bench	K-SEED
	question: How many towels are in the image? choice_a: One choice_b: Two choice_c: Three choice_d: Four	question: 이미지에 수건이 몇 개 있나요? choice_a: 한 개 choice_b: 두 개 choice_c: 세 개 choice_d: 네 개

Inference Prompt

<image>
{question}
A. {choice_a}
B. {choice_b}
C. {choice_c}
D. {choice_d}

주어진 선택지 중 해당 옵션의 문자로 직접 답하세요.

Results

Below are the evaluation results of various vision-language models, including VARCO-VISION-14B on K-SEED.

References

[1] Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299–13308, 2024.

Citation

If you use K-SEED in your research, please cite the following:

@misc{ju2024varcovisionexpandingfrontierskorean,
      title={VARCO-VISION: Expanding Frontiers in Korean Vision-Language Models}, 
      author={Jeongho Ju and Daeyoung Kim and SunYoung Park and Youngjune Kim},
      year={2024},
      eprint={2411.19103},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2411.19103}, 
}