ScholarWorks@성균관대학교

초록

Objective This study aims to develop and validate of a span-based annotation framework for clinical named entity recognition (NER) using large language models (LLMs) based on Korean emergency department clinical notes.Materials and Methods Two datasets with the same entity types but different annotation spans (word- vs phrase-level) were constructed, with the phrase-level dataset further was expanded into a doubled version. A Korean language-specific LLM was fine-tuned on each dataset, producing three variants that were compared with two baseline models, few-shot LLM and fine-tuned small language model (SLM). The final variant fine-tuned on the doubled phrase-level dataset was further evaluated against a human annotator.Results In all experimental settings, three variants outperformed the baselines by achieving the highest F1 scores across all metrics. The final variant achieved F1 scores exceeding 0.80 across all averaging strategies and evaluation metrics, including token-based, span-based exact, and span-based partial evaluations demonstrating its robustness applicable in a practical setting.Discussion While prompt engineering with few-shot is widely adopted for LLM-based clinical NER, our results proved that supervised fine-tuning (SFT) is consistently superior. The final variant outperformed the human annotator, emphasizing its potential as an automatic labeling tool.Conclusion This study introduced a novel span-based annotation framework for LLM-based clinical NER verified by three independent experiments. In multilingual and real-world clinical settings, LLMs have proven in handling complex entity spans that include word-level and phrase-level annotations, particularly for long and attribute-rich entities. Identifying medical terms in clinical notes is crucial for patient care and research, but computer systems often have difficulty recognizing these terms when they appear as longer phrases. This study developed a new method to improve how computers identify medical terms in Korean emergency department notes.We created datasets using two approaches: one with short, word-level labels and another with longer, phrase-level labels that include more context. A Korean large language model was trained on these datasets and compared against baseline methods. The model trained on phrase-level data achieved the highest performance, with accuracy exceeding 80% and even surpassing a human annotator in some evaluations.Our results demonstrate that training language models with carefully designed, span-based annotations are more effective than simpler prompting approaches. This method proved robust across multiple experiments and could serve as an automated tool for labeling medical records. Such tools may help researchers analyze health data more efficiently and support clinicians by making patient information more searchable and accessible, ultimately contributing to care delivery.

키워드