“A little learning is a dangerous thing”

– Alexander Pope

The Temptation

You have a large body of unstructured text, and a problem to solve. Perhaps you need to summarize the text or identify topics. Perhaps you need to extract the essential information from the text or find authoritative answers to questions.

The traditional approach would be to annotate data to train a model. If the task is straightforward, say finding all prescription information, defining an annotation scheme may be too; in fact, maybe there is already a labeled data set you can use.

But what if the task is more open-ended? You might need an expert even to identify what information is valuable to extract, or what should be included or omitted from a summary. Suppose your source corpus is heterogenous. For example, suppose you need to extract information from all clinical notes from multiple hospitals. Then even an expert might not anticipate all the use cases, and you may need to expand and refine your schema continually.

This can be a daunting task. Is there a shortcut? Maybe a large language model would help? After all, the large pre-trained models have read more natural language than any human expert, and their ability to recall information in many fields is impressive. Why not leverage that pre-training to reduce the amount of annotated data you need?

Before taking that leap of faith, consider this cautionary tale.

Parsing Clinical Text – A Cautionary Tale

The problem matched the example I described above – extracting clinically significant information from medical notes.

The architecture was a context-free grammar (CFG), a rule-based system which can parse text into a tree structure. A handful of classifiers trained from labeled data for disambiguation and topic classification supplemented the grammar. The system was designed and originally built by a single researcher, Jeffrey Hill¹, starting in 2010, working at a small company called Humedica. The second member of the team, Sam Pilato, added more semantic structure and a sophisticated analysis of prescription discussions which yielded information available nowhere else.

The architecture may seem outdated today, but it made sense at the time. This was long before instruction-trained LLMs like ChatGPT (2022)², long before contextual embeddings like ELMO (2018)³, and BERT (2018)⁴, or even fixed embeddings like Word2vec (2013)^5,6. Instead of using statistical knowledge of linguistics, it relied on human knowledge. And the system worked, both technically and commercially. Real world evidence (RWE) was the latest trend in the pharmaceutical industry, and top companies paid $1 M or more to license the results. The business was eager to extend the system to build on this success; I joined as the third member of the team in 2013, and within a year, the team had grown to 5.

Designing an annotation and evaluation system for such a broad system was a fundamentally difficult problem, and early attempts failed. However, without one, the team had to rely on manual review of differences in output to decide whether a proposed change was beneficial, and development slowed. Furthermore, our clients, who had previously relied on white papers, needed a published refereed paper to cite in their research, and I took on the task, knowing that the business depended on it. Building on an approach which Sam Pilato had described, I eventually succeeded, but not before our clients lost confidence in our results.

Why Care?

Why should you care about a context-free grammar system, in the era of generative large language models?

1. Because our ability to build large natural language understanding systems is even greater today, but still outpaces our ability to evaluate them reliably.
2. Because evaluation is a function of the task and its results, not the methodology, so the solutions to past evaluation challenges are just as relevant today.
3. And most importantly, because large language models and few-shot learning offer the same temptation: get a solution working first, and then worry about how to validate it. If you spend too much time on the solution before tackling evaluation, you may fall into the same trap.

The Nature of the Trap

To be clear, neither the challenging task nor the technology killed the Humedica system. The technology was a sound choice at the time, and results proved that a context-free grammar could extract a wide variety of valuable information from clinical notes. Even the decision to rely on human linguistic knowledge rather than annotated training sets was justified at first. In fact, without that decision, the system might never have been built at all. What doomed the system was the lack of an annotation system for evaluation. Without one, there was no way to validate changes, or even to tell where improvements were needed, and keeping up with the demand for more data became impossible.

Generative large language models pose similar risks today. In place of human knowledge of language, we have billions of parameters worth of pre-training, allowing the models to answer questions with just a few examples⁷ or even none at all. In medical NLP, the debate about whether specialized models trained on clinical notes can outperform their larger general-purpose cousins is ongoing^8,9. Natural language output further compounds the difficulty of evaluation. Many teams adopt the LLM-as-a-judge¹⁰ approach, relying on one LLM to validate another, and this choice may be based on the lack of practical alternatives, rather than a careful study of its effectiveness.

Avoiding the Trap

Can we avoid this trap, without giving up on generative language models? The key is to think about evaluation at the start of a project, and to keep working on it as the project develops. You don’t need to defer other work until you solve the evaluation problem, but the problem won’t solve itself. The longer you wait, the less objective information you have to guide you.

In a future post, I’ll talk more about the challenges of evaluating LLM output and some proven approaches.

Sources

1. https://www.linkedin.com/in/jefhil/ ↩︎
2. Ouyang, Long, Jeff Wu, Xu Jiang, et al. 2022. “Training Language Models to Follow Instructions with Human Feedback.” arXiv.Org, March 4. https://arxiv.org/abs/2203.02155v1.
   ↩︎
3. Peters, Matthew E., Mark Neumann, Mohit Iyyer, et al. 2018. “Deep Contextualized Word Representations.” arXiv:1802.05365. Version 1. Preprint, arXiv, February 15. https://doi.org/10.48550/arXiv.1802.05365.
   ↩︎
4. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding.” arXiv:1810.04805 [Cs], October 10. http://arxiv.org/abs/1810.04805.
   ↩︎
5. Logeswaran, Lajanugen, and Honglak Lee. 2018. “An Efficient Framework for Learning Sentence Representations.” arXiv:1803.02893. Preprint, arXiv, March 7. https://doi.org/10.48550/arXiv.1803.02893.
   ↩︎
6. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. “Distributed representations of words and phrases and their compositionality.” In Advances in Neural Information Processing Systems 26: 3111–3119. ↩︎
7. Brown, Tom, Benjamin Mann, Nick Ryder, et al. 2020. “Language Models Are Few-Shot Learners.” Advances in Neural Information Processing Systems 33: 1877–901.
   ↩︎
8. Liu, Fenglin, Zheng Li, Hongjian Zhou, Qingyu Yin, Jingfeng Yang, Xianfeng Tang, Chen Luo et al. “Large language models in the clinic: a comprehensive benchmark.” arXiv preprint arXiv:2405.00716 (2024). ↩︎
9. Vishwanath, Krithik, Anton Alyakin, Mrigayu Ghosh, et al. 2026. “General-Purpose Large Language Models Outperform Specialized Clinical AI Tools on Medical Benchmarks.” Nature Medicine 32 (7): 2405–9. https://doi.org/10.1038/s41591-026-04431-5.
   ↩︎
10. Zheng, Lianmin, Wei-Lin Chiang, Ying Sheng, et al. 2023. “Judging Llm-as-a-Judge with Mt-Bench and Chatbot Arena.” Advances in Neural Information Processing Systems 36: 46595–46623.
    ↩︎