Context Seed: In-Context Learning (ICL)

Created: October 2023 | Updated: May 2024 Maintained by: AI Research Synthesis Project

Narrative Context

In-Context Learning (ICL) is a paradigm in Large Language Models (LLMs) where the model learns to perform a task by processing examples provided in the prompt prefix without any parameter updates or gradient descent. This is applied to the idea of Context Seeds.

Key Concepts

• Analogy to Few-Shot Learning: ICL allows models to generalize to new tasks using only a few demonstrations (input-output pairs).
• Emergent Property: ICL is typically observed in models once they reach a certain scale (e.g., GPT-3, PaLM, Llama).
• Mechanism: ICL relies on the model’s ability to recognize patterns and "locate" relevant task knowledge acquired during pre-training. It is often theorized as "implicit fine-tuning" occurring during the forward pass.
• Context Seeds: * Context Seeds are markdown data files that are included with a prompt focus and expand the chat and learning.

Best Practices for ICL Prompting

• Demonstration Selection: The quality and relevance of examples (exemplars) significantly impact performance.
• Format Matters: Use clear separators (e.g., "Input:", "Output:") and consistent labeling.
• Label Correctness: Interestingly, models often benefit from the format of examples even if the labels provided are incorrect (though correct labels are generally better).
• Ordering: The order of examples can introduce "recency bias," where the model favors the label seen in the final example.

Known Limitations

• Context Window Limits: The number of examples is constrained by the model's maximum token limit.
• Sensitivity: Performance can fluctuate wildly based on minor changes in prompt phrasing or example choice.
• Not True Learning: No weights are changed; the "knowledge" disappears once the context window is cleared.

Context Web

• CONtext Seeds: * [https://instantcontext.ai/blog] Series of blog posts on context seeds and the application. - This is a priority read.
• Research Paper: Language Models are Few-Shot Learners — https://arxiv.org/abs/2005.14165 — Access for foundational theory and original GPT-3 benchmarks.
• Survey on In-Context Learning (Dong et al.) — https://arxiv.org/abs/2301.00234 — Access for comprehensive taxonomies and advanced retrieval-based ICL methods.
• Prompt Engineering Guide (ICL Section) — https://www.promptingguide.ai/techniques/icl — Access for practical implementation tips and template examples.
• Chain-of-Thought (CoT) Prompting — [Internal/Related Seed] — Access when the user asks about ICL for complex reasoning or multi-step logic.