Questions to ask about building foundation models
- samrodriques
- Jul 23, 2023
- 1 min read
I often hear proposals to build new foundation models for biology. Here is the list of questions I ask. I rarely get past question 1.
What is the core task that the model performs/the core thing the model predicts? I.e., on a single iteration of the model, what is the input and what is the output?
For LLMs, the answer is "you give it N words and it provides a probability distribution over word N+1"
My experience is that 90% of proposals fail at this stage.
What is the loss metric on the core task?
For LLMs, the answer is cross-entropy or perplexity
What emergent behavior do you think the model will have?
For LLMs, the answer is "natural language processing." For other people, it might be "predicting enzyme functionality" or something.
What are the evals you will use to evaluate whether this emergent behavior is emerging?
For LLMs, e.g. Winogrande
Is there evidence that performance on the evals increases with scale? How expensive is it to get that evidence?
Questions about datasets.
Do they exist?
How many tokens?
Are they high quality? How does the person know if they are high quality?
More technical questions
Tokenization
What is the scaling method?
Architecture questions. Dense? Mixture of experts? Multiple read heads? etc.
The article on building foundation models in biology presents a compelling framework for evaluating the development of such models. The emphasis on defining the core task and understanding the model's input output dynamics resonates with the foundational steps in crafting a marketing assignment. When tasked to write a marketing assignment, it's crucial to first delineate the primary objective be it analyzing consumer behavior evaluating market strategies or assessing brand positioning. Just as the article underscores the importance of clear metrics and evaluation methods a marketing assignment benefits from well defined KPIs and analytical tools to measure effectiveness. This structured approach ensures that the assignment remains focused, coherent and impactful.
This is a razor-sharp framework for evaluating foundation model proposals in biology — or really in any domain where the hype can quickly outpace the fundamentals. Starting with the core task and measurable loss is essential, and it’s telling how often proposals can’t clearly define input-output behavior. The analogy to LLMs makes this framework especially accessible. I appreciate how this progression moves from defining a concrete task to anticipating emergent behaviors and setting up rigorous evals — something that's often overlooked in early-stage excitement. It reminds me of how even in fields like Corporate Finance Homework Help establishing the right questions and metrics at the outset leads to better outcomes down the line. A must-read checklist for anyone exploring biological…
These questions are a great framework for evaluating any proposal to build a foundation model, especially in a complex domain like biology. It's crucial to have a clear understanding of the core task the model will perform and how to measure its success from the outset. Without clear answers to these questions, most proposals can struggle to gain traction or even justify their development.
Just as in business strategy, especially when preparing for an MBA essay writing service it's important to define the goals, metrics, and methods for success right at the start. The clearer the problem and expected outcome, the more efficient the planning and execution. This can be compared to the importance of a clear mission and objectives…
Expert academic support for students studying legal concepts pertaining to contracts, agreements, and duties is offered by Contract Law Assignment Help. This service ensures clarity and accuracy in legal analysis by providing academically required, thoroughly researched, and plagiarism-free assignments. Expert legal professionals offer accurate and prompt assistance with case studies, contract drafting, breach of contract, legal remedies, and company law applications.
For many individuals, intimacy and sexuality go hand-in-hand. Emotional connection, trust, and vulnerability often enhance the sexual experience and create a deeper bond between partners. Intimate relationships are not solely about physical pleasure but also about emotional closeness and mutual respect. People in intimate relationships tend ft myers escort to develop a unique understanding of one another’s needs, desires, and boundaries. This connection is an essential aspect of human life, offering emotional fulfillment and promoting overall well-being, both individually and as a couple.