Questions to ask about building foundation models
- samrodriques
- Jul 23, 2023
- 1 min read
I often hear proposals to build new foundation models for biology. Here is the list of questions I ask. I rarely get past question 1.
What is the core task that the model performs/the core thing the model predicts? I.e., on a single iteration of the model, what is the input and what is the output?
For LLMs, the answer is "you give it N words and it provides a probability distribution over word N+1"
My experience is that 90% of proposals fail at this stage.
What is the loss metric on the core task?
For LLMs, the answer is cross-entropy or perplexity
What emergent behavior do you think the model will have?
For LLMs, the answer is "natural language processing." For other people, it might be "predicting enzyme functionality" or something.
What are the evals you will use to evaluate whether this emergent behavior is emerging?
For LLMs, e.g. Winogrande
Is there evidence that performance on the evals increases with scale? How expensive is it to get that evidence?
Questions about datasets.
Do they exist?
How many tokens?
Are they high quality? How does the person know if they are high quality?
More technical questions
Tokenization
What is the scaling method?
Architecture questions. Dense? Mixture of experts? Multiple read heads? etc.
I had a really positive experience with The Researchers Hub as a UK student trying to balance multiple academic tasks. Their dissertation help gave me the structure and guidance I needed to improve the quality of my writing. Alongside this, their research assistance and thesis guidance made even the toughest topics easier to approach. The team was supportive, professional, and quick to respond whenever I had questions. For students in the UK who need dependable academic support and trustworthy writing services, I would definitely recommend The Researchers Hub as a reliable platform.
Exploring questions to ask about building foundation models often starts with understanding the dataset potential biases scalability and how the model handles unexpected inputs. It’s also essential to consider transparency and interpretability, which help ensure the model can be trusted in real world applications. On a personal note, I recently got a pawtenant and managing my daily tasks became challenging. Using resources like The Pets Help made it easier to stay organized while caring for my companion. This experience highlighted the comfort and stability an emotional support animal can provide.
In the middle of reading this post on foundation models, I was reminded of a time during my studies when exams piled up and the pressure felt endless. It was in those overwhelming nights that I realized how an online exam help service could make a difference not by replacing effort, but by offering guidance when balance and focus seemed nearly impossible to maintain.
My experience with Rapid Assignment Help has been outstanding. Their Help With Finance Assignment service provided me with in-depth solutions that were customized to my needs. The expert team handled my topic with professionalism and delivered a paper that was 100% original. I loved how they included real-world examples to explain complex financial concepts. Their service is affordable, reliable, and definitely worth it. Thanks to Rapid Assignment Help, my grades improved, and I feel more confident in finance.
With increasing awareness about alternative medicine, cupping therapy in bangalore is now considered a safe, non-invasive treatment that addresses a wide range of health concerns effectively.