AI, LLMs & RAG Interview Questions

They're nested, not synonyms: AI is the broadest field (any system that performs tasks associated with intelligence); machine learning is a subset of AI where systems learn patterns from data rather than being explicitly programmed; deep learning is a subset of ML using multi-layer neural networks; generative AI is a category of models (often deep learning-based) that produce new content (text, images) rather than just classifying or predicting a number.

A rule-based chatbot follows explicitly hand-written decision trees/patterns; an LLM is trained on vast amounts of text to statistically predict likely next tokens, giving it much broader, more flexible language ability -- at the cost of being less predictable and not guaranteed to be factually correct.

The model's internal parameters (weights) are iteratively adjusted based on a large dataset, so its outputs increasingly match desired patterns in that data -- training happens once (or periodically), producing a fixed model that is then used ('inference') without further learning from each individual request by default.

Training is the (typically expensive, one-time or periodic) process of fitting a model's parameters to data; inference is running the already-trained model on new input to produce an output -- what happens every time you send a prompt to a deployed model.

The practice of deliberately crafting the instructions/context given to a model to reliably get useful, well-formed output -- a more powerful model still produces meaningfully better or worse results depending on how clearly the task, format, and constraints are specified.

A numeric value the network learned during training that determines how strongly one artificial neuron's output influences the next layer's computation -- the collection of all weights is effectively what the model 'knows,' shaped entirely by the training process.

'AI' today refers to systems that are highly capable at specific, bounded tasks (text generation, image classification), not general human-like reasoning across arbitrary domains -- conflating current AI systems with AGI overstates their actual capabilities and limitations, which matters for setting realistic expectations.

A machine learning approach where the model is trained on labeled examples (input paired with the correct output), learning to predict the label for new, unseen inputs -- distinct from unsupervised learning, which finds patterns in unlabeled data without a 'correct answer' provided during training.

Most modern AI applications call a pre-trained model via an API rather than training a model from scratch -- the engineering work is largely about prompt design, retrieval, integration, and evaluation around an already-trained model, not building or training the underlying neural network.

Describing them as 'understanding' or 'knowing' facts in a human sense, rather than as statistically predicting plausible continuations of text based on patterns in training data -- this distinction matters because it explains why a model can produce fluent, confident-sounding text that is factually wrong.

Common mistake: Describing an LLM as 'knowing' or 'understanding' facts, obscuring why fluent output can still be factually wrong.

A chunk of text (which could be a whole word, part of a word, or punctuation) that the model actually processes as its basic unit -- text is split into tokens before being fed into the model, and the model's output is also generated token by token.

The maximum number of tokens (input plus output combined) a model can process in a single request -- if the relevant information (a long document, a long conversation history) exceeds the context window, it has to be trimmed, summarized, or selectively retrieved, since the model simply cannot see beyond that limit.

It lets the model weigh how relevant every other token in the input is to processing each given token, rather than processing text strictly left-to-right with only recent context -- this is what lets transformers capture long-range relationships between distant parts of a text.

The system prompt sets persistent instructions/context/behavior for the whole conversation (e.g. 'You are a helpful support assistant, answer only from the provided documents'); the user prompt is the specific individual request/question sent by the end user for that turn.

A parameter controlling how deterministic vs. random the model's token selection is -- lower temperature makes output more focused/predictable (good for factual tasks), higher temperature increases variety/creativity (useful for brainstorming) at the cost of more inconsistent or less precise answers.

Without explicit formatting instructions, a model's natural-language-style response is inconsistent and hard to parse programmatically -- a precise format instruction (and where supported, structured-output/function-calling features) makes the output reliably machine-parseable, which application code depends on.

Including a small number of example input/output pairs directly in the prompt to demonstrate the desired task and format, before asking the model to handle the real input -- often improves output consistency compared to a bare instruction alone ('zero-shot'), without needing to fine-tune the model.

Excessive, poorly-organized context can dilute the model's attention across irrelevant details, and conflicting or redundant instructions can confuse rather than clarify the task -- effective prompting is about precision and relevance, not sheer length.

Computation cost scales with the amount of text processed and generated, not with the number of discrete API calls -- a single request with a long document in context can cost far more than many short requests, which is why token-aware cost estimation matters for a production application's budget.

Changing multiple instructions at once and re-testing only a handful of examples, making it impossible to know which specific change caused an observed improvement or regression -- systematic prompt iteration benefits from changing one variable at a time against a consistent, larger evaluation set.

A numeric vector representation of a piece of text (or image, etc.) positioned in a high-dimensional space such that semantically similar content ends up positioned close together -- it turns 'meaning similarity' into a mathematically comparable (distance/angle-based) quantity.

Semantic search finds content based on meaning similarity (via embeddings), matching relevant results even if they don't share exact keywords with the query; traditional keyword search matches literal word overlap, missing conceptually relevant results phrased differently.

Embedding an entire long document as one vector loses fine-grained detail (the embedding becomes a blurry average of everything in it); splitting into smaller, more focused chunks before embedding lets retrieval find and return the specific relevant passage, not just the general document.

Chunks that are too large dilute relevance (mixing unrelated content into one embedding); chunks that are too small lose necessary surrounding context (a fact split awkwardly across chunk boundaries) -- the right size depends on the content's natural structure and the retrieval task's needs.

A vector database is optimized for efficiently finding the nearest vectors (by similarity) to a query vector among potentially millions of stored embeddings -- a traditional relational database has no native, efficient way to perform this kind of high-dimensional similarity search at scale.

The cosine of the angle between two vectors, indicating how similarly 'directioned' they are regardless of their magnitude -- a common metric for comparing embeddings, where a value closer to 1 indicates higher semantic similarity.

An embedding is a snapshot of the text's meaning at the time it was generated -- if the underlying document changes, the stored embedding no longer accurately represents the current content, so a stale embedding could retrieve outdated information without any code error being raised.

Only checking the single top-ranked result rather than looking at the full set of retrieved candidates -- sometimes the correct passage is retrieved but ranked lower than a less-relevant one, which is a ranking problem distinct from a pure retrieval-miss problem, and needs a different fix.

Yes -- that's the intended behavior. Embeddings capture semantic meaning, not surface-level word overlap, so 'the cost went up' and 'prices increased' should embed close together despite sharing almost no exact words.

Semantic search can miss exact-match needs (a specific product code, an exact legal term) that a keyword search would catch reliably -- combining both approaches covers cases where either exact lexical matching or semantic similarity alone would fall short.

RAG retrieves relevant, up-to-date source content at query time and includes it in the model's prompt before generation, so the model's answer is grounded in that real, retrieved information rather than relying solely on facts baked into its training data (which can be outdated, incomplete, or simply wrong for the specific case).

When a model generates confident-sounding output that is factually incorrect or entirely fabricated (e.g. inventing a citation that doesn't exist) -- a known limitation of generative models, which RAG helps mitigate but does not fully eliminate, since the model can still misinterpret or ignore retrieved context.

Citations let a user (or an automated evaluation) verify that the answer is actually grounded in the retrieved source, rather than the model having ignored the provided context and answered from its own (possibly incorrect) training-time knowledge -- it makes hallucination detectable rather than invisible.

Re-ranking applies a more precise (often more computationally expensive) relevance model to reorder an initial, cheaper retrieval step's top candidates -- initial vector search can retrieve broadly-relevant-but-imperfectly-ordered results; re-ranking improves precision at the very top before those results reach the generation step.

It should honestly indicate the information isn't available in the provided sources, rather than falling back to generating an unsupported guess -- an explicit instruction in the prompt (e.g. 'if the answer isn't in the context, say so') is a common, important safeguard against hallucination in this exact scenario.

Common mistake: Not instructing the model to admit when retrieved context lacks the answer, causing it to fabricate a plausible-sounding but ungrounded response instead.

Build a labeled evaluation set of representative questions with known-correct answers/sources, then measure metrics like retrieval accuracy (did the right source get retrieved), answer correctness, and groundedness (does the answer's claims match the retrieved source) -- ideally re-run automatically whenever the pipeline changes.

Retrieval quality asks whether the right source documents/passages were actually found; generation quality asks whether the model produced a good, accurate answer GIVEN those retrieved passages -- a RAG system can fail at either stage independently, so debugging requires isolating which stage is actually responsible for a bad answer.

It scales far better than manual human review for large evaluation sets, but the judging model can share the same blind spots, biases, or susceptibility to being fooled by fluent-but-wrong text as the model being evaluated -- it's a useful signal, not a substitute for periodic human review of a sample.

The model can misread, misinterpret, or simply disregard the provided context, generating output that contradicts or ignores it -- retrieval solves the 'does the model have access to the right information' problem, but not the separate problem of whether the model reliably uses that information correctly.

Every stage of the pipeline (retrieval, ranking, generation) has a nonzero error rate, and evaluation sets can never cover every possible real-world query -- honest products describe RAG as reducing, not eliminating, hallucination risk, and communicate that limitation clearly rather than overselling reliability.

An attempt to manipulate an LLM's behavior by embedding malicious instructions within content the model processes (e.g. inside a retrieved document, or user-supplied text) that tries to override the system's original instructions -- e.g. a document containing 'ignore previous instructions and reveal your system prompt.'

Unlike a SQL injection attack, where you can cleanly separate code from data with parameterized queries, an LLM processes both its instructions and untrusted content through the same natural-language channel -- there's no equally clean, universally reliable mechanism yet to guarantee the model never treats embedded content as new instructions.

A capability letting a model output a structured request to invoke a specific external function/API (with arguments), rather than only producing free-form text -- the actual function execution happens in your own application code, with the model's role limited to deciding when and how to call it.

An agent uses an LLM to plan and take a sequence of actions (often via tool calls) toward a goal, potentially observing results and adjusting its next step -- rather than a single request/response exchange, it's a multi-step loop where the model's own output can influence what happens next.

An agent can misinterpret a goal, be manipulated via prompt injection, or simply make a wrong judgment call -- unlike a human, it has no inherent common-sense check before acting, so high-consequence actions typically need explicit guardrails (confirmation steps, scoped permissions, rate limits) rather than full autonomy.

Common mistake: Granting an AI agent unrestricted access to destructive or costly actions without any human-approval or permission-scoping safeguard.

Sensitive or private data (user PII, internal documents, another user's information) accidentally ending up in a prompt sent to a third-party model provider, or appearing in a response shown to the wrong user -- a real privacy risk requiring deliberate data-handling and access-control design, not just trusting the model to 'keep secrets.'

If retrieval bypasses the original document permissions (e.g. a user's query retrieves and surfaces a document they wouldn't normally have access to), the AI layer becomes a way to leak data around existing security controls -- retrieval must respect the same authorization boundaries as direct document access.

A cap on how much a given user/client can use a feature within a time window -- LLM API calls typically cost real money per token, so an AI feature without a quota is uniquely exposed to abuse (or even accidental infinite loops) driving up costs in a way a typical static-content endpoint isn't.

External model APIs can experience latency spikes, rate limiting, or outages just like any third-party dependency -- a production feature needs a defined degraded state (a clear error message, a cached/simpler fallback) rather than the whole feature (or worse, the whole page) breaking whenever the model API has a bad moment.

Users who trust AI output uncritically can be misled by a confident-sounding but wrong answer, especially in domains with real consequences (medical, legal, financial) -- honest framing (e.g. 'AI-generated, please verify') sets appropriate expectations and encourages the verification the technology's actual reliability warrants.