A concise technical reference covering the key parameters that control how LLM-based systems retrieve context and generate text — including 

When a RAG system or semantic search pipeline queries a vector database, it doesn't return 

 matching documents — it returns only the 

Figure: Cosine similarity ranks candidates by angle between vectors; 

 controls how many top results are returned.

from sentence_transformers import SentenceTransformer, util
import torch

model = SentenceTransformer('all-MiniLM-L6-v2')

docs = [
    "quantum computing basics",
    "artificial intelligence overview",
    "deep learning and neural networks",
    "machine learning algorithms",
    "natural language processing",
    "computer vision techniques",
    "reinforcement learning",
    "transformers and attention",
]

doc_embeddings = model.encode(docs, convert_to_tensor=True)
query = "What is artificial intelligence?"
query_embedding = model.encode([query], convert_to_tensor=True)

# Cosine similarity between query and all documents
cos_scores = util.pytorch_cos_sim(query_embedding, doc_embeddings)[0]

# Retrieve top-k results
k = 3
top_k = torch.topk(cos_scores, k=k)

print(f"Query: '{query}'\n")
print(f"Top {k} results:")
for score, idx in zip(top_k.values, top_k.indices):
    print(f"  [{score:.4f}] {docs[idx.item()]}")

# Output:
# [0.7231] artificial intelligence overview
# [0.5843] deep learning and neural networks
# [0.5102] machine learning algorithms


 — measuring the angle between the query vector and each document vector:

cosine_similarity(A, B) = (A · B) / (|A| × |B|)


 specifies the number of highest-ranked items we retrieve. The actual ranking is based on cosine similarity — a probability-like score — not distance metrics like Euclidean or Manhattan distance."

# In a LangChain RAG pipeline, k is set on the retriever
retriever = vectorstore.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 4}   # retrieve top 4 most similar chunks
)


2. Temperature: Controlling Output Randomness

 that controls how random or deterministic the model's token selections are.

Figure: Low temperature sharpens the probability distribution (deterministic); high temperature flattens it (creative/random).

During generation, the model produces a probability distribution over the entire vocabulary at each step. Temperature scales the 

P(token_i) = exp(logit_i / T) / Σ exp(logit_j / T)


from transformers import pipeline

generator = pipeline("text-generation", model="gpt2")

prompt = "The future of artificial intelligence is"

# Low temperature — deterministic, focused output
low_temp = generator(
    prompt,
    max_new_tokens=30,
    temperature=0.2,
    do_sample=True
)

# High temperature — creative, diverse output
high_temp = generator(
    prompt,
    max_new_tokens=30,
    temperature=1.4,
    do_sample=True
)

print("Low temp: ", low_temp[0]['generated_text'])
print("High temp:", high_temp[0]['generated_text'])


, is an alternative (or complement) to temperature for controlling generation diversity.

Instead of sampling from the full vocabulary, top-p selects the 

Select tokens t_1, t_2, ..., t_m  such that:
    P(t_1) + P(t_2) + ... + P(t_m) ≥ p


Token probabilities (sorted descending):
  "learning"   → 0.45   cumsum: 0.45
  "models"     → 0.25   cumsum: 0.70
  "systems"    → 0.15   cumsum: 0.85
  "networks"   → 0.08   cumsum: 0.93  ← cumsum exceeds 0.9 here
  "algorithms" → 0.04   cumsum: 0.97
  ... (thousands more low-prob tokens)

Nucleus (p=0.9): {"learning", "models", "systems", "networks"}
Sample only from these 4 tokens; ignore the rest.


3.2 Why Top-p is Better Than Fixed Top-k Sampling

 to the model's confidence at each step. When the model is sure, the nucleus is small; when uncertain, it expands.

from transformers import pipeline

generator = pipeline("text-generation", model="gpt2")

result = generator(
    "Artificial intelligence will",
    max_new_tokens=40,
    do_sample=True,
    top_p=0.9,          # nucleus sampling: cumulative prob threshold
    temperature=0.8,    # can be combined with top-p
)
print(result[0]['generated_text'])


# Using the Anthropic / OpenAI API style
import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=200,
    messages=[{"role": "user", "content": "Explain neural networks briefly."}],
    temperature=0.7,    # lower = more deterministic
    top_p=0.9,          # nucleus sampling threshold
)
print(response.content[0].text)


4. Parameter Interactions & Practical Defaults

4.1 How All Three Parameters Work Together

User Query
    │
    ▼
[RETRIEVAL STAGE]
    Encode query → cosine similarity vs. document embeddings
    Return top-k chunks  ← controlled by k
    │
    ▼
[GENERATION STAGE]
    LLM generates token-by-token using retrieved context
    At each step:
      1. Compute logits over vocabulary
      2. Apply temperature scaling  ← controlled by temperature
      3. Apply nucleus sampling     ← controlled by top-p
      4. Sample next token
    │
    ▼
Final Response


# Full RAG + generation pipeline with all parameters configured
from langchain.chains import RetrievalQA
from langchain.chat_models import ChatOpenAI

llm = ChatOpenAI(
    model="gpt-4",
    temperature=0.3,   # low = factual, deterministic answers
    top_p=0.9,         # nucleus sampling
    max_tokens=512
)

retriever = vectorstore.as_retriever(
    search_kwargs={"k": 4}   # retrieve top 4 chunks
)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=retriever,
    return_source_documents=True
)

response = qa_chain({"query": "What is nucleus sampling?"})
print(response["result"])


Generated from session notes on AI & Cloud Blueprint — Retrieval & Generation Parameters.