AI Agents & Automation interview questions

Prevent shipped updates from breaking tasks that used to work.

Agent Evaluation · multiple choice · easy · 6 min

Compute unit economics for an AI support agent.

LLM Product Metrics · numeric · easy · 5 min

Choose the scoreboard for an AI support agent.

LLM Product Metrics · scenario · medium · 15 min

Diagnose why a high-scoring agent loses real users.

Agent Evaluation · scenario · medium · 20 min

You're shipping an AI support agent.

Agent Evaluation · multiple choice · medium · 6 min

Your team uses a strong LLM to automatically grade your agent's outputs ('LLM-as-judge') at scale.

Agent Evaluation · scenario · medium · 15 min

You're standing up evaluation for a new AI coding/support agent.

Agent Evaluation · scenario · medium · 15 min

Your retrieval-augmented (RAG) assistant sometimes states facts not supported by the retrieved documents.

LLM Product Metrics · multiple choice · medium · 6 min

You launched an AI agent that can take actions (issue refunds, change orders) for customers.

LLM Product Metrics · scenario · medium · 15 min

An agent solves 70% of tasks on the first attempt and 90% within three attempts.

LLM Product Metrics · multiple choice · easy · 6 min

As the product manager at Booking.com, design a generative-AI assistant that creates personalized travel itineraries using destination, activity, and accommodation preferences.

LLM Product Metrics · scenario · medium · 20 min

You are designing a research workflow that must search sources, extract evidence, draft a report, and verify citations.

Agent Evaluation · scenario · hard · 20 min

An email assistant can read messages, search internal documents, draft replies, and create calendar events.

Agent Evaluation · scenario · hard · 22 min

A finance agent can read invoices, prepare payments, and submit them to the bank.

LLM Product Metrics · multiple choice · easy · 7 min

An agent handles 10,000 tasks per month.

LLM Product Metrics · numeric · medium · 12 min