AI Engineering Reference: System Design and Real-World Practices

A comprehensive reference for engineers, tech leads, and teams who build AI systems: from choosing an approach, through core techniques (prompting, RAG, fine-tuning, agents), to production engineering, data operations, specialized systems, and governance.

This guide started from practitioner discussions (Reddit engineering threads) and was validated and expanded against primary sources: Anthropic and OpenAI engineering guides, the OWASP Top 10 for LLM Applications (2025), the 12-Factor Agents methodology, OpenTelemetry GenAI conventions, academic work on retrieval and long context, and current regulation (EU AI Act as amended by the 2026 Digital Omnibus). Full references at the end.

Principle: treat AI systems as probabilistic distributed systems, not as ordinary API integrations. Design for containment and recovery, not for a model that never fails.

How to use this document: each section stands alone. Skim the contents, read the sections relevant to your current problem, and use the appendices (failure handling, checklist, anti-patterns, metrics) as working documents during design reviews and launches.

Contents

• Part I. Decisions Before Code
  1. The intervention ladder
  2. API vs self-hosting
  3. Model selection
  4. Build vs buy
• Part II. Core Techniques 5. Prompt engineering that survives production 6. Structured outputs 7. Context engineering 8. RAG system design 9. Embeddings and vector infrastructure 10. Fine-tuning and model adaptation
• Part III. Agentic Systems 11. The architecture ladder: workflows vs agents 12. Tools and the agent-computer interface 13. MCP (Model Context Protocol) 14. Agent memory 15. Multi-agent systems 16. Human-in-the-loop
• Part IV. Production Engineering 17. Reference architecture 18. Core design rules 19. Reliability patterns 20. Security 21. Evaluation strategy 22. Observability 23. Deployment and change management 24. Cost and latency engineering 25. Inference and serving (self-hosted)
• Part V. Data Engineering 26. Pipelines and dataset management 27. Annotation operations 28. Synthetic data 29. Document processing and OCR
• Part VI. Specialized Systems 30. Real-time voice AI 31. Multilingual and Arabic-specific notes
• **Part VII. Product and Organizati