Cost and latency
A few cents per token. Thousands of dollars a month at scale. And there is a right order for bringing the cost down.
1 steps
60 XP
A free account is needed
Start the lesson →
Sources
- Anthropic - · Prompt Caching Documentation · docs.anthropic.com
- Chen, L. et al. 2023 · FrugalGPT: How to Use Large Language Models While Reducing Cost · arXiv:2305.05176
- Kwon, W. et al. 2023 · Efficient Memory Management for LLM Serving with PagedAttention · SOSP 2023
ML Academy · an interactive machine learning course that runs in your browser ·
All lessons