GitHub - microsoft/LLMLingua: To speed up LLMs' inference an

To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss. - GitHub - microsoft/LLMLingua: To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.

Related Keywords

Llmlingua Longllmlingua , Yuqing Yang , Xufang Luo , Qianhui Wu , Lili Qiu , Huiqiang Jiang , Dongsheng Li , Microsoft , Association For Computational Linguistics , Compressing Prompts , Accelerated Inference , Large Language Models , Chin Yew Lin , Long Context Scenarios , Prompt Compression , Under Review , Large Language , Empirical Methods , Natural Language , Online Meeting , Contributor License Agreement , Microsoft Open Source Code ,