Quasar: A Programming Language Specialized for LLM Code Actions

Large language models (LLMs) often call external tools to solve tasks. An effective strategy is for LLMs to write code to do so, enabling them to use complex control flow such as conditionals and loops. Such code actions are typically generated as Python code, since LLMs are quite proficient at it. However, there are many programming language features that can support effective code actions that are difficult to implement in Python. Our key insight is to separate internal code that captures program logic from external calls to tools that interact with the world. Then, we can implement a new feature by (1) annotating external calls with their effects that are relevant to that feature, and (2) modifying the execution of the internal code to track this information. We propose a novel programming language called QUASAR that implements this idea. To illustrate its utility, we implement several useful features on top of Quasar to enhance code actions: conformal prediction for uncertainty quantification to mitigate hallucinations, access control with batched user queries to improve security, and autoparallelization of external calls to improve performance.