Android Bench is a benchmark for evaluating Large Language Models (LLMs) on tasks specific for Android development.
We are continuously refining and creating new tasks, and this repository aims to gather even more challenging tasks to further evolve LLMs abilities and help developers in Android agentic development.
We are looking for complex, realistic tasks, especially those that are a common pitfall for the Android Developers. More about what are we looking for in a task can be found in task-guideline.md
To ensure smooth and easy setup up Android Bench is using Harbor framework.
Important
We will be reviewing submissions, but do not guarantee that the tasks will be merged into the official Android Bench dataset.
We are sharing an opportunity for you to contribute your own tasks to the benchmark. This gives you a chance to let us know which tasks and unique use cases are important for your workflow, so we can represent it within the benchmark.
Interested in the process? You can read our guide in CONTRIBUTING.md
Learn more information about Android Bench on the Android Bench website.
Learn more about Harbor framework and what it provides.
We are using Apache 2.0 License - see LICENSE for more information.
Contributions to this project must be accompanied by a Contributor License Agreement (CLA). You (or your employer) retain the copyright to your contribution; this simply gives us permission to use and redistribute your contributions as part of the project.
If you or your current employer have already signed the Google CLA (even if it was for a different project), you probably don't need to do it again.
Visit https://cla.developers.google.com/ to see your current agreements or to sign a new one.
This project follows Google's Open Source Community Guidelines.