Expert Support case study: Bolstering a RAG app with LLM-as-a-Judge
•
24
Aymeric Roucher
m-ric
AI & ML interests
Leading Agents at Hugging Face 🤗
Recent Activity
updated
a model
about 11 hours ago
rhymes-ai/Aria
liked
a Space
about 13 hours ago
Nymbo/Flux.1-dev-Controlnet-Upscaler
updated
a Space
about 19 hours ago
AIEnergyScore/launch-computation-example
Articles
Organizations
Posts
74
Post
517
𝗦𝗵𝗼𝘄𝗨𝗜: 𝗮 𝘀𝗺𝗮𝗹𝗹 𝗲𝗻𝗱-𝘁𝗼-𝗲𝗻𝗱 𝗮𝗴𝗲𝗻𝘁 𝘁𝗵𝗮𝘁 𝗰𝗮𝗻 𝗻𝗮𝘃𝗶𝗴𝗮𝘁𝗲 𝗮𝗻𝘆 𝗨𝗜 𝗮𝗻𝗱 𝗼𝘂𝘁𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝘀 𝗺𝘂𝗰𝗵 𝗯𝗶𝗴𝗴𝗲𝗿 𝘀𝘆𝘀𝘁𝗲𝗺𝘀! 📲
A team from NUS and Microsoft just released an agent that can act on any UI (Desktop, Android, Web) without needing additional text information. It works extremely well : they applied their method on a tiny Qwen2-VL-2B, and they managed to beat methods that use either much more powerful vision models (like GPT-4V) without using any additional info (e.g. leveraging the DOM of a webpage) like previous methods did ! 👏👏
They started from the idea that most existing methods rely heavily on text, which makes them less generalizable, while letting aside rich UI structure that user actually rely on when navigating this interfaces.
⚙️ They put several good ideas to work:
💡 Simplify screenshots to the max:
They prune a lot the heavy visual content of UI screenshots, by removing cloned image patches (like any vast patch of the same color will be reduced to a small patch, while maintaining positional embeddings), then group patches from the same GUI elements together to simplify even further
💡 Build a truly generalist dataset:
To train a general UI agent, you need trajectories from each possible UI, and express them in a common language. Authors merge datasets like OmniAct for Desktop, Mind2Web for websites, AMEX for Android trajectories to create a high-quality and diverse dataset.
➡️ Nice results ensued:
They fine-tune a tiny Qwen-2-VL-2B on their method, and it reaches SOTA on several task (element identification, web navigation), even beating methods that either use additional info from the DOM or use much bigger VLMS like GPT-4v! 🏆
And performance could certainly jump with a slightly bigger vision model. Let's hope the community builds this soon! 🚀
Paper added to my "Agents" collection 👉 m-ric/agents-65ba776fbd9e29f771c07d4e
A team from NUS and Microsoft just released an agent that can act on any UI (Desktop, Android, Web) without needing additional text information. It works extremely well : they applied their method on a tiny Qwen2-VL-2B, and they managed to beat methods that use either much more powerful vision models (like GPT-4V) without using any additional info (e.g. leveraging the DOM of a webpage) like previous methods did ! 👏👏
They started from the idea that most existing methods rely heavily on text, which makes them less generalizable, while letting aside rich UI structure that user actually rely on when navigating this interfaces.
⚙️ They put several good ideas to work:
💡 Simplify screenshots to the max:
They prune a lot the heavy visual content of UI screenshots, by removing cloned image patches (like any vast patch of the same color will be reduced to a small patch, while maintaining positional embeddings), then group patches from the same GUI elements together to simplify even further
💡 Build a truly generalist dataset:
To train a general UI agent, you need trajectories from each possible UI, and express them in a common language. Authors merge datasets like OmniAct for Desktop, Mind2Web for websites, AMEX for Android trajectories to create a high-quality and diverse dataset.
➡️ Nice results ensued:
They fine-tune a tiny Qwen-2-VL-2B on their method, and it reaches SOTA on several task (element identification, web navigation), even beating methods that either use additional info from the DOM or use much bigger VLMS like GPT-4v! 🏆
And performance could certainly jump with a slightly bigger vision model. Let's hope the community builds this soon! 🚀
Paper added to my "Agents" collection 👉 m-ric/agents-65ba776fbd9e29f771c07d4e
Post
891
Need a measurement for traction of a GitHub repo, a more reliable one than Github star history? (which is a bit too hype-driven) 📈
➡️ I've made a Space to visualize PyPI downloads.
Try it here 👉 m-ric/package-download-history
➡️ I've made a Space to visualize PyPI downloads.
Try it here 👉 m-ric/package-download-history
Collections
14
-
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
Paper • 2206.10789 • Published • 4 -
Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws
Paper • 2401.00448 • Published • 28 -
Training Compute-Optimal Large Language Models
Paper • 2203.15556 • Published • 10 -
Scaling Laws for Neural Language Models
Paper • 2001.08361 • Published • 6
spaces
25
pinned
Running
66
🗺️🏕️
AI Travel Planner
Plan your next vacation with the help of an AI!
pinned
Running
187
🍫👀
Chunk Visualizer
Pick a text splitter => visualize chunks. Great for RAG.
pinned
Running
on
Zero
118
✍👀
Beam Search Visualizer
View how beam search decoding works, in detail!
pinned
Running
101
🤔📊
Agent Data Analyst
Need to analyze data? Let a Llama-3.1 agent do it for you!
Running
10
📈🚀
Package Download History
Github Star History, but for PyPI package downloads.
Running
on
CPU Upgrade
11
📈🚀
Llm Race To The Top
View Chatbot Arena ELO of top models increasing
models
3
datasets
15
m-ric/newsletter
Updated
•
45
m-ric/songs_july_24
Viewer
•
Updated
•
85.6k
•
41
m-ric/huggingface_doc_qa_eval
Viewer
•
Updated
•
65
•
395
•
6
m-ric/agents_medium_benchmark
Viewer
•
Updated
•
172
•
62
•
3
m-ric/songs
Preview
•
Updated
•
20
m-ric/rag_scores_cookbook
Viewer
•
Updated
•
5
•
76
m-ric/images
Viewer
•
Updated
•
1
•
18
m-ric/agents_small_benchmark
Viewer
•
Updated
•
100
•
58
•
8
m-ric/huggingface_doc
Viewer
•
Updated
•
2.65k
•
2.21k
•
9
m-ric/transformers_documentation_en
Viewer
•
Updated
•
359
•
58