Report

Given the widespread adoption and usage of Large Language Models (LLMs), it is crucial to have flexible and interpretable evaluations of their instruction-following ability. Preference judgments between model outputs have become the de facto evaluation standard, despite distilling complex, multi-faceted preferences into a single ranking. Furthermore, as human annotation is slow and costly, LLMs are increasingly used to make these judgments, at the expense of reliability and interpretability. In this work, we propose TICK (Targeted Instruct-evaluation with ChecKlists), a fully automated, interpretable evaluation protocol that structures evaluations with LLM-generated, instruction-specific checklists. We first show that, given an instruction, LLMs can reliably produce high-quality, tailored evaluation checklists that decompose the instruction into a series of YES/NO questions. Each question asks whether a candidate response meets a specific requirement of the instruction. We demonstrate that using TICK leads to a significant increase (46.4% $\to$ 52.2%) in the frequency of exact agreements between LLM judgements and human preferences, as compared to having an LLM directly score an output. We then show that STICK (Self-TICK) can be used to improve generation quality across multiple benchmarks via self-refinement and Best-of-N selection. STICK self-refinement on LiveBench reasoning tasks leads to an absolute gain of $+$7.8%, whilst Best-of-N selection with STICK attains $+$6.3% absolute improvement on the real-world instruction dataset, WildBench. In light of this, structured, multi-faceted self-improvement is shown to be a promising way to further advance LLM capabilities. Finally, by providing LLM-generated checklists to human evaluators tasked with directly scoring LLM responses to WildBench instructions, we notably increase inter-annotator agreement (0.194 $\to$ 0.256).

RAG 2.0 is really about grounding general purpose agents in proprietary enterprise context. Instead of simply one-shot answering a simple question, the agent…

Vision model based document ingestion.

Colivara is a suite of services that allows you to store, search, and retrieve documents based on their visual embedding. ColiVara has state of the art retrieval performance on both text and visual...

Omni turns documents, slide decks, websites and more into the data you need. You'll never need to copy + paste data into spreadsheets again.

Full Stack application for retrieving Stock Data and News using LLM, LangChain and LangGraph - vinay-gatech/stocks-insights-ai-agent

Compare performance, cost, and accuracy of leading Vision Language Models including GPT-4V, Claude 3.5, and open-source alternatives. Real-world testing on document processing tasks.

We investigate whether large language models (LLMs) can successfully perform financial statement analysis in a way similar to a professional human analyst

PaSa -- an advanced paper search agent powered by large language models. It can autonomously make a series of decisions, including invoking search tools, reading papers, and selecting relevant refe...

Python tool for converting files and office documents to Markdown. - microsoft/markitdown

PyMuPDF is a high-performance Python library for data extraction, analysis, conversion & manipulation of PDF (and other) documents.

Extract any data from any document. Resumes, invoices, contracts, academic research, bank statements, utility bills and more – Sensible can parse them all.

The document processing platform built for the next generation.

Reducto is an API that provides high quality data ingestion for large language models (LLMs). It works with any vector database or embedding system. It can parse PDFs, Excel, PowerPoint, and more.

A system for agentic LLM-powered data processing and ETL - ucbepic/docetl

An AI Hedge Fund Team. Contribute to virattt/ai-hedge-fund development by creating an account on GitHub.

But what exactly are MoEs? 🤔 A Mixture of Experts (MoE) is a machine learning framework that resembles a team of specialists, each adept at handling different aspects of a complex task. It's like… — Akshay 🚀 (@akshay_pachaar)

For “R”, our DSE dropped the document processing and improved relevancy modeling by preserving the content integration. Now for “G”, we propose VISA. Aiming to take a step towards more verifiable and intuitive V-RAG.… — Xueguang Ma (@xueguang_ma)

LlamaIndex is a simple, flexible framework for building knowledge assistants using LLMs connected to your enterprise data.

Explore the future of RAG in report generation, enhancing decision-making and resource allocation for businesses.