One resource -> all the info you'll need across all the major LLMs

Launching a new, free, resource. Plus two interesting papers and a prompt template.

Part of the benefit of using a tool like PromptHub is that we make it really easy to test prompts across different models.

Supporting all those models means we need to keep up to date information on key model metrics (context windows, costs, max outputs, etc) and functionality (vision, functions etc).

Believe it or not, keeping that information up to date and accurate is actually harder than it sounds, for a number of reasons:

• Providers share information in various formats.

• Key details are often scattered across multiple sources (Ex: Google’s pricing is on one page, while all other API-related information is nested within the documentation.)

• Prices and features can change over time, like price reductions

That’s why we’ve built and launched the LLM Model Card Directory.

The LLM Model Card Directory

A public-facing, up-to-date source of truth for key information about LLMs.

Here’s what you’ll find in the Model Card Directory

• Key information: Easily access data on token costs, features, context windows, and more.

• Regular updates: We are committed to keeping the directory up-to-date, so you always have the latest information at your fingertips.

• User-friendly access: No account is needed to browse the directory! We wanted to make sure it was accessible for everyone in the community.

Why we are doing this

We are always looking for ways to help the community. It’s why we run this Substack, post videos to Youtube, launched a free prompt generator and publish often on our blog.

More importantly, I want to spare others from the research and digging I have to do whenever we added support for a new model in PromptHub 😂.

I hope the directory is helpful!

2 other papers I’m reading this week

Investigating the Role of Prompting and External Tools in Hallucination Rates of Large Language Models

I love reading and writing about hallucinations. They are still one of the biggest challenges to overcome when working with LLMs. This paper looks at a variety of ways to reduce hallucinations via prompt engineering methods and external tools. Prompt engineering methods like Chain-of-Thought (CoT), Self-Consistency (SC) and Multiagent Debate (MAD), were tested as well as using tools like web search.

Self-Consistency was a winner.

ExpertPrompting: Instructing Large Language Models to be Distinguished Experts

This paper presents a framework that will automatically select an expert persona that is relevant to the task at hand. This is more or less automated persona prompting.

I’ve been skeptical about how effective role based prompting is on performance for non-writing based tasks. You can find papers, like the one above, that point to increases in performance. You can also find other papers like this one and this one, that say the opposite.

Prompt template of the week

System 2 Attention prompting is a prompt engineering method inspired by Daniel Kahneman’s famous distinction between System 1 and System 2 thinking.

System 1 is fast and reactive (moving your hand away from a hot stove), while System 2 involves slow, deliberate thinking (planning a wedding).

System 2 Attention prompting instructs the model to first remove any irrelevant or biased information. This allows the model to think over all the context and the question, before having to answer it.

You can access the template here.

Happy prompting!