components.models
get_model
def get_model(
model: LLMModel
logger:
temperature: float
)Get an interface for interacting with an LLM
Uses Haystack Generators to provide an interface to a model. If the model_name is a GPT, then the interface is to a remote OpenAI model. Otherwise, uses a LlamaCppGenerator to start a llama.cpp model and provide an interface.
Parameters
model: LLMModel
The name of the model
temperature: float
The temperature for the model
logger: logging.Logger|None
The logger for the model
Returns
object
An interface to generate text using an LLM
Implemented models
| Model name | Summary |
|---|---|
| llama-3.1-8b | Recommended Meta’s Llama 3.1 with 8 billion parameters, quantized to 4 bits |
| llama-2-7b-chat | Meta’s Llama 2 with 7 billion parameters, quantized to 4 bits |
| llama-3-8b | Meta’s Llama 3 with 8 billion parameters, quantized to 4 bits |
| llama-3-70b | Meta’s Llama 3 with 70 billion parameters, quantized to 4 bits |
| gemma-7b | Google’s Gemma with 7 billion parameters, quantized to 4 bits |
| llama-3.2-3b | Meta’s Llama 3.2 with 3 billion parameters, quantized to 6 bits |
| mistral-7b | Mistral at 7 billion parameters, quantized to 4 bits |
| kuchiki-l2-7b | A merge of several models at 7 billion parameters, quantized to 4 bits |
| tinyllama-1.1b-chat | Llama 2 extensively pre-trained, with 1.1 billion parameters, quantized to 4 bits |
| biomistral-7b | Mistral at 7 billion parameters, pre-trained on biomedical data, qunatized to 4 bits |
| qwen2.5-3b-instruct | Alibaba’s Qwen 2.5 at 3 billion parameters, quantized to 5 bits |
| airoboros-3b | Llama 2 pre-trained on the airoboros 3.0 dataset at 3 billion parameters, quantized to 4 bits |
| medicine-chat | Llama 2 pre-trained on medical data, quantized to 4 bits |
| medicine-llm-13b | Llama pre-trained on medical data at 13 billion parameters, quantized to 4 bits |
| med-llama-3-8b-v1 | Llama 3 at 8 billion parameters, pre-trained on medical data, quantized to 5 bits |
| med-llama-3-8b-v2 | Llama 3 at 8 billion parameters, pre-trained on medical data, quantized to 4 bits |
| med-llama-3-8b-v3 | Llama 3 at 8 billion parameters, pre-trained on medical data, quantized to 3 bits |
| med-llama-3-8b-v4 | Llama 3 at 8 billion parameters, pre-trained on medical data, quantized to 3 bits |
If you would like to add a model, raise an issue