Skip to content

ChatModuleClient

Prerequisites

  1. Install the required packages using the following commands:
pip install --no-deps --pre --force-reinstall mlc-ai-nightly-cu118 mlc-chat-nightly-cu118 -f https://mlc.ai/wheels
pip install transformers
git lfs install

Adjust the pip wheels according to your OS/platform by referring to the provided commands in MLC packages.

Running MLC Llama-2 models

  1. Create a directory for prebuilt models:
mkdir -p dist/prebuilt
  1. Clone the necessary libraries from the repository:
git clone https://github.com/mlc-ai/binary-mlc-llm-libs.git dist/prebuilt/lib
cd dist/prebuilt
  1. Choose a Llama-2 model from MLC LLMs and clone the model repository:
git clone https://huggingface.co/mlc-ai/mlc-chat-Llama-2-7b-chat-hf-q4f16_1
  1. Initialize the ChatModuleClient within your program with the desired parameters. Here's an example call:

llama = dspy.ChatModuleClient(model='dist/prebuilt/mlc-chat-Llama-2-7b-chat-hf-q4f16_1', model_path='dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-cuda.so')
Please refer to the official MLC repository for more detailed information and documentation.