ChatModuleClient
Prerequisites
- Install the required packages using the following commands:
pip install --no-deps --pre --force-reinstall mlc-ai-nightly-cu118 mlc-chat-nightly-cu118 -f https://mlc.ai/wheels
pip install transformers
git lfs install
Adjust the pip wheels according to your OS/platform by referring to the provided commands in MLC packages.
Running MLC Llama-2 models
- Create a directory for prebuilt models:
- Clone the necessary libraries from the repository:
- Choose a Llama-2 model from MLC LLMs and clone the model repository:
- Initialize the
ChatModuleClient
within your program with the desired parameters. Here's an example call:
llama = dspy.ChatModuleClient(model='dist/prebuilt/mlc-chat-Llama-2-7b-chat-hf-q4f16_1', model_path='dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-cuda.so')