Skip to main content

Together

Togetherß​

Adapted from documentation provided by https://github.com/insop​

Prerequisites​

  • Together api_key and api_base (for non-cached examples). Set these within your developer environment .env as follows:
TOGETHER_API_BASE = ...
TOGETHER_API_KEY = ...

which will be retrieved within the Together Client as:

self.api_base = os.getenv("TOGETHER_API_BASE")
self.token = os.getenv("TOGETHER_API_KEY")

Setting up the Together Client​

The constructor initializes the HFModel base class to support the handling of prompting models. This requires the following parameters:

Parameters:

  • model (str): ID of model hosted on Together endpoint.
  • **kwargs: Additional keyword arguments to configure the Together client.

Example of the Together constructor:

class Together(HFModel):
    def __init__(self, model, **kwargs):

Under the Hood​

_generate(self, prompt, use_chat_api=False, **kwargs):​

Parameters:

  • prompt (str): Prompt to send to Together.
  • use_chat_api (bool): Flag to use the Together Chat models endpoint. Defaults to False.
  • **kwargs: Additional keyword arguments for completion request.

Returns:

  • dict: dictionary with prompt and list of response choices.

Internally, the method handles the specifics of preparing the request prompt and corresponding payload to obtain the response.

The Together token is set within the request headers to ensure authorization to send requests to the endpoint.

If use_chat_api is set, the method sets up Together url chat endpoint and prompt template for chat models. It then retrieves the generated JSON response and sets up the completions list by retrieving the response's message : content.

If use_chat_api is not set, the method uses the default Together url endpoint. It similarly retrieves the generated JSON response and but sets up the completions list by retrieving the response's text as the completion.

Finally, after processing the requests and responses, the method constructs the response dictionary with two keys: the original request prompt and choices, a list of dictionaries representing generated completions with the key text holding the response's generated text.

Using the Together client​

together = dspy.Together(model="mistralai/Mistral-7B-v0.1")

Sending Requests via Together Client​

  1. Recommended Configure default LM using dspy.configure.

This allows you to define programs in DSPy and simply call modules on your input fields, having DSPy internally call the prompt on the configured LM.

dspy.configure(lm=together)

#Example DSPy CoT QA program
qa = dspy.ChainOfThought('question -> answer')

response = qa(question="What is the capital of Paris?") #Prompted to together
print(response.answer)
  1. Generate responses using the client directly.
response = together(prompt='What is the capital of Paris?')
print(response)

Written By: Arnav Singhvi