Skip to main content

Anyscale

Anyscale​

Prerequisites​

  • Anyscale api_key and api_base (for non-cached examples). Set these within your developer environment .env as follows:
ANYSCALE_API_BASE = ...
ANYSCALE_API_KEY = ...

which will be retrieved within the Anyscale Client as:

self.api_base = os.getenv("ANYSCALE_API_BASE")
self.token = os.getenv("ANYSCALE_API_KEY")

Setting up the Anyscale Client​

The constructor initializes the HFModel base class to support the handling of prompting models. This requires the following parameters:

Parameters:

  • model (str): ID of model hosted on Anyscale endpoint.
  • **kwargs: Additional keyword arguments to configure the Anyscale client.

Example of the Anyscale constructor:

class Anyscale(HFModel):
    def __init__(self, model, **kwargs):

Under the Hood​

_generate(self, prompt, use_chat_api=False, **kwargs):​

Parameters:

  • prompt (str): Prompt to send to Anyscale.
  • use_chat_api (bool): Flag to use the Anyscale Chat models endpoint. Defaults to False.
  • **kwargs: Additional keyword arguments for completion request.

Returns:

  • dict: dictionary with prompt and list of response choices.

Internally, the method handles the specifics of preparing the request prompt and corresponding payload to obtain the response.

The Anyscale token is set within the request headers to ensure authorization to send requests to the endpoint.

If use_chat_api is set, the method sets up Anyscale url chat endpoint and prompt template for chat models. It then retrieves the generated JSON response and sets up the completions list by retrieving the response's message : content.

If use_chat_api is not set, the method uses the default Anyscale url endpoint. It similarly retrieves the generated JSON response and but sets up the completions list by retrieving the response's text as the completion.

Finally, after processing the requests and responses, the method constructs the response dictionary with two keys: the original request prompt and choices, a list of dictionaries representing generated completions with the key text holding the response's generated text.

Using the Anyscale client​

anyscale = dspy.Anyscale(model='meta-llama/Llama-2-70b-chat-hf')

Sending Requests via Anyscale Client​

  1. Recommended Configure default LM using dspy.configure.

This allows you to define programs in DSPy and simply call modules on your input fields, having DSPy internally call the prompt on the configured LM.

dspy.configure(lm=anyscale)

#Example DSPy CoT QA program
qa = dspy.ChainOfThought('question -> answer')

response = qa(question="What is the capital of Paris?") #Prompted to anyscale
print(response.answer)
  1. Generate responses using the client directly.
response = anyscale(prompt='What is the capital of Paris?')
print(response)

Written By: Arnav Singhvi