Skip to main content

teleprompt.BootstrapFewShotWithRandomSearch

Constructor​

The constructor initializes the BootstrapFewShotWithRandomSearch class and sets up its attributes. It inherits from the BootstrapFewShot class and introduces additional attributes for the random search process.

class BootstrapFewShotWithRandomSearch(BootstrapFewShot):
    def __init__(self, metric, teacher_settings={}, max_bootstrapped_demos=4, max_labeled_demos=16, max_rounds=1, num_candidate_programs=16, num_threads=6, max_errors=10, stop_at_score=None, metric_threshold=None):
        self.metric = metric
        self.teacher_settings = teacher_settings
        self.max_rounds = max_rounds
        self.num_threads = num_threads
        self.stop_at_score = stop_at_score
        self.metric_threshold = metric_threshold
        self.min_num_samples = 1
        self.max_num_samples = max_bootstrapped_demos
        self.max_errors = max_errors
        self.num_candidate_sets = num_candidate_programs
        self.max_num_traces = 1 + int(max_bootstrapped_demos / 2.0 * self.num_candidate_sets)

        self.max_bootstrapped_demos = self.max_num_traces
        self.max_labeled_demos = max_labeled_demos

        print("Going to sample between", self.min_num_samples, "and", self.max_num_samples, "traces per predictor.")
        print("Going to sample", self.max_num_traces, "traces in total.")
        print("Will attempt to train", self.num_candidate_sets, "candidate sets.")

Parameters:

  • metric (callable, optional): Metric function to evaluate examples during bootstrapping. Defaults to None.
  • teacher_settings (dict, optional): Settings for teacher predictor. Defaults to an empty dictionary.
  • max_bootstrapped_demos (int, optional): Maximum number of bootstrapped demonstrations per predictor. Defaults to 4.
  • max_labeled_demos (int, optional): Maximum number of labeled demonstrations per predictor. Defaults to 16.
  • max_rounds (int, optional): Maximum number of bootstrapping rounds. Defaults to 1.
  • num_candidate_programs (int): Number of candidate programs to generate during random search. Defaults to 16.
  • num_threads (int): Number of threads used for evaluation during random search. Defaults to 6.
  • max_errors (int): Maximum errors permitted during evaluation. Halts run with the latest error message. Defaults to 10. Configure to 1 if no evaluation run error is desired.
  • stop_at_score (float, optional): Score threshold for random search to stop early. Defaults to None.
  • metric_threshold (float, optional): Score threshold for metric to determine successful example. Defaults to None.

Method​

Refer to teleprompt.BootstrapFewShot documentation.

Example​

#Assume defined trainset
#Assume defined RAG class
...

#Define teleprompter and include teacher
teacher = dspy.OpenAI(model='gpt-3.5-turbo', api_key = openai.api_key, api_provider = "openai", model_type = "chat")
teleprompter = BootstrapFewShotWithRandomSearch(teacher_settings=dict({'lm': teacher}))

# Compile!
compiled_rag = teleprompter.compile(student=RAG(), trainset=trainset)