Skip to main content

Automatic Hyperparameter Iteration

Automatic hyperparameter iteration is an advanced feature within Queryloop designed to optimize configurations by evaluating various combinations of hyperparameters. This functionality enables Queryloop to identify the most effective setups by iterating through parameters for both retrieval and generation models. It minimizes the need for manual adjustments, yielding fine-tuned results tailored to specific data and applications.

Through these iterations, Queryloop ensures that each component—whether for information retrieval or language generation—is optimized for accuracy, efficiency, and performance. Hyperparameter tuning within Queryloop is categorized into two primary areas: Retrieval Optimization and Generation Optimization, each following customized workflows based on the application's needs.

Automatic Hyperparameter Iteration for Retrieval​

In retrieval optimization, Queryloop systematically evaluates various hyperparameter configurations to ensure precise and relevant information retrieval. Below are the key parameters involved in retrieval hyperparameter iteration:

  1. Chunk Size

    • Influences the granularity of text division for retrieval tasks. Queryloop evaluates different chunk sizes (e.g., tiny, small, medium) to find the optimal size that balances response accuracy with processing speed.

  2. Re-Rank Method

    • Supports algorithms like Maximal Marginal Relevance (MMR) or no re-ranking, where relevant results are reordered to improve quality based on similarity to the query. Each method is tested to identify the best balance between relevance and diversity.

  3. Metric Type

    • Iterates over similarity metrics (Cosine, Dot Product, etc.) to evaluate how closely responses align with the query. This ensures the chosen metric offers the highest retrieval relevance.

  4. Query Preprocessing

    • Adjusts text input (e.g., normalization, stemming) before retrieval. Different preprocessing techniques are systematically applied to enhance accuracy.

  5. Top-K Results

    • Defines the number of top results returned for each query. Queryloop tests various K-values (e.g., 1, 5, 10) to identify the optimal number of results that balances comprehensiveness and specificity.

  6. Embedding Model

    • Impacts the quality of information retrieval, as embeddings transform queries into vector representations. Queryloop iterates over selected embedding models (e.g., text-embedding-ada-002 and text-embedding-3-large) to determine the model that yields the highest accuracy.

This systematic iteration across retrieval parameters is especially beneficial for applications requiring precise document retrieval or content summarization. Users can achieve superior query results without manual tuning.

Automatic Hyperparameter Iteration for Generation​

Generation optimization within Queryloop is divided into two workflows: Foundational Language Models (LLMs) and Fine-Tuned Models. Each workflow utilizes automatic hyperparameter iteration to enhance the quality and contextual relevance of generated responses.

Foundational Language Model Iterations​

For foundational LLMs, Queryloop leverages models from various providers (OpenAI, Google, Mixtral, Meta, and Anthropic) and iterates across the following parameters:

  1. Model Selection

    • Evaluates a diverse array of models for suitability to the task, selecting the one that best balances processing time with response quality.

  2. Prompt Combinations

    • Tests different prompt formulations to find those that best fulfill task requirements, enhancing user experience and response precision.

  3. Output Token Limits

    • Dictates the length of the generated text. Various token limits are tested to identify the optimal setting for complete responses.

  4. Temperature

    • Controls the randomness of the output. Different temperature settings are iterated to balance creativity and relevance.

Fine-Tuning Model Iterations​

For applications requiring customized models, Queryloop supports fine-tuning iterations, testing combinations of parameters to yield accurate and contextually relevant outputs. Key fine-tuning parameters include:

  1. Batch Size

    • Affects processing speed and memory efficiency. Different batch sizes are tested to identify the most suitable configuration.

  2. Epoch

    • Defines the number of training cycles over the dataset. Various epoch counts are iterated to find the optimal balance between training depth and overfitting risk.

  3. Learning Rate

    • Influences how quickly the model adjusts during training. Different learning rates are tested for effective adjustments without instability.

  4. QLoRA Parameters (Alpha/Rank Ratio, Rank)

    • For models fine-tuned using QLoRA, specific parameters like Alpha/Rank Ratio and Rank are iterated to balance knowledge retention with the introduction of new patterns.

Summary​

By automating hyperparameter iteration across retrieval and generation configurations, Queryloop optimizes each component for superior performance. In retrieval, it fine-tunes parameters like chunk size and embedding models, while in generation, it adjusts model selection, prompt formulation, and tuning parameters like temperature and batch size. This robust approach reduces manual experimentation efforts, enabling efficient application deployment that meets high standards of accuracy and relevance.