HyperAI Introduction to Automatic Parameter Adjustment

A lot is needed in the process of building machine learning models Hyperparameters, HyperparametersThe choice greatly affects the final performance of the obtained model. Automatic parameter tuning refers to the method of automatically obtaining the optimal set of hyperparameters through the system.

Automatic parameter tuning currently supports creation through interfaces and commands. If you need to create it through the command line. Please proceed first install bayes Command line tools, And understand its basic usage method.

To use auto tuning, the following tasks need to be done:

1. Complete the configuration of automatic parameter tuning. It mainly consists of two parts:
   • Determine what needs to be automatically adjusted Hyperparameters, What to specify for the system based on the configuration document for automatic parameter tuning Hyperparameters Need to adjust and the range of adjustment
   • Determine what needs to be obtained Key indicators, And during model training HyperAItool Submit to HyperAI In the system
2. Modify existing code. Support for automatic parameter tuning system generated Hyperparameters The Reading

Quick try

HyperAI-hypertuning-example It is a sample code repository. Currently, the warehouse contains TensorFlow and PyTorch Example of Version. Please separate them separately /tensorflow and /pytorch Catalog View. Each directory contains completed HyperAI.yaml And corresponding supporting code. Follow the steps to run an example of automatic parameter tuning.

Modify your own code to support automatic parameter tuning

Obtain the parameters generated by the automatic parameter tuning service

The task of automatic parameter tuning can obtain its provided parameters in two ways:

1. read HyperAI_params.json: As mentioned above parameter_specs It will generate four parameters. So in the working directory of the container, there will be files containing these contents HyperAI_params.json:

   HyperAI_params.json

   {
      "regularization": 6.933098216541059,
      "latent_factors": 29,
      "unobs_weight": 3.439490399012204,
      "feature_wt_factor": 102.9461653166788
   }

   Before executing the code, simply read the parameters in the file and use them

2. Read command line parameters: The parameters of tasks created through automatic parameter tuning will be added to the execution command in the following form:

   python main.py \
       --regularization=6.933 \
       --latent_factors=29 \
       --unobs_weight=3.439 \
       --feature_wt_factor=102.9

   use argparse Can parse and use these parameters.

Report key indicators

stay System indicators and custom indicators The chapter introduces HyperAItool There are tools available log_metric The method can report custom indicators. Automatic parameter tuning also uses this tool to achieve indicator reporting. At the end of the program, reporting can be achieved through the following code:

import HyperAItool

HyperAItool.log_metric("precision", 0.809)

Please note that the specific indicator names reported here should be consistent with HyperAI.yaml Configured in the middle hyperparameter_metric agreement.

Create automatic parameter tuning

Create through interface

After logging into the console. You can access it from the sidebar "Automatic parameter tuning" Create. Or from the top right corner of the list "Create automatic parameter tuning" Create.

At present, we support two ways to configure automatic parameter tuning parameters. Visual Editor and YAML editor. Editing in any editor will synchronize updates. That is to say, when the user updates the parameters in the visual editor. Switch to YAML After the editor. Configuration will be updated synchronously. The reverse is also the same.

Create through command-line tools

be careful

When creating auto parameters through command-line tools. We need to bayes Have some understanding of command-line tools. You can refer to bayes Introduction to Command Line Tools, At the same time, it is also necessary to understand HyperAI configuration file.

Get ready [Automatic parameter tuning configuration file](#Automatic Parameter Adjustment Configuration Instructions)after. By command bayes gear run hypertuning Create automatic parameter tuning task:

View automatic parameter tuning

As shown above. The automatic parameter tuning page will pass through "Chart" "Parallel coordinates" Display the current execution status in various ways. Users can select the best execution as the result or update the automatic parameter range based on the current result to continue the new automatic parameter tuning task.among "Parallel coordinates" It is a very suitable view for filtering the results of automatic parameter tuning. By defining the parameter range on the interface, the displayed parameter combinations can be controlled.

Automatic Parameter Adjustment Configuration Instructions

Through the command line bayes gear init <container-name> (Related documents)Bind the current directory and container. After executing this command, files will appear in the directory HyperAI.yaml.stay HyperAI.yaml Specify according to the configuration requirements Hyperparameters And its scope.

One HyperAI.yaml The document is divided into two parts:

1. Basic configuration, stay HyperAI configuration file This section has corresponding introductions
2. Automatic parameter tuning configuration. Only in hyper_tuning Configuration of automatic parameter tuning under parameters

Generated by default HyperAI.yaml Below are examples of the automatic parameter tuning configuration section:

HyperAI.yaml

hyper_tuning:
  max_job_count: 3
  hyperparameter_metric: precision
  side_metrics: []
  goal: MINIMIZE
  algorithm: Bayesian
  parameter_specs:
  - name: regularization
    type: DOUBLE
    min_value: 0.001
    max_value: 10.0
    scale_type: UNIT_LOG_SCALE
  - name: latent_factors
    type: INTEGER
    min_value: 5
    max_value: 50
    scale_type: UNIT_LINEAR_SCALE
  - name: unobs_weight
    type: DOUBLE
    min_value: 0.001
    max_value: 5.0
    scale_type: UNIT_LOG_SCALE
  - name: feature_wt_factor
    type: DOUBLE
    min_value: 1
    max_value: 200
    scale_type: UNIT_LOG_SCALE
  - name: level
    type: DISCRETE
    discrete_values: [1, 2, 3, 4]
  - name: category
    type: CATEGORICAL
    categorical_values: ["A", "B", "C"]

hyper_tuning The parameters below are the configurations required for automatic parameter tuning:

1. max_job_count

The number of attempts at automatic parameter tuning once. Maximum support 100 second.

2. parallel_count

Number of parallel execution tasks. This value is limited by the maximum number of parallel instances for a single resource type of the user (see [Parallel limit of computing power](/docs/gear/limitation/#Parallel limit of computing power)), The more parallel there are, the faster the automatic parameter tuning task will be executed.

3. hyperparameter_metric

It is the key indicators that need to be obtained. In the following text [Report key indicators](/docs/hypertuning/#Report key indicators) Part of it will be used.

4. goal

The direction of obtaining the optimal solution for the key indicators needed (MAXIMIZE or MINIMIZE).

5. parameter_specs

Hyperparameters The Regulations. It can be set immediately Hyperparameters Type and its scope, among Hyperparameters The types can be the following four:

type	Numerical Range	data type	describe
DOUBLE	min_value - max_value	float
INTEGER	min_value - max_value	integer
CATEGORICAL		Enumeration type	character string
DISCRETE		Discrete type	Sequential numerical values

among INTEGER and DOUBLE The type can also be set to obtain the distribution type of parameters (scale_type):

Distribution type	describe
UNIT_LINEAR_SCALE	Linear distribution. Refers to data in min_value and max_value It follows uniform distribution sampling
UNIT_LOG_SCALE	LOG distribution, finger log(value) stay min_value and max_value Following a uniform distribution between them. That is to say, during the sampling process. The smaller the value, the higher the sampling density, in use UNIT_LOG_SCALE When min_value Must be greater than 0

6. algorithm

Algorithms that can be used for automatic parameter tuning, support Bayesian Random or Grid:

algorithm	describe
Grid	For only DISCRETE as well as CATEGORICAL The scenario of type parameters can be achieved through GridSearch Traverse all combinations of parameters
Random	in the light of INTEGER as well as DOUBLE type. Based on the supported distribution types, stay min_value and max_value Randomly select numerical values between them, about DISCRETE and CATEGORICAL type. Their behavior and Grid Similar approach
Bayesian	Consider the previous when generating parameters each time "parameter" - "Target variable" The result. Provide parameters through the updated distribution function to expect better results. Its algorithm can be referred to This article

7. parameters

be careful

This parameter does not appear in hyper_tuning Configure below. But it appears at the root level, stay config to configure Partially introduced. Creating "Automatic parameter tuning task" Providing this field will still have an effect. However, the priority of the parameter configuration provided here will be lower than the configuration generated during the automatic parameter tuning process. That is to say, if there are and parameter_specs The parameter under has a duplicate name. So this parameter will be parameter_specs Covered by the parameters below.

Give me an example. We have configured the following HyperAI.yaml:

HyperAI.yaml

...Omit other parts...
parameters:
  filter_size: 5
hyper_tuning:
  parameter_specs:
    - name: filter_size
      type: DISCRETE
      discrete_values: [1, 2, 3, 4]
...Omit other parts...

So every time the parameter combination generated by automatic parameter tuning, there will definitely be filter_size This field. So the generated value will overwrite it parameters Below filter_size The value of a field.

8. side_metrics

As mentioned earlier hyperparameter_metric It is a key indicator used for automatic parameter tuning. In addition, multiple settings can be made side_metrics As an auxiliary indicator of the final indicator. For example, relative to accuracy f1_score Indicators or during the training process loss Can be added as side_metrics The most reference indicator. This indicator Has no impact on the automatic parameter tuning process. It will only be displayed in the visualization of the indicators. As a reference for the quality of automatic parameter tuning results.

As shown in the above figure. As side_metrics Reference indicators loss It will also be displayed. But be careful. It should also be used during the coding process HyperAItool.log_metric Method: Record the indicators. Otherwise, it cannot be displayed in the visualization. You can refer to HyperAI hypertuning in pytorch Middle right loss Handling of indicators.

Explanation of the state of automatic parameter tuning

During the execution of automatic parameter tuning tasks, the following states may occur:, Here are some explanations for different states:

1. Running: The execution runs normally under automatic parameter tuning
2. Cancelled: The automatic parameter tuning task was closed due to user initiated termination
3. No corresponding indicators: Refers to the failure to provide during execution under automatic parameter tuning Indicator reporting The code. The automatic parameter tuning system terminated the automatic parameter tuning process after discovering this issue
4. Execution error: Refers to the execution error or termination due to certain reasons under automatic parameter tuning (Perhaps a single execution has been closed. For example, being shut down by the system due to unpaid fees or other reasons)
5. System error: The automatic parameter tuning system has encountered some problems that have caused the automatic parameter tuning to be suspended. In this case, you can contact the system administrator to report the issue
6. Search space exhausted: There are two situations where this result may occur. One is that the automatic parameter tuning system considers the user provided Hyperparameters The parameter combinations produced within the scope have been exhausted, and the parameters generated next are duplicated; Another reason is that the system believes that there is currently no way to produce better results. Proactively terminated the search