Spherical linear kernels for high dimensional BO

Spherical Linear Kernel is useful for optimizing high-dimensional problems.

from bofire.benchmarks.svm import SVM
from bofire.data_models.strategies.api import SoboStrategy
from bofire.data_models.kernels.api import SphericalLinearKernel
from bofire.data_models.surrogates.api import SingleTaskGPSurrogate, BotorchSurrogates
import bofire.strategies.api as strategies

We use the SVM benchmark.

# problem setup for spherical linear kernels
benchmark = SVM()
candidates = benchmark._domain.inputs.sample(benchmark.dim+1, seed=benchmark.seed)
experiments = candidates.copy()
result = benchmark._f(experiments)
# Add empty columns 'y' and 'valid_y' to experiments DataFrame
experiments["y"], experiments["valid_y"] = result["y"], result["valid_y"]
sobo_strategy_data_model = SoboStrategy(
    domain=benchmark._domain,
    seed=benchmark.seed,
    surrogate_specs=BotorchSurrogates(
        surrogates=[
            SingleTaskGPSurrogate(
                inputs=benchmark._domain.inputs,
                outputs=benchmark._domain.outputs,
                kernel=SphericalLinearKernel(),
            )
        ]
    ),
)
strategy = strategies.map(sobo_strategy_data_model)

Downloading SVM data...
Download complete.

Running the optimization loop

strategy.tell(experiments, replace=True)
num_steps = 3 # set the number of steps here (the original paper uses 1000 steps)
for step_number in range(num_steps):
    print(f"Step {step_number+1}/{num_steps}")
    new_candidates = strategy.ask(candidate_count=1)
    new_experiments = new_candidates.copy()
    result = benchmark._f(new_candidates)
    new_experiments["y"], new_experiments["valid_y"] = result["y"], result["valid_y"]
    print(f"New experiment:\n{new_experiments}")
    strategy.tell(experiments=new_experiments)
# save all the experiments
all_experiments = strategy.experiments

/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/bofire/surrogates/botorch.py:181: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:213.)
  torch.from_numpy(Y.values).to(**tkwargs),

Step 1/3

/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/linear_operator/utils/cholesky.py:41: NumericalWarning: A not p.d., added jitter of 1.0e-08 to the diagonal
  warnings.warn(
/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/botorch/optim/optimize.py:796: RuntimeWarning: Optimization failed in `gen_candidates_scipy` with the following warning(s):
[NumericalWarning('A not p.d., added jitter of 1.0e-08 to the diagonal'), OptimizationWarning('Optimization failed within `scipy.optimize.minimize` with status 2 and message ABNORMAL: .'), OptimizationWarning('Optimization failed within `scipy.optimize.minimize` with status 2 and message ABNORMAL: .'), NumericalWarning('A not p.d., added jitter of 1.0e-08 to the diagonal')]
Trying again with a new set of initial conditions.
  return _optimize_acqf_batch(opt_inputs=opt_inputs)

New experiment:
        x_1      x_10     x_100    x_101     x_102     x_103     x_104  \
0  0.685903  0.425335  0.386223  0.64829  0.786647  0.032225  0.457765   

      x_105     x_106     x_107  ...      x_95      x_96      x_97      x_98  \
0  0.734912  0.097881  0.469358  ...  0.244262  0.186416  0.277618  0.667862   

       x_99    y_pred     y_sd     y_des         y  valid_y  
0  0.333728  0.236782  0.00201 -0.236782  0.227778        1  

[1 rows x 393 columns]
Step 2/3
New experiment:
        x_1      x_10    x_100  x_101  x_102     x_103     x_104  x_105  \
0  0.930712  0.822686  0.26518    1.0    0.0  0.189294  0.187052    1.0   

      x_106     x_107  ...     x_95  x_96      x_97      x_98      x_99  \
0  0.384634  0.557943  ...  0.72041   1.0  0.337819  0.288115  0.834431   

     y_pred      y_sd     y_des         y  valid_y  
0  0.201733  0.001903 -0.201733  0.235216        1  

[1 rows x 393 columns]
Step 3/3

/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/linear_operator/utils/cholesky.py:41: NumericalWarning: A not p.d., added jitter of 1.0e-08 to the diagonal
  warnings.warn(

New experiment:
        x_1  x_10     x_100  x_101     x_102     x_103  x_104     x_105  \
0  0.845768   1.0  0.794827    1.0  0.336173  0.198046    0.0  0.709444   

      x_106  x_107  ...      x_95      x_96      x_97  x_98      x_99  \
0  0.726032    1.0  ...  0.200613  0.642428  0.341984   1.0  0.633679   

     y_pred      y_sd     y_des         y  valid_y  
0  0.214495  0.002069 -0.214495  0.234204        1  

[1 rows x 393 columns]

One can use the results obtained in all_experiments to get the evolution of the optimum with respect to the iterations.