Spherical linear kernels for high dimensional BO

Spherical Linear Kernel is useful for optimizing high-dimensional problems.

from bofire.benchmarks.svm import SVM
from bofire.data_models.strategies.api import SoboStrategy
from bofire.data_models.kernels.api import SphericalLinearKernel
from bofire.data_models.surrogates.api import SingleTaskGPSurrogate, BotorchSurrogates
import bofire.strategies.api as strategies

/opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`.
  warnings.warn(

We use the SVM benchmark.

# problem setup for spherical linear kernels
benchmark = SVM()
candidates = benchmark._domain.inputs.sample(benchmark.dim+1, seed=benchmark.seed)
experiments = candidates.copy()
result = benchmark._f(experiments)
# Add empty columns 'y' and 'valid_y' to experiments DataFrame
experiments["y"], experiments["valid_y"] = result["y"], result["valid_y"]
sobo_strategy_data_model = SoboStrategy(
    domain=benchmark._domain,
    seed=benchmark.seed,
    surrogate_specs=BotorchSurrogates(
        surrogates=[
            SingleTaskGPSurrogate(
                inputs=benchmark._domain.inputs,
                outputs=benchmark._domain.outputs,
                kernel=SphericalLinearKernel(),
            )
        ]
    ),
)
strategy = strategies.map(sobo_strategy_data_model)

Downloading SVM data...
Download complete.

Running the optimization loop

strategy.tell(experiments, replace=True)
num_steps = 3 # set the number of steps here (the original paper uses 1000 steps)
for step_number in range(num_steps):
    print(f"Step {step_number+1}/{num_steps}")
    new_candidates = strategy.ask(candidate_count=1)
    new_experiments = new_candidates.copy()
    result = benchmark._f(new_candidates)
    new_experiments["y"], new_experiments["valid_y"] = result["y"], result["valid_y"]
    print(f"New experiment:\n{new_experiments}")
    strategy.tell(experiments=new_experiments)
# save all the experiments
all_experiments = strategy.experiments

/opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/bofire/utils/torch_tools.py:51: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /__w/pytorch/pytorch/torch/csrc/utils/tensor_numpy.cpp:213.)
  return torch.from_numpy(np.ascontiguousarray(df.to_numpy())).to(**tkwargs)

Step 1/3

/opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/linear_operator/utils/cholesky.py:41: NumericalWarning: A not p.d., added jitter of 1.0e-08 to the diagonal
  warnings.warn(

New experiment:
        x_1      x_10     x_100     x_101     x_102     x_103    x_104  \
0  0.522446  0.524803  0.317933  0.116095  0.160888  0.199463  0.97305   

     x_105     x_106     x_107  ...      x_95      x_96      x_97      x_98  \
0  0.84544  0.714217  0.068977  ...  0.909508  0.933229  0.911664  0.031016   

       x_99    y_pred     y_sd     y_des         y  valid_y  
0  0.307294  0.236782  0.00201 -0.236782  0.231838        1  

[1 rows x 393 columns]
Step 2/3

/opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/linear_operator/utils/cholesky.py:41: NumericalWarning: A not p.d., added jitter of 1.0e-08 to the diagonal
  warnings.warn(
/opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/botorch/optim/optimize.py:844: RuntimeWarning: Optimization failed in `gen_candidates_scipy` with the following warning(s):
[NumericalWarning('A not p.d., added jitter of 1.0e-08 to the diagonal'), OptimizationWarning('Optimization failed within `scipy.optimize.minimize` with status 2 and message ABNORMAL: .'), NumericalWarning('A not p.d., added jitter of 1.0e-08 to the diagonal')]
Trying again with a new set of initial conditions.
  return _optimize_acqf_batch(opt_inputs=opt_inputs)

New experiment:
        x_1      x_10     x_100     x_101     x_102     x_103     x_104  \
0  0.185648  0.279444  0.733382  0.899774  0.419927  0.285609  0.159954   

      x_105    x_106     x_107  ...      x_95      x_96      x_97      x_98  \
0  0.726991  0.04226  0.245952  ...  0.643117  0.478726  0.216508  0.039968   

       x_99    y_pred      y_sd     y_des         y  valid_y  
0  0.693343  0.236664  0.002022 -0.236664  0.233895        1  

[1 rows x 393 columns]
Step 3/3

/opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/linear_operator/utils/cholesky.py:41: NumericalWarning: A not p.d., added jitter of 1.0e-08 to the diagonal
  warnings.warn(

New experiment:
        x_1      x_10     x_100     x_101     x_102     x_103     x_104  \
0  0.930648  0.820992  0.230556  0.895862  0.565248  0.457259  0.484121   

      x_105    x_106     x_107  ...     x_95      x_96      x_97      x_98  \
0  0.484007  0.48987  0.818264  ...  0.52289  0.091771  0.295479  0.642068   

       x_99    y_pred      y_sd     y_des         y  valid_y  
0  0.614281  0.236586  0.002024 -0.236586  0.231279        1  

[1 rows x 393 columns]

One can use the results obtained in all_experiments to get the evolution of the optimum with respect to the iterations.