Transcript slides

Schreiber, Yevgeny.
Value-Ordering Heuristics: Search
Performance vs. Solution Diversity.
In: D. Cohen (Ed.) CP 2010, LNCS 6308, pp. 429-444.
Springer- Heidelberg (2010).
~ Presented by:
Michael Gould
CS 275
December 7, 2010
Introduction



Problem: Given a CSP, generate a large
number of diverse solutions as fast as
possible.
Trade-off: Solution search performance
(time) vs. the diversity of the generated
solutions.
Tool: Use value-ordering heuristics to help
solve problem.
General Solution Search Method

Input: A CSP (X, D, C)

a) Repeatedly select an unassigned variable x.



b) Try to assign to x one of the remaining values
in its domain.
c) Propagate every constraint c that involves x by
removing the conflicting values in the domains of
other unassigned variables.
d) Backtrack if necessary.
Value-Ordering Heuristics


A value-ordering heuristic determines which
value is selected by the backtracking
search algorithm for a given unassigned
variable.
Ex. Survivors-first heuristics use simple
statistics accumulated during the search to
select the value that has been involved in
the least number of conflicts.
Solution Diversity


Solution Diversity: Requirement that the
multiple solutions we find be as “different”
from each other as possible.
Solution Diversity is different from solution
distribution.
Consider a solution space composed of a small subset S1
of solutions where each variable is assigned a different value
and a much larger set of solutions S2 so that only a single
variable is assigned a different value in each solution.
Solution Diversity (continued)



There is a trade-off between the search
performance and the solution diversity.
The MAXDIVERSEkSET problem is to
compute k maximally diverse solutions of a
given CSP.
One measure of the distance between a
pair of solutions is Hamming Distance:
Given solution s = <s1,...,sn> and solution s' = <s1',...,sn'> we
define Hi(s,s') = 1 if si ≠ si' and 0 otherwise for 1≤i≤n. The
Hamming distance is the sum of all Hi values.
Automatic Test Generation Problem (ATGP)





A real-world example of a CSP where solution
diversity is important.
The problem is to generate automatically a valid test
for a given hardware specification which is a
sequence of a large number of instructions.
These instructions must be diverse in order to trigger
as many as possible different hardware events.
Time is important: thousands of tests have to be
generated even for a small subset of a modern
hardware specification
We consider the overall quality of the whole testbase: size and diversity of tests.
The RANDOM Heuristic


Simply select a uniformly random value
from the domain of a variable.
Often achieves relatively high solution
diversity and performs relatively fast. Reason:
in these problems there is a relatively large number of
values in every domain whose selection does not lead to
a conflict.

Procedure itself of randomly selecting a
value is very fast.
(1) The LeastFails Heuristic
(2) BestSuccessRatio Heuristic
(3) & (4) Probabilistic Versions
Heuristics (5) and (6)
Initial Ordering of Values




No information is available at the beginning of the search for a
first solution. Thus, all the values in each domain are initially
unordered.
The heuristics can only order values that have already been
attempted to be assigned to variables in the past. It must be
decided what probability should be used for the values whose
order cannot be determined.
We define the conservativeness of a heuristic. The more
conservative a heuristic is, the lower the probability to select an
unordered value. We can define the conservativeness C(H) of
a heuristic H separately for each variable whose domain
contains at least one unordered value.
Low C(H) value → initially random behavior but prevents
situations where many unordered values are never selected.
Heuristic Parameters



α : Controls the conservativeness of a heuristic.
Represents an “initial score” of an unordered
value u.
β : Used in the probabilistic heuristics to control
the aggressiveness of the heuristic (the
distribution of probabilities).
γ : A tie-range parameter used to smooth
differences between “sufficiently close” values
for the heuristics LeastFails and
BestSuccessRatio.
Experiments

(1) Randomly generated problems.
<a,b, c, d>: a = number of variables, b = domain size of each variable, c
= number of binary constraints, d = number of incompatible value
pairs in each constraint.
Tested <50, 10, 225, 35> and <50, 10, 225, 37> with 30 problems in
each set. Looked for 30 solutions for each problem, and solved using
the variable-ordering that picked the variable with the minimal
domain.

(2) ATGP problems.
Thousands of variables and constraints that model Intel 64 and
IA-32 processor architecture.
<n, m>: n = # of problems in set, m = avg. # of instructions
required to generate for each problem.
Tested <15,21> and <26, 7>.
Results

Results summarized in tables. Table entries:
<A, B, C, D, E>
A: the acronym of the heuristic.
B: the value of α used as the “initial score” of an unordered
value.
C: the value of either β or γ.
D: ratio of the avg. time for the heuristic to find a single
solution over time required for the random heuristic.
E: ratio of avg. Hamming distance between solutions found
by the heuristic over distance achieved by random.
Analysis





Entries in table sorted by D/E. Lower the ratio → better
the performance/quality. Heuristic considered better than
RANDOM if D <= E.
Hard to achieve better solution diversity than RANDOM.
Only a few entries have E > 1.
Many heuristic configurations run much faster than
RANDOM. (e.g. LeastFails can run 10 to 20 times faster).
Very high speedup → significant loss of solution diversity.
Some good heuristic configurations that achieve
significant speedup and hardly any loss of solution
diversity at top of tables.
Analysis (continued)



LeastFails and BestSuccessRatio: Usually achieve very high
speed-up, but often accompanied by a significant loss of
solution diversity. In randomly generated problems, loss of
solution diversity was lower than speed-up gain.
ProbMostFails and ProbWorstSuccessRatio: Usually do not
achieve a better solution diversity than Random, but can be
much slower.
ProbLeastFails and ProbBestSuccessRation: Often achieve
a moderate speed-up without sacrificing much solution
diversity.
Questions





Do the experimental results hold for larger
problems (hundreds of variables, larger domain
sizes)?
Do the experimental results hold for other realworld CSP problems?
Do other variable orderings achieve better
performance?
Can we combine value-ordering heuristics with
inference methods such as search look-ahead or
arc-consistency or is the overhead too high?
Is there a consensus on which value-ordering
heuristic is the best?