Autoscaling Policy and Preferences

Default Preferences

The following shows the default cluster preferences. This is applied automatically by Solr when no explicit cluster preferences have been set using the Autoscaling API.

[
  {"minimize":"cores"}
]

Minimize Cores; Maximize Free Disk

In this example, we want to minimize the number of Solr cores and in case of a tie, maximize the amount of free disk space on each node.

[
  {"minimize" : "cores"},
  {"maximize" : "freedisk"}
]

Add Precision to Free Disk; Minimize System Load

In this example, we add a precision to the freedisk parameter so that nodes with free disk space within 10GB of each other are considered equal. In such a case, the tie is broken by minimizing sysLoadAvg.

[
  {"minimize" : "cores"},
  {"maximize" : "freedisk", "precision" : 10},
  {"minimize" : "sysLoadAvg"}
]

Rule Types

Policy rules can be either global or per-collection:

Global rules have three parts:

Per-collection rules have five parts:

Node Selector

A node selector is specified using the node nodeset attribute. This is used to filter the set of nodes where this rules needs to be applied

examples

{ "replica" : "<2", "node":"#ANY"}

{  "replica" : "3",  "nodeset":["node-name1","node-name2"]}

{ "nodeset":{"<property-name>":"<property-value>"}}

The property names can be one of: node, host, sysprop., freedisk, ip_, nodeRole, heapUsage, metrics.*.

when using the nodeset attribute, an optional attribute put can be used to specify how to distribute the replicas in that node set.

example: put one replica on each node with a system property zone=east

{ "replica":1, "put" :"on-each-node", "nodeset":{"sysprop.zone":"east"}}

example: put a total of 2 replicas on the set of nodes with property zone=east

{ "replica":2, "put" :"on-each-node", "nodeset":{"sysprop.zone":"east"}}

Rule evaluation is restricted to node(s) matching the value of one of the following attributes: node, port, ip_*, sysprop.*, or diskType. For replica/core count constraints other than #EQUAL, a condition specified in one of the following attributes may instead be used to select nodes: freedisk, host, sysLoadAvg, heapUsage, nodeRole, or metrics.*.

Except for node, the attributes above cause selected nodes to be partitioned into node groups. A node group is referred to as a "bucket". Those attributes usable with the #EQUAL directive may define buckets either via the special function #EACH or an array ["value1", …​] (a subset of all possible values); in both cases, each node is placed in the bucket corresponding to the matching attribute value.

The node attribute always places each selected node into its own bucket, regardless of the attribute value’s form (#ANY, node-name, or ["node1-name", …​]).

Replica and core count constraints, described below, are evaluated against the total number in each bucket.

Core Count Constraint

The cores attribute value can be specified in one of the following forms:

Replica Selector and Rule Evaluation Context

Rule evaluation can be restricted to replicas that meet any combination of conditions specified with the following attributes:

If none of the above attributes is specified, then the rule is evaluated separately for each collection against all types of replicas of all shards.

Specifying #EACH as the shard attribute value causes the rule to be evaluated separately for each shard of each collection.

Replica Count Constraint

The replica attribute value can be specified in one of the following forms:

Specifying Replica and Core Count Constraints

Replica count constraints ("replica":"…​") and core count constraints ("cores":"…​") allow specification of acceptable counts for replicas (cores tied to a collection) and cores (regardless of the collection to which they belong), respectively.

You can specify one of the following as the value of a replica and cores policy rule attribute:

Rule Strictness

This attribute is usable in all rules:

Global Rule Attributes

Per-collection Rule Attributes

The following attributes are usable with per-collection policy rules, in addition to the attributes in the Node Selection Attributes section below:

Node Selection Attributes

One and only one of the following attributes can be specified in addition to the above attributes. See the Node Selector section for more information:

Special Functions

This supports values calculated at the time of execution.

Limit Replica Placement

Do not place more than one replica of the same shard on the same node. The rule is evaluated separately for each shard in each collection. The rule is applied to any node.

{"replica": "<2", "shard": "#EACH", "node": "#ANY"}

Limit Cores per Node

Do not place more than 10 cores in any node. This rule can only be added to the cluster policy because it is a global rule.

{"cores": "<10", "node": "#ANY"}

Place Replicas Based on Port

Place exactly 1 replica of each shard of collection xyz on a node running on port 8983.

{"replica": 1, "shard": "#EACH", "collection": "xyz", "nodeset": {"port": "8983"}}

Place Replicas Based on a System Property

Place all replicas on nodes with system property availability_zone=us-east-1a.

{"replica": "#ALL", "nodeset": {"sysprop.availability_zone": "us-east-1a"}}

Use Percentage

Place a maximum of (roughly) a third of the replicas of each shard in any node. In the following example, the value of replica is computed in real time as a percentage of the replicas of each shard of each collection:

{"replica": "33%", "shard": "#EACH", "node": "#ANY"}

If the number of replicas in a shard is 2, 33% of 2 = 0.66. This means a node may have a maximum of 1 and a minimum of 0 replicas of each shard.

It is possible to get the same effect by hard coding the value of replica as a decimal value:

{"replica": 0.66, "shard": "#EACH", "node": "#ANY"}

or using the range operator:

{"replica": "0-1", "shard": "#EACH", "node": "#ANY"}

Multiple Percentage Rules

Distribute replicas of each shard of each collection across datacenters east and west at a 1:2 ratio:

{"replica": "33%", "shard": "#EACH", "nodeset":{ "sysprop.zone": "east"}}
{"replica": "66%", "shard": "#EACH", "nodeset":{"sysprop.zone": "west"}}

For the above rules to work, all nodes must the started with a system property called "zone"

Distribute Replicas Equally in Each Zone

For each shard of each collection, distribute replicas equally across the east and west zones.

{"replica": "#EQUAL", "shard": "#EACH", "nodeset":[{"sysprop.zone": "east"},{"sysprop.zone":  "west"}]}}

Place Replicas Based on Node Role

Do not place any replica on any node that has the overseer role. Note that the role is added by the addRole collection API. It is not automatically the node which is currently the overseer.

{"replica": 0, "put" :"on-each-node", "nodeset":{ "nodeRole": "overseer"}}

Place Replicas Based on Free Disk

Place all replicas in nodes where freedisk is greater than 500GB.

{"replica": "#ALL", "nodeset":{ "freedisk": ">500"}}

Keep all replicas in nodes where freedisk percentage is greater than 50%.

{"replica": "#ALL", "nodeset":{"freedisk": ">50%"}}

Try to Place Replicas Based on Free Disk

When possible, place all replicas in nodes where freedisk is greater than 500GB. Here we use the strict attribute to signal that this rule is to be honored on a best effort basis.

{"replica": "#ALL", "nodeset":{ "freedisk": ">500"}, "strict": false}

Place All Replicas of Type TLOG on Nodes with SSD Drives

{"replica": "#ALL", "type": "TLOG", "nodeset": {"diskType": "ssd"}}

Place All Replicas of Type PULL on Nodes with Rotational Disk Drives

{"replica": "#ALL", "type": "PULL", "nodeset" : {"diskType": "rotational"}}

Scenario Context

Scenario has a context, which is simply a map of key-value pairs. Before executing each command the context is updated to contain the current values for the following properties:

Command parameters support variable expansion using string values from the current context (non-string values, including numeric, are ignored) and from system properties, with the context values taking precedence if set.

For example, assuming a system property is set 'foo=bar', the following command will load a snapshot from /tmp/bar:

load_snapshot path=/tmp/${foo}

Scenario Commands

The following commands are supported (command names are case insensitive, but parameter names are not):

Example Scenarios

Example scenario testing the behavior of .autoAddReplicas trigger:

# standard comment
// java comment
create_cluster numNodes=2 // inline comment
// load autoscaling config from a JSON string. Notice that the value must be URL-encoded
load_autoscaling json={'cluster-policy'+:+[{'replica'+:+'<3',+'shard'+:+'#EACH',+'collection'+:+'testCollection','node':'#ANY'}]}&defaultWaitFor=10
solr_request /admin/collections?action=CREATE&autoAddReplicas=true&name=testCollection&numShards=2&replicationFactor=2&maxShardsPerNode=2
wait_collection collection=testCollection&shards=2&replicas=2
// prepare a listener for trigger events and the processing state SUCCEEDED
event_listener trigger=.auto_add_replicas&stage=SUCCEEDED
// kill a random node
kill_nodes node=${_random_node_}
// wait for the listener to capture the event
wait_event trigger=.auto_add_replicas&wait=60
// the collection should have the same shape as before
wait_collection collection=testCollection&shards=2&replicas=2
save_snapshot path=${snapshotPath}

Example scenario testing the behavior of indexSize trigger. Notice the use of POST SolrRequest and the use of assert command with an object path:

create_cluster numNodes=100
solr_request /admin/collections?action=CREATE&autoAddReplicas=true&name=testCollection&numShards=2&replicationFactor=2&maxShardsPerNode=2
wait_collection collection=testCollection&shards=2&replicas=2
// example of defining a trigger config
solr_request /admin/autoscaling?httpMethod=POST&stream.body={'set-trigger':{'name':'indexSizeTrigger','event':'indexSize','waitFor':'10s','aboveDocs':1000,'enabled':true,'actions':[{'name':'compute_plan','class':'solr.ComputePlanAction'},{'name':'execute_plan','class':'solr.ExecutePlanAction'}]}}
// prepare an event listener
event_listener trigger=indexSizeTrigger&stage=SUCCEEDED
// add documents
index_docs collection=testCollection&numDocs=3000
// run for 60 sec
run
// wait for a trigger event (as defined in the listener)
wait_event trigger=indexSizeTrigger&wait=60
// even is stored in the context
assert condition=not_null&key=_trigger_event_indexSizeTrigger
assert condition=equals&key=_trigger_event_indexSizeTrigger/eventType&expected=INDEXSIZE
assert condition=equals&key=_trigger_event_indexSizeTrigger/properties/requestedOps[0]/action&expected=SPLITSHARD
wait_collection collection=testCollection&shards=6&withInactive=true&requireLeaders=false&replicas=2

Example scenario where context variables are used for conditional execution of loops. Depending on the value of iterative and justCalc the two loops will execute 0 or more times. Notice also how the scenario picks up a random node to consistently add replicas to it.

create_cluster numNodes=2
solr_request /admin/collections?action=CREATE&autoAddReplicas=true&name=testCollection&numShards=2&replicationFactor=2&maxShardsPerNode=10
wait_collection collection=testCollection&shards=2&replicas=2
ctx_set key=myNode&value=${_random_node_}
solr_request /admin/collections?action=ADDREPLICA&collection=testCollection&shard=shard1&node=${myNode}
solr_request /admin/collections?action=ADDREPLICA&collection=testCollection&shard=shard1&node=${myNode}
loop_start iterations=${iterative}
  calculate_suggestions
  apply_suggestions
  solr_request /admin/collections?action=ADDREPLICA&collection=testCollection&shard=shard1&node=${myNode}
  solr_request /admin/collections?action=ADDREPLICA&collection=testCollection&shard=shard1&node=${myNode}
loop_end
loop_start iterations=${justCalc}
  calculate_suggestions
loop_end
dump redact=true