Bitalos

# Service Scaling

Bitalostored service scaling:

  • Vertical scaling: increase/decrease the number of nodes in a shard
  • Horizontal scaling: increase/decrease the number of shards

Bitalosproxy service scaling:

  • Increase/decrease the number of proxy nodes

# Bitalostored Scaling

# Vertical Scaling

If the read requests of single node in a shard is too high, which causes high load (cpu or I/O), it is advised to add node within shard.

Scaling up(add node), refer to: Add Node

Scaling down(remove node), refer to: Remove Node

# Horizontal Scaling

The data volume of a single node in a shard is too large, or the write request of a single shard is too high, resulting in high pressure on the CPU, memory, or I/O; consider creating new shard(scale out).

Horizontal-scaling for shard is completed by migrate slot and related data between different shards. The number of slots for Bitalostored service is 1024, and range of slotId is 0-1023.

Example of scale out (add shard):

Currently there is only one shard (shard-1) in Bitalostored service. Prepare to scale out to two shards. The operation steps: 1, first create new shard(shard-2); 2, migrate slots(512-1023) from shard-1 to shard-2.

Example of scale in (remove shard):

Currently there are two shards: shard-1 (include slots 0-511), shard-2 (include slots 512-1023). Prepare to scale in to one shard. The operation steps: 1, migrate slots(512-1023) from shard-2 to shard-1; 2, remove all nodes of shard-1 and shard-1 from Bitalosdashboard.

# Slot Migration Command

# migrateslots
> migrateslots [tohost] [toport] [slotId]

Description: migrateslots will migrate slotId to target node(tohost:toport)

tohost is ip of target node, toport is port of target node.

Example:

A certain slot in shard-1 is migrated to shard-2. This command needs to be executed in the master of shard-1, and it returns immediately. Then the migration task will be started in the background.

# migratestatus
> migratestatus [slotId]

Description: migratestatus will show migration process status about slotId

Returns the migration status of the current slotId

Command output:

unixtime, start time of migration
from, migration source node
to, migration target node
costs, total time spent of migration
status, migration status (1=in progress, 2=successful, 3=failed)
total, the number of migrated keys
fails, the number of keys that failed to be migrated
# migrateend
> migrateend [slotId]

Description: migrateend will notify source node that migration task is finished

migrateend command should be executed by master of migration source node, which can reset migration information.

# Slot Migration Operation

The migration operation page is as follows.

migrate

Slot migration supports two methods:

  • MigrateRange：Migrate slots within a certain range to the target shard
  • MigrateSome：Migrate numbers of slots of some shard to the target shard

MigrateRange parameters

From: start point of slot
To: end point of slot
Group: target shard ID
Data migration: whether to migrate data. In most scenarios, data migration is required, and the migration time is positively related to the number of keys. Specifically, if service needs to scale out emergently, and the storage data can be discarded (like full cache), choose "No".

MigrateSome parameters

NumberOfSlots: Number of slots to be migrated
FromGroup: source shard ID
ToGroup: target shard ID

The above two migration operations will generate multiple migration tasks according to slot.

Single slot migration steps (Suppose: migrate from shard-1 to shard-2):

  • Bitalosdashboard sends migration tasks to master of shard-1 ("migrateslots" command), target is master of shard-2
  • Master of shard-1 starts migrating the corresponding slot data to master of shard-2
  • Bitalosdashboard periodically checks the migration status ("migratestatus" command) until migration status becomes successful or failed.
  • If status is failed, Bitalosdashboard resends migration task, migration will be executed until finished (no more migration slot data in shard-1)
  • If status is successful, Bitalosdashboard will modify the route of the current slot to shard 2, notify all Bitalosproxy nodes to modify the route, and finally send "migrateend" command to the master of shard-1.

# Slot Migration Details

# Request During Migration

For example, slot-10 is migrated from shard-1 to shard-2.

  • Requests related to slot-10 will still be routed to shard-1

  • Check key exist or not when the node in shard-1 processes request related to slot-10

    • If existed, request will be process by this node
    • If no existed, this node will redirect this request to master of shard-2, and return response data
# Parallel Migration

Bitalosdashboard schedules migration task grouped by shard.

  • Execute migration task sequentially in one shard
  • Execute migration tasks parallelly for multiple shards
# Migration Interrupted
  • The current migration slot cannot be interrupted
  • Sequential migration of multiple slots will interrupt the migration of the next slot

# Bitalosproxy Scaling

Bitalosproxy nodes are stateless, adding and removing nodes requires no additional operations.

Add Bitalosproxy node, refer to: Add proxy node

Reduce Bitalosproxy node, refer to: Remove proxy node

← Management backend manual →

ZUOYEBANG