milvus-logo
LFAI
Home
  • Administration Guide

dataNode-related Configurations

dataNode.dataSync.flowGraph.maxQueueLength

Description Default Value
Maximum length of task queue in flowgraph 16

dataNode.dataSync.flowGraph.maxParallelism

Description Default Value
Maximum number of tasks executed in parallel in the flowgraph 1024

dataNode.dataSync.maxParallelSyncMgrTasks

Description Default Value
The max concurrent sync task number of datanode sync mgr globally 256

dataNode.dataSync.skipMode.enable

Description Default Value
Support skip some timetick message to reduce CPU usage true

dataNode.dataSync.skipMode.skipNum

Description Default Value
Consume one for every n records skipped 4

dataNode.dataSync.skipMode.coldTime

Description Default Value
Turn on skip mode after there are only timetick msg for x seconds 60

dataNode.segment.insertBufSize

Description Default Value
  • The maximum size of each binlog file in a segment buffered in memory. Binlog files whose size exceeds this value are then flushed to MinIO or S3 service.
  • Unit: Byte
  • Setting this parameter too small causes the system to store a small amount of data too frequently. Setting it too large increases the system's demand for memory.
    		•  16777216

    dataNode.segment.deleteBufBytes

    Description Default Value
    Max buffer size in bytes to flush del for a single channel, default as 16MB 16777216

    dataNode.segment.syncPeriod

    Description Default Value
    The period to sync segments if buffer is not empty. 600

    dataNode.memory.forceSyncEnable

    Description Default Value
    Set true to force sync if memory usage is too high true

    dataNode.memory.forceSyncSegmentNum

    Description Default Value
    number of segments to sync, segments with top largest buffer will be synced. 1

    dataNode.memory.checkInterval

    Description Default Value
    the interal to check datanode memory usage, in milliseconds 3000

    dataNode.memory.forceSyncWatermark

    Description Default Value
    memory watermark for standalone, upon reaching this watermark, segments will be synced. 0.5

    dataNode.channel.workPoolSize

    Description Default Value
  • specify the size of global work pool of all channels
  • if this parameter <= 0, will set it as the maximum number of CPUs that can be executing
  • suggest to set it bigger on large collection numbers to avoid blocking
    		•  -1

    dataNode.channel.updateChannelCheckpointMaxParallel

    Description Default Value
  • specify the size of global work pool for channel checkpoint updating
  • if this parameter <= 0, will set it as 10
    		•  10

    dataNode.channel.updateChannelCheckpointInterval

    Description Default Value
    the interval duration(in seconds) for datanode to update channel checkpoint of each channel 60

    dataNode.channel.updateChannelCheckpointRPCTimeout

    Description Default Value
    timeout in seconds for UpdateChannelCheckpoint RPC call 20

    dataNode.channel.maxChannelCheckpointsPerPRC

    Description Default Value
    The maximum number of channel checkpoints per UpdateChannelCheckpoint RPC. 128

    dataNode.channel.channelCheckpointUpdateTickInSeconds

    Description Default Value
    The frequency, in seconds, at which the channel checkpoint updater executes updates. 10

    dataNode.import.maxConcurrentTaskNum

    Description Default Value
    The maximum number of import/pre-import tasks allowed to run concurrently on a datanode. 16

    dataNode.import.maxImportFileSizeInGB

    Description Default Value
    The maximum file size (in GB) for an import file, where an import file refers to either a Row-Based file or a set of Column-Based files. 16

    dataNode.import.readBufferSizeInMB

    Description Default Value
    The data block size (in MB) read from chunk manager by the datanode during import. 16

    dataNode.compaction.levelZeroBatchMemoryRatio

    Description Default Value
    The minimal memory ratio of free memory for level zero compaction executing in batch mode 0.05

    dataNode.compaction.levelZeroMaxBatchSize

    Description Default Value
    Max batch size refers to the max number of L1/L2 segments in a batch when executing L0 compaction. Default to -1, any value that is less than 1 means no limit. Valid range: >= 1. -1

    dataNode.gracefulStopTimeout

    Description Default Value
    seconds. force stop node without graceful stop 1800

    dataNode.slot.slotCap

    Description Default Value
    The maximum number of tasks(e.g. compaction, importing) allowed to run concurrently on a datanode 16

    dataNode.clusteringCompaction.memoryBufferRatio

    Description Default Value
    The ratio of memory buffer of clustering compaction. Data larger than threshold will be flushed to storage. 0.1

    dataNode.clusteringCompaction.workPoolSize

    Description Default Value
    worker pool size for one clustering compaction job. 8

    dataNode.ip

    Description Default Value
    TCP/IP address of dataNode. If not specified, use the first unicastable address

    dataNode.port

    Description Default Value
    TCP port of dataNode 21124

    dataNode.grpc.serverMaxSendSize

    Description Default Value
    The maximum size of each RPC request that the dataNode can send, unit: byte 536870912

    dataNode.grpc.serverMaxRecvSize

    Description Default Value
    The maximum size of each RPC request that the dataNode can receive, unit: byte 268435456

    dataNode.grpc.clientMaxSendSize

    Description Default Value
    The maximum size of each RPC request that the clients on dataNode can send, unit: byte 268435456

    dataNode.grpc.clientMaxRecvSize

    Description Default Value
    The maximum size of each RPC request that the clients on dataNode can receive, unit: byte 536870912
    Table of contents
    Feedback

    Was this page helpful?