clickhouse-copier

将数据从一个集群中的表复制到另一个（或相同）集群中的。

您可以在不同的服务器上运行多个clickhouse-copier实例来执行相同的任务。ZooKeeper用于同步进程。

clickhouse-copier开始后：

连接到Zookeeper然后接收：
- 复制任务。
- 复制任务的状态。
执行任务。每个运行中的进程选择源集群中“最近”的分片，并将数据拷贝到目标集群中，如果有必要的话还会对数据重新分片。

clickhouse-copier 跟踪ZooKeeper中的更改并动态应用它们。

为了减少网络通信，我们建议在源数据所在的同一台服务器上运行clickhouse-copier。

运行 Clickhouse-copier

工具需要手动运行：

$ clickhouse-copier copier --daemon --config zookeeper.xml --task-path /task/path --base-dir /path/to/dir

参数：

daemon — 在后台模式下启动clickhouse-copier。
config — ZooKeeper. xml文件的路径，其中包含ZooKeeper连接的参数。
task-path — ZooKeeper节点的路径。此节点用于同步clickhouse-copier进程和存储任务。任务存储在$task-path/description中。
task-file — 可选的任务配置的文件路径，用于初始上传到ZooKeeper。
task-upload-force — 强制上传任务文件，即使节点已经存在。
base-dir — 日志和附属文件的路径。当它启动时，clickhouse-copier在$base-dir中创建clickhouse-copier_YYYYMMHHSS_<PID>子目录。如果省略此参数，则在启动clickhouse-copier的目录中创建目录。

Zookeeper.xml 格式

<yandex>
    <logger>
        <level>trace</level>
        <size>100M</size>
        <count>3</count>
    </logger>

    <zookeeper>
        <node index="1">
            <host>127.0.0.1</host>
            <port>2181</port>
        </node>
    </zookeeper>
</yandex>

同步任务配置

<yandex>
    <!-- Configuration of clusters as in an ordinary server config -->
    <remote_servers>
        <source_cluster>
            <shard>
                <internal_replication>false</internal_replication>
                    <replica>
                        <host>127.0.0.1</host>
                        <port>9000</port>
                    </replica>
            </shard>
            ...
        </source_cluster>

        <destination_cluster>
        ...
        </destination_cluster>
    </remote_servers>

    <!-- How many simultaneously active workers are possible. If you run more workers superfluous workers will sleep. -->
    <max_workers>2</max_workers>

    <!-- Setting used to fetch (pull) data from source cluster tables -->
    <settings_pull>
        <readonly>1</readonly>
    </settings_pull>

    <!-- Setting used to insert (push) data to destination cluster tables -->
    <settings_push>
        <readonly>0</readonly>
    </settings_push>

    <!-- Common setting for fetch (pull) and insert (push) operations. Also, copier process context uses it.
         They are overlaid by <settings_pull/> and <settings_push/> respectively. -->
    <settings>
        <connect_timeout>3</connect_timeout>
        <!-- Sync insert is set forcibly, leave it here just in case. -->
        <insert_distributed_sync>1</insert_distributed_sync>
    </settings>

    <!-- Copying tasks description.
         You could specify several table task in the same task description (in the same ZooKeeper node), they will be performed
         sequentially.
    -->
    <tables>
        <!-- A table task, copies one table. -->
        <table_hits>
            <!-- Source cluster name (from <remote_servers/> section) and tables in it that should be copied -->
            <cluster_pull>source_cluster</cluster_pull>
            <database_pull>test</database_pull>
            <table_pull>hits</table_pull>

            <!-- Destination cluster name and tables in which the data should be inserted -->
            <cluster_push>destination_cluster</cluster_push>
            <database_push>test</database_push>
            <table_push>hits2</table_push>

            <!-- Engine of destination tables.
                 If destination tables have not be created, workers create them using columns definition from source tables and engine
                 definition from here.

                 NOTE: If the first worker starts insert data and detects that destination partition is not empty then the partition will
                 be dropped and refilled, take it into account if you already have some data in destination tables. You could directly
                 specify partitions that should be copied in <enabled_partitions/>, they should be in quoted format like partition column of
                 system.parts table.
            -->
            <engine>
            ENGINE=ReplicatedMergeTree('/clickhouse/tables/{cluster}/{shard}/hits2', '{replica}')
            PARTITION BY toMonday(date)
            ORDER BY (CounterID, EventDate)
            </engine>

            <!-- Sharding key used to insert data to destination cluster -->
            <sharding_key>jumpConsistentHash(intHash64(UserID), 2)</sharding_key>

            <!-- Optional expression that filter data while pull them from source servers -->
            <where_condition>CounterID != 0</where_condition>

            <!-- This section specifies partitions that should be copied, other partition will be ignored.
                 Partition names should have the same format as
                 partition column of system.parts table (i.e. a quoted text).
                 Since partition key of source and destination cluster could be different,
                 these partition names specify destination partitions.

                 NOTE: In spite of this section is optional (if it is not specified, all partitions will be copied),
                 it is strictly recommended to specify them explicitly.
                 If you already have some ready partitions on destination cluster they
                 will be removed at the start of the copying since they will be interpeted
                 as unfinished data from the previous copying!!!
            -->
            <enabled_partitions>
                <partition>'2018-02-26'</partition>
                <partition>'2018-03-05'</partition>
                ...
            </enabled_partitions>
        </table_hits>

        <!-- Next table to copy. It is not copied until previous table is copying. -->
        </table_visits>
        ...
        </table_visits>
        ...
    </tables>
</yandex>

clickhouse-copier跟踪/task/path/description中的更改，并动态应用它们。例如，如果更改max_workers的值，那么运行任务的进程数量也会发生变化。

「如果这篇文章对你有用，请支持一下哦」

ClickHouse官方文档翻译_操作-工具-clickhouse-copier

TOC

clickhouse-copier

运行 Clickhouse-copier

Zookeeper.xml 格式

同步任务配置

FEATURED TAGS

FRIENDS