2023-03-16 16:53:58,131 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-03-16 16:53:58,455 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-03-16 16:53:58,570 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-03-16 16:53:58,632 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-16 16:53:58,640 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-16 16:53:58,653 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=340, ProcessCount=169, AvailableMemoryMB=5523
2023-03-16 16:53:58,655 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-16 16:53:58,655 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-16 16:53:58,779 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv103668902:84 , srv1087195354:84 , srv1123951373:84 , srv1166309597:84 , srv118371573:84 , srv1197340013:84 , srv1214029134:84 , srv12466798:84 , srv1261094494:84 , srv1279409791:84 , srv128876955:84 , srv1293355275:84 , srv1330034346:84 , srv1386693349:84 , srv1400047303:84 , srv1433657675:84 , srv1435065375:84 , srv1439009911:84 , srv1448684350:84 , srv145088493:84 , srv1483377930:84 , srv1492081335:84 , srv1509650501:84 , srv151029645:84 , srv1537110994:84 , srv1570429535:84 , srv1591343837:84 , srv1620128431:84 , srv1636975821:84 , srv1664703088:84 , srv1683269992:84 , srv1700747164:84 , srv1702393967:84 , srv1703888263:84 , srv17164774:84 , srv1721612176:84 , srv1746856027:84 , srv1772109042:84 , srv1791901636:84 , srv1804987538:84 , srv1818537260:84 , srv1837550902:84 , srv1846311037:84 , srv1906865536:84 , srv1908890994:84 , srv1923730841:84 , srv1955656973:84 , srv1974537658:84 , srv2016650758:84 , srv2054902041:84 , srv2065107805:84 , srv2067126174:84 , srv207017730:84 , srv208136066:84 , srv2115074907:84 , srv2143547004:84 , srv215490106:84 , srv21785387:84 , srv227726524:84 , srv269907061:84 , srv28519384:84 , srv331743160:84 , srv375534768:84 , srv380405265:84 , srv383992620:84 , srv386679443:84 , srv395582397:84 , srv421173086:84 , srv451140721:84 , srv46348177:84 , srv472675870:84 , srv502236679:84 , srv516273189:84 , srv525546643:84 , srv543216919:84 , srv558344755:84 , srv577763746:84 , srv608874234:84 , srv646513565:84 , srv669653747:84 , srv684262003:84 , srv702175633:84 , srv704945237:84 , srv720344089:84 , srv726043560:84 , srv726855384:84 , srv736664967:84 , srv761023379:84 , srv769190726:84 , srv789204188:84 , srv800119893:84 , srv82459977:84 , srv858532042:84 , srv869801345:84 , srv900230017:84 , srv90158031:84 , srv939287518:84 , srv948242320:84 , srv974951708:84 , srv1587006062:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-03-16 16:53:58,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-16 16:53:58,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1837550902=42, srv1279409791=9, srv1955656973=47, srv1923730841=46, srv1846311037=43, srv1791901636=39, srv1636975821=29, srv669653747=80, srv2067126174=52, srv1591343837=27, srv502236679=72, srv375534768=63, srv789204188=90, srv1804987538=40, srv151029645=23, srv543216919=75, srv269907061=60, srv646513565=79, srv1087195354=1, srv2065107805=51, srv577763746=77, srv974951708=99, srv331743160=62, srv46348177=70, srv608874234=78, srv1492081335=21, srv869801345=94, srv948242320=98, srv1448684350=18, srv227726524=59, srv451140721=69, srv1166309597=3, srv118371573=4, srv1330034346=12, srv1293355275=11, srv2054902041=50, srv761023379=88, srv1746856027=37, srv900230017=95, srv2016650758=49, srv21785387=58, srv558344755=76, srv720344089=84, srv1620128431=28, srv421173086=68, srv1261094494=8, srv684262003=81, srv1702393967=33, srv28519384=61, srv1818537260=41, srv82459977=92, srv103668902=0, srv1700747164=32, srv386679443=66, srv939287518=97, srv1509650501=22, srv1537110994=24, srv858532042=93, srv769190726=89, srv128876955=10, srv215490106=57, srv1439009911=17, srv726043560=85, srv12466798=7, srv1400047303=14, srv2143547004=56, srv90158031=96, srv1483377930=20, srv702175633=82, srv1664703088=30, srv726855384=86, srv380405265=64, srv1703888263=34, srv1772109042=38, srv472675870=71, srv1386693349=13, srv1908890994=45, srv383992620=65, srv208136066=54, srv525546643=74, srv1433657675=15, srv1570429535=25, srv1721612176=36, srv1214029134=6, srv17164774=35, srv1123951373=2, srv736664967=87, srv207017730=53, srv516273189=73, srv1197340013=5, srv1435065375=16, srv704945237=83, srv1587006062=26, srv145088493=19, srv1683269992=31, srv1906865536=44, srv395582397=67, srv2115074907=55, srv1974537658=48, srv800119893=91} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-16 16:53:58,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-16 16:53:58,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-16 16:53:58,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-16 16:53:58,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-16 16:53:58,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-16 16:53:58,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-16 16:53:58,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-16 16:53:58,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-16 16:53:58,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-16 16:53:58,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-16 16:53:58,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-16 16:53:58,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-16 16:53:58,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-16 16:53:58,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-16 16:53:58,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-16 16:53:58,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-16 16:53:58,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-16 16:53:58,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-16 16:53:58,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-16 16:53:58,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-16 16:53:58,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-16 16:53:58,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-16 16:53:58,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-16 16:53:58,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-16 16:53:58,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-16 16:53:58,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-16 16:53:58,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-16 16:53:58,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-16 16:53:58,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-16 16:53:58,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-16 16:53:58,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-16 16:53:58,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-16 16:53:58,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-16 16:53:58,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-16 16:53:58,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-16 16:53:58,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-16 16:53:58,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-16 16:53:58,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-16 16:53:58,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-16 16:53:58,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-16 16:53:58,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-16 16:53:58,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-16 16:53:58,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-16 16:53:58,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-16 16:53:58,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-16 16:53:58,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-16 16:53:58,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-16 16:53:58,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-16 16:53:58,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-16 16:53:58,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-16 16:53:58,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-16 16:53:58,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-16 16:53:58,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-16 16:53:58,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-16 16:53:58,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-16 16:53:58,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-16 16:53:58,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-16 16:53:58,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-16 16:53:58,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-16 16:53:58,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-16 16:53:58,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-16 16:53:58,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-16 16:53:58,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-16 16:53:58,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-16 16:53:58,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-16 16:53:58,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-16 16:53:58,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-16 16:53:58,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-16 16:53:58,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-16 16:53:58,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-16 16:53:58,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-16 16:53:58,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-16 16:53:58,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-16 16:53:58,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-16 16:53:58,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-16 16:53:58,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-16 16:53:58,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-16 16:53:58,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-16 16:53:58,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-16 16:53:58,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-16 16:53:58,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-16 16:53:58,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-16 16:53:58,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-16 16:53:58,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-16 16:53:58,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-16 16:53:58,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-16 16:53:58,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-16 16:53:58,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-16 16:53:58,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-16 16:53:58,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-16 16:53:58,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-16 16:53:58,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-16 16:53:58,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-16 16:53:58,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-16 16:53:58,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-16 16:53:58,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-16 16:53:58,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-16 16:53:58,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-16 16:53:58,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-16 16:53:58,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-16 16:53:58,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-16 16:53:59,183 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 16:53:59,184 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-16 16:53:59,184 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-16 16:53:59,492 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 576 ms to try 5000 different iterations.  Found a solution that moves 4715 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17206325957319046. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036842925673947056, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05445165897365446, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5238888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.036842925673947056, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.167, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23616666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 16:53:59,506 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv516273189:82 , srv145088493:83 , srv151029645:83 , srv1570429535:83 , srv1721612176:83 , srv375534768:83 , srv1261094494:84 , srv128876955:84 , srv1330034346:84 , srv1439009911:84 , srv1483377930:84 , srv1537110994:84 , srv1591343837:84 , srv1636975821:84 , srv1746856027:84 , srv1804987538:84 , srv1974537658:84 , srv208136066:84 , srv215490106:84 , srv383992620:84 , srv525546643:84 , srv720344089:84 , srv1123951373:85 , srv1166309597:85 , srv1279409791:85 , srv1433657675:85 , srv1492081335:85 , srv1791901636:85 , srv1818537260:85 , srv1906865536:85 , srv2016650758:85 , srv2065107805:85 , srv227726524:85 , srv28519384:85 , srv380405265:85 , srv543216919:85 , srv669653747:85 , srv726855384:85 , srv736664967:85 , srv939287518:85 , srv1087195354:86 , srv12466798:86 , srv1293355275:86 , srv1386693349:86 , srv17164774:86 , srv1908890994:86 , srv21785387:86 , srv269907061:86 , srv331743160:86 , srv472675870:86 , srv702175633:86 , srv769190726:86 , srv948242320:86 , srv974951708:86 , srv103668902:87 , srv118371573:87 , srv1435065375:87 , srv1509650501:87 , srv1702393967:87 , srv1923730841:87 , srv1955656973:87 , srv395582397:87 , srv451140721:87 , srv46348177:87 , srv558344755:87 , srv608874234:87 , srv684262003:87 , srv761023379:87 , srv800119893:87 , srv900230017:87 , srv1197340013:88 , srv1214029134:88 , srv1683269992:88 , srv1700747164:88 , srv1703888263:88 , srv2054902041:88 , srv2115074907:88 , srv386679443:88 , srv726043560:88 , srv1448684350:89 , srv1664703088:89 , srv1772109042:89 , srv1837550902:89 , srv577763746:89 , srv789204188:89 , srv82459977:89 , srv869801345:89 , srv1620128431:90 , srv502236679:90 , srv704945237:90 , srv90158031:90 , srv2067126174:91 , srv207017730:91 , srv1846311037:92 , srv858532042:93 , srv2143547004:95 , srv421173086:96 , srv1400047303:97 , srv646513565:97 , srv1587006062:417 }
2023-03-16 16:53:59,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-16 16:53:59,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1837550902=42, srv1279409791=9, srv1955656973=47, srv1923730841=46, srv1846311037=43, srv1791901636=39, srv1636975821=29, srv669653747=80, srv2067126174=52, srv1591343837=27, srv502236679=72, srv375534768=63, srv789204188=90, srv1804987538=40, srv151029645=23, srv543216919=75, srv269907061=60, srv646513565=79, srv1087195354=1, srv2065107805=51, srv577763746=77, srv974951708=99, srv331743160=62, srv46348177=70, srv608874234=78, srv1492081335=21, srv869801345=94, srv948242320=98, srv1448684350=18, srv227726524=59, srv451140721=69, srv1166309597=3, srv118371573=4, srv1330034346=12, srv1293355275=11, srv2054902041=50, srv761023379=88, srv1746856027=37, srv900230017=95, srv2016650758=49, srv21785387=58, srv558344755=76, srv720344089=84, srv1620128431=28, srv421173086=68, srv1261094494=8, srv684262003=81, srv1702393967=33, srv28519384=61, srv1818537260=41, srv82459977=92, srv103668902=0, srv1700747164=32, srv386679443=66, srv939287518=97, srv1509650501=22, srv1537110994=24, srv858532042=93, srv769190726=89, srv128876955=10, srv215490106=57, srv1439009911=17, srv726043560=85, srv12466798=7, srv1400047303=14, srv2143547004=56, srv90158031=96, srv1483377930=20, srv702175633=82, srv1664703088=30, srv726855384=86, srv380405265=64, srv1703888263=34, srv1772109042=38, srv472675870=71, srv1386693349=13, srv1908890994=45, srv383992620=65, srv208136066=54, srv525546643=74, srv1433657675=15, srv1570429535=25, srv1721612176=36, srv1214029134=6, srv17164774=35, srv1123951373=2, srv736664967=87, srv207017730=53, srv516273189=73, srv1197340013=5, srv1435065375=16, srv704945237=83, srv1587006062=26, srv145088493=19, srv1683269992=31, srv1906865536=44, srv395582397=67, srv2115074907=55, srv1974537658=48, srv800119893=91} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-16 16:53:59,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-16 16:53:59,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-16 16:53:59,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-16 16:53:59,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-16 16:53:59,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-16 16:53:59,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-16 16:53:59,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-16 16:53:59,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-16 16:53:59,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-16 16:53:59,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-16 16:53:59,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-16 16:53:59,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-16 16:53:59,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-16 16:53:59,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-16 16:53:59,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-16 16:53:59,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-16 16:53:59,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-16 16:53:59,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-16 16:53:59,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-16 16:53:59,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-16 16:53:59,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-16 16:53:59,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-16 16:53:59,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-16 16:53:59,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-16 16:53:59,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-16 16:53:59,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-16 16:53:59,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-16 16:53:59,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-16 16:53:59,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-16 16:53:59,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-16 16:53:59,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-16 16:53:59,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-16 16:53:59,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-16 16:53:59,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-16 16:53:59,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-16 16:53:59,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-16 16:53:59,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-16 16:53:59,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-16 16:53:59,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-16 16:53:59,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-16 16:53:59,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-16 16:53:59,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-16 16:53:59,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-16 16:53:59,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-16 16:53:59,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-16 16:53:59,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-16 16:53:59,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-16 16:53:59,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-16 16:53:59,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-16 16:53:59,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-16 16:53:59,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-16 16:53:59,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-16 16:53:59,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-16 16:53:59,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-16 16:53:59,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-16 16:53:59,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-16 16:53:59,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-16 16:53:59,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-16 16:53:59,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-16 16:53:59,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-16 16:53:59,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-16 16:53:59,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-16 16:53:59,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-16 16:53:59,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-16 16:53:59,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-16 16:53:59,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-16 16:53:59,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-16 16:53:59,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-16 16:53:59,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-16 16:53:59,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-16 16:53:59,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-16 16:53:59,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-16 16:53:59,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-16 16:53:59,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-16 16:53:59,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-16 16:53:59,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-16 16:53:59,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-16 16:53:59,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-16 16:53:59,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-16 16:53:59,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-16 16:53:59,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-16 16:53:59,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-16 16:53:59,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-16 16:53:59,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-16 16:53:59,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-16 16:53:59,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-16 16:53:59,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-16 16:53:59,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-16 16:53:59,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-16 16:53:59,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-16 16:53:59,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-16 16:53:59,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-16 16:53:59,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-16 16:53:59,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-16 16:53:59,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-16 16:53:59,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-16 16:53:59,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-16 16:53:59,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-16 16:53:59,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-16 16:53:59,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-16 16:53:59,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-16 16:53:59,725 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036842925673947056, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05445165897365446, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.036842925673947056, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.167, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23616666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 16:53:59,726 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-16 16:53:59,726 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17203024443754383, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036842925673947056, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05445165897365446, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.036842925673947056, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.167, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23616666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-16 16:53:59,983 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 421 ms to try 5000 different iterations.  Found a solution that moves 2662 regions; Going from a computed imbalance of 0.17203024443754383 to a new imbalance of 0.015290939008496889. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020919742240605957); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03375158474061338, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29577777777777775, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020919742240605957); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044333333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 16:53:59,993 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1400047303:75 , srv646513565:77 , srv2143547004:78 , srv421173086:78 , srv1846311037:79 , srv207017730:80 , srv1620128431:81 , srv577763746:81 , srv858532042:81 , srv1664703088:82 , srv704945237:82 , srv789204188:82 , srv869801345:82 , srv90158031:82 , srv1214029134:83 , srv1448684350:83 , srv1683269992:83 , srv2054902041:83 , srv118371573:84 , srv1197340013:84 , srv1700747164:84 , srv1772109042:84 , srv331743160:84 , srv386679443:84 , srv502236679:84 , srv608874234:84 , srv726043560:84 , srv82459977:84 , srv900230017:84 , srv103668902:85 , srv1123951373:85 , srv1703888263:85 , srv1818537260:85 , srv1837550902:85 , srv1955656973:85 , srv2065107805:85 , srv2067126174:85 , srv2115074907:85 , srv46348177:85 , srv684262003:85 , srv939287518:85 , srv948242320:85 , srv1087195354:86 , srv1261094494:86 , srv128876955:86 , srv1386693349:86 , srv1435065375:86 , srv1483377930:86 , srv1702393967:86 , srv1791901636:86 , srv1908890994:86 , srv1923730841:86 , srv2016650758:86 , srv395582397:86 , srv472675870:86 , srv543216919:86 , srv669653747:86 , srv702175633:86 , srv726855384:86 , srv761023379:86 , srv769190726:86 , srv800119893:86 , srv974951708:86 , srv12466798:87 , srv1293355275:87 , srv1433657675:87 , srv1492081335:87 , srv1509650501:87 , srv1537110994:87 , srv17164774:87 , srv1746856027:87 , srv1906865536:87 , srv215490106:87 , srv21785387:87 , srv227726524:87 , srv269907061:87 , srv28519384:87 , srv451140721:87 , srv1166309597:88 , srv1279409791:88 , srv1330034346:88 , srv145088493:88 , srv1591343837:88 , srv1804987538:88 , srv1974537658:88 , srv208136066:88 , srv525546643:88 , srv558344755:88 , srv720344089:88 , srv736664967:88 , srv1439009911:89 , srv1636975821:89 , srv380405265:89 , srv151029645:90 , srv1570429535:90 , srv383992620:90 , srv516273189:90 , srv1721612176:91 , srv375534768:91 , srv1587006062:543 }
2023-03-16 16:53:59,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-16 16:53:59,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1837550902=42, srv1279409791=9, srv1955656973=47, srv1923730841=46, srv1846311037=43, srv1791901636=39, srv1636975821=29, srv669653747=80, srv2067126174=52, srv1591343837=27, srv502236679=72, srv375534768=63, srv789204188=90, srv1804987538=40, srv151029645=23, srv543216919=75, srv269907061=60, srv646513565=79, srv1087195354=1, srv2065107805=51, srv577763746=77, srv974951708=99, srv331743160=62, srv46348177=70, srv608874234=78, srv1492081335=21, srv869801345=94, srv948242320=98, srv1448684350=18, srv227726524=59, srv451140721=69, srv1166309597=3, srv118371573=4, srv1330034346=12, srv1293355275=11, srv2054902041=50, srv761023379=88, srv1746856027=37, srv900230017=95, srv2016650758=49, srv21785387=58, srv558344755=76, srv720344089=84, srv1620128431=28, srv421173086=68, srv1261094494=8, srv684262003=81, srv1702393967=33, srv28519384=61, srv1818537260=41, srv82459977=92, srv103668902=0, srv1700747164=32, srv386679443=66, srv939287518=97, srv1509650501=22, srv1537110994=24, srv858532042=93, srv769190726=89, srv128876955=10, srv215490106=57, srv1439009911=17, srv726043560=85, srv12466798=7, srv1400047303=14, srv2143547004=56, srv90158031=96, srv1483377930=20, srv702175633=82, srv1664703088=30, srv726855384=86, srv380405265=64, srv1703888263=34, srv1772109042=38, srv472675870=71, srv1386693349=13, srv1908890994=45, srv383992620=65, srv208136066=54, srv525546643=74, srv1433657675=15, srv1570429535=25, srv1721612176=36, srv1214029134=6, srv17164774=35, srv1123951373=2, srv736664967=87, srv207017730=53, srv516273189=73, srv1197340013=5, srv1435065375=16, srv704945237=83, srv1587006062=26, srv145088493=19, srv1683269992=31, srv1906865536=44, srv395582397=67, srv2115074907=55, srv1974537658=48, srv800119893=91} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-16 16:54:00,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-16 16:54:00,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-16 16:54:00,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-16 16:54:00,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-16 16:54:00,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-16 16:54:00,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-16 16:54:00,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-16 16:54:00,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-16 16:54:00,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-16 16:54:00,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-16 16:54:00,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-16 16:54:00,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-16 16:54:00,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-16 16:54:00,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-16 16:54:00,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-16 16:54:00,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-16 16:54:00,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-16 16:54:00,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-16 16:54:00,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-16 16:54:00,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-16 16:54:00,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-16 16:54:00,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-16 16:54:00,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-16 16:54:00,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-16 16:54:00,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-16 16:54:00,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-16 16:54:00,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-16 16:54:00,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-16 16:54:00,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-16 16:54:00,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-16 16:54:00,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-16 16:54:00,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-16 16:54:00,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-16 16:54:00,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-16 16:54:00,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-16 16:54:00,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-16 16:54:00,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-16 16:54:00,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-16 16:54:00,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-16 16:54:00,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-16 16:54:00,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-16 16:54:00,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-16 16:54:00,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-16 16:54:00,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-16 16:54:00,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-16 16:54:00,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-16 16:54:00,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-16 16:54:00,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-16 16:54:00,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-16 16:54:00,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-16 16:54:00,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-16 16:54:00,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-16 16:54:00,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-16 16:54:00,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-16 16:54:00,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-16 16:54:00,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-16 16:54:00,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-16 16:54:00,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-16 16:54:00,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-16 16:54:00,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-16 16:54:00,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-16 16:54:00,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-16 16:54:00,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-16 16:54:00,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-16 16:54:00,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-16 16:54:00,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-16 16:54:00,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-16 16:54:00,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-16 16:54:00,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-16 16:54:00,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-16 16:54:00,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-16 16:54:00,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-16 16:54:00,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-16 16:54:00,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-16 16:54:00,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-16 16:54:00,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-16 16:54:00,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-16 16:54:00,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-16 16:54:00,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-16 16:54:00,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-16 16:54:00,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-16 16:54:00,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-16 16:54:00,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-16 16:54:00,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-16 16:54:00,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-16 16:54:00,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-16 16:54:00,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-16 16:54:00,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-16 16:54:00,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-16 16:54:00,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-16 16:54:00,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-16 16:54:00,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-16 16:54:00,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-16 16:54:00,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-16 16:54:00,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-16 16:54:00,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-16 16:54:00,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-16 16:54:00,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-16 16:54:00,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-16 16:54:00,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-16 16:54:00,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-16 16:54:00,168 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020919742240605957); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03375158474061338, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020919742240605957); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044333333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 16:54:00,168 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-16 16:54:00,168 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015272299286102112, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020919742240605957); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03375158474061338, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020919742240605957); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044333333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-16 16:54:00,425 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 385 ms to try 5000 different iterations.  Found a solution that moves 975 regions; Going from a computed imbalance of 0.015272299286102112 to a new imbalance of 9.508666207358007E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005322847904476634); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009026709338484401); MoveCostFunction : (multiplier=7.0, imbalance=0.10833333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005322847904476634); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 16:54:00,427 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2067126174:81 , srv1509650501:83 , srv1721612176:83 , srv1837550902:83 , srv380405265:83 , srv383992620:83 , srv421173086:83 , srv451140721:83 , srv502236679:83 , srv558344755:83 , srv646513565:83 , srv12466798:84 , srv1435065375:84 , srv1570429535:84 , srv1702393967:84 , srv1703888263:84 , srv2115074907:84 , srv2143547004:84 , srv21785387:84 , srv269907061:84 , srv375534768:84 , srv726043560:84 , srv736664967:84 , srv858532042:84 , srv103668902:85 , srv1279409791:85 , srv1293355275:85 , srv1330034346:85 , srv1386693349:85 , srv1400047303:85 , srv1433657675:85 , srv1439009911:85 , srv1492081335:85 , srv151029645:85 , srv1591343837:85 , srv1636975821:85 , srv1700747164:85 , srv17164774:85 , srv1772109042:85 , srv1804987538:85 , srv1923730841:85 , srv1955656973:85 , srv1974537658:85 , srv208136066:85 , srv28519384:85 , srv395582397:85 , srv46348177:85 , srv761023379:85 , srv800119893:85 , srv90158031:85 , srv974951708:85 , srv1087195354:86 , srv1166309597:86 , srv1197340013:86 , srv1214029134:86 , srv1448684350:86 , srv1620128431:86 , srv1683269992:86 , srv1746856027:86 , srv1906865536:86 , srv1908890994:86 , srv2016650758:86 , srv2054902041:86 , srv207017730:86 , srv215490106:86 , srv227726524:86 , srv386679443:86 , srv472675870:86 , srv516273189:86 , srv525546643:86 , srv684262003:86 , srv702175633:86 , srv704945237:86 , srv720344089:86 , srv726855384:86 , srv769190726:86 , srv82459977:86 , srv869801345:86 , srv1123951373:87 , srv118371573:87 , srv1261094494:87 , srv128876955:87 , srv1483377930:87 , srv1664703088:87 , srv1791901636:87 , srv1818537260:87 , srv1846311037:87 , srv331743160:87 , srv543216919:87 , srv608874234:87 , srv669653747:87 , srv900230017:87 , srv145088493:88 , srv1537110994:88 , srv2065107805:88 , srv577763746:88 , srv789204188:88 , srv948242320:88 , srv939287518:89 , srv1587006062:545 }
2023-03-16 16:54:00,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-16 16:54:00,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1837550902=42, srv1279409791=9, srv1955656973=47, srv1923730841=46, srv1846311037=43, srv1791901636=39, srv1636975821=29, srv669653747=80, srv2067126174=52, srv1591343837=27, srv502236679=72, srv375534768=63, srv789204188=90, srv1804987538=40, srv151029645=23, srv543216919=75, srv269907061=60, srv646513565=79, srv1087195354=1, srv2065107805=51, srv577763746=77, srv974951708=99, srv331743160=62, srv46348177=70, srv608874234=78, srv1492081335=21, srv869801345=94, srv948242320=98, srv1448684350=18, srv227726524=59, srv451140721=69, srv1166309597=3, srv118371573=4, srv1330034346=12, srv1293355275=11, srv2054902041=50, srv761023379=88, srv1746856027=37, srv900230017=95, srv2016650758=49, srv21785387=58, srv558344755=76, srv720344089=84, srv1620128431=28, srv421173086=68, srv1261094494=8, srv684262003=81, srv1702393967=33, srv28519384=61, srv1818537260=41, srv82459977=92, srv103668902=0, srv1700747164=32, srv386679443=66, srv939287518=97, srv1509650501=22, srv1537110994=24, srv858532042=93, srv769190726=89, srv128876955=10, srv215490106=57, srv1439009911=17, srv726043560=85, srv12466798=7, srv1400047303=14, srv2143547004=56, srv90158031=96, srv1483377930=20, srv702175633=82, srv1664703088=30, srv726855384=86, srv380405265=64, srv1703888263=34, srv1772109042=38, srv472675870=71, srv1386693349=13, srv1908890994=45, srv383992620=65, srv208136066=54, srv525546643=74, srv1433657675=15, srv1570429535=25, srv1721612176=36, srv1214029134=6, srv17164774=35, srv1123951373=2, srv736664967=87, srv207017730=53, srv516273189=73, srv1197340013=5, srv1435065375=16, srv704945237=83, srv1587006062=26, srv145088493=19, srv1683269992=31, srv1906865536=44, srv395582397=67, srv2115074907=55, srv1974537658=48, srv800119893=91} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-16 16:54:00,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-16 16:54:00,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-16 16:54:00,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-16 16:54:00,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-16 16:54:00,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-16 16:54:00,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-16 16:54:00,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-16 16:54:00,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-16 16:54:00,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-16 16:54:00,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-16 16:54:00,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-16 16:54:00,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-16 16:54:00,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-16 16:54:00,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-16 16:54:00,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-16 16:54:00,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-16 16:54:00,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-16 16:54:00,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-16 16:54:00,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-16 16:54:00,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-16 16:54:00,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-16 16:54:00,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-16 16:54:00,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-16 16:54:00,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-16 16:54:00,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-16 16:54:00,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-16 16:54:00,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-16 16:54:00,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-16 16:54:00,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-16 16:54:00,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-16 16:54:00,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-16 16:54:00,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-16 16:54:00,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-16 16:54:00,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-16 16:54:00,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-16 16:54:00,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-16 16:54:00,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-16 16:54:00,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-16 16:54:00,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-16 16:54:00,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-16 16:54:00,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-16 16:54:00,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-16 16:54:00,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-16 16:54:00,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-16 16:54:00,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-16 16:54:00,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-16 16:54:00,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-16 16:54:00,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-16 16:54:00,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-16 16:54:00,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-16 16:54:00,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-16 16:54:00,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-16 16:54:00,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-16 16:54:00,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-16 16:54:00,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-16 16:54:00,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-16 16:54:00,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-16 16:54:00,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-16 16:54:00,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-16 16:54:00,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-16 16:54:00,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-16 16:54:00,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-16 16:54:00,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-16 16:54:00,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-16 16:54:00,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-16 16:54:00,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-16 16:54:00,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-16 16:54:00,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-16 16:54:00,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-16 16:54:00,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-16 16:54:00,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-16 16:54:00,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-16 16:54:00,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-16 16:54:00,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-16 16:54:00,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-16 16:54:00,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-16 16:54:00,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-16 16:54:00,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-16 16:54:00,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-16 16:54:00,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-16 16:54:00,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-16 16:54:00,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-16 16:54:00,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-16 16:54:00,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-16 16:54:00,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-16 16:54:00,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-16 16:54:00,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-16 16:54:00,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-16 16:54:00,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-16 16:54:00,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-16 16:54:00,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-16 16:54:00,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-16 16:54:00,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-16 16:54:00,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-16 16:54:00,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-16 16:54:00,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-16 16:54:00,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-16 16:54:00,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-16 16:54:00,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-16 16:54:00,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-16 16:54:00,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-16 16:54:00,573 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005322847904476634); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009026709338484401); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005322847904476634); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 16:54:00,574 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-16 16:54:00,574 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.440395248173536E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005322847904476634); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009026709338484401); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005322847904476634); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-16 16:54:00,920 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 459 ms to try 5000 different iterations.  Found a solution that moves 246 regions; Going from a computed imbalance of 9.440395248173536E-4 to a new imbalance of 1.9355963557641567E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.027333333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 16:54:00,921 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1166309597:83 , srv145088493:83 , srv1537110994:83 , srv769190726:83 , srv789204188:83 , srv82459977:83 , srv939287518:83 , srv948242320:83 , srv1087195354:84 , srv118371573:84 , srv1279409791:84 , srv128876955:84 , srv1293355275:84 , srv1330034346:84 , srv1400047303:84 , srv1448684350:84 , srv151029645:84 , srv1636975821:84 , srv1664703088:84 , srv17164774:84 , srv1772109042:84 , srv1791901636:84 , srv1906865536:84 , srv1908890994:84 , srv1923730841:84 , srv2054902041:84 , srv2065107805:84 , srv2115074907:84 , srv227726524:84 , srv375534768:84 , srv386679443:84 , srv472675870:84 , srv516273189:84 , srv525546643:84 , srv543216919:84 , srv577763746:84 , srv608874234:84 , srv669653747:84 , srv702175633:84 , srv704945237:84 , srv720344089:84 , srv726855384:84 , srv761023379:84 , srv800119893:84 , srv858532042:84 , srv900230017:84 , srv103668902:85 , srv1123951373:85 , srv1197340013:85 , srv1214029134:85 , srv12466798:85 , srv1261094494:85 , srv1386693349:85 , srv1433657675:85 , srv1435065375:85 , srv1439009911:85 , srv1483377930:85 , srv1492081335:85 , srv1509650501:85 , srv1570429535:85 , srv1591343837:85 , srv1683269992:85 , srv1700747164:85 , srv1702393967:85 , srv1703888263:85 , srv1721612176:85 , srv1746856027:85 , srv1804987538:85 , srv1818537260:85 , srv1837550902:85 , srv1846311037:85 , srv1974537658:85 , srv2016650758:85 , srv2067126174:85 , srv207017730:85 , srv208136066:85 , srv2143547004:85 , srv215490106:85 , srv21785387:85 , srv269907061:85 , srv28519384:85 , srv331743160:85 , srv380405265:85 , srv383992620:85 , srv395582397:85 , srv421173086:85 , srv451140721:85 , srv46348177:85 , srv502236679:85 , srv558344755:85 , srv646513565:85 , srv684262003:85 , srv736664967:85 , srv869801345:85 , srv90158031:85 , srv974951708:85 , srv1620128431:86 , srv1955656973:86 , srv726043560:86 , srv1587006062:636 }
2023-03-16 16:54:00,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-16 16:54:00,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1837550902=42, srv1279409791=9, srv1955656973=47, srv1923730841=46, srv1846311037=43, srv1791901636=39, srv1636975821=29, srv669653747=80, srv2067126174=52, srv1591343837=27, srv502236679=72, srv375534768=63, srv789204188=90, srv1804987538=40, srv151029645=23, srv543216919=75, srv269907061=60, srv646513565=79, srv1087195354=1, srv2065107805=51, srv577763746=77, srv974951708=99, srv331743160=62, srv46348177=70, srv608874234=78, srv1492081335=21, srv869801345=94, srv948242320=98, srv1448684350=18, srv227726524=59, srv451140721=69, srv1166309597=3, srv118371573=4, srv1330034346=12, srv1293355275=11, srv2054902041=50, srv761023379=88, srv1746856027=37, srv900230017=95, srv2016650758=49, srv21785387=58, srv558344755=76, srv720344089=84, srv1620128431=28, srv421173086=68, srv1261094494=8, srv684262003=81, srv1702393967=33, srv28519384=61, srv1818537260=41, srv82459977=92, srv103668902=0, srv1700747164=32, srv386679443=66, srv939287518=97, srv1509650501=22, srv1537110994=24, srv858532042=93, srv769190726=89, srv128876955=10, srv215490106=57, srv1439009911=17, srv726043560=85, srv12466798=7, srv1400047303=14, srv2143547004=56, srv90158031=96, srv1483377930=20, srv702175633=82, srv1664703088=30, srv726855384=86, srv380405265=64, srv1703888263=34, srv1772109042=38, srv472675870=71, srv1386693349=13, srv1908890994=45, srv383992620=65, srv208136066=54, srv525546643=74, srv1433657675=15, srv1570429535=25, srv1721612176=36, srv1214029134=6, srv17164774=35, srv1123951373=2, srv736664967=87, srv207017730=53, srv516273189=73, srv1197340013=5, srv1435065375=16, srv704945237=83, srv1587006062=26, srv145088493=19, srv1683269992=31, srv1906865536=44, srv395582397=67, srv2115074907=55, srv1974537658=48, srv800119893=91} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-16 16:54:00,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-16 16:54:00,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-16 16:54:00,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-16 16:54:00,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-16 16:54:00,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-16 16:54:00,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-16 16:54:00,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-16 16:54:00,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-16 16:54:00,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-16 16:54:00,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-16 16:54:00,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-16 16:54:00,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-16 16:54:00,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-16 16:54:00,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-16 16:54:00,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-16 16:54:00,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-16 16:54:00,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-16 16:54:00,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-16 16:54:00,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-16 16:54:00,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-16 16:54:00,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-16 16:54:00,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-16 16:54:00,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-16 16:54:00,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-16 16:54:00,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-16 16:54:00,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-16 16:54:00,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-16 16:54:00,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-16 16:54:00,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-16 16:54:00,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-16 16:54:00,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-16 16:54:00,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-16 16:54:00,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-16 16:54:00,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-16 16:54:00,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-16 16:54:00,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-16 16:54:00,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-16 16:54:00,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-16 16:54:00,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-16 16:54:00,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-16 16:54:00,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-16 16:54:00,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-16 16:54:00,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-16 16:54:00,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-16 16:54:00,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-16 16:54:00,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-16 16:54:00,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-16 16:54:00,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-16 16:54:00,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-16 16:54:00,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-16 16:54:00,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-16 16:54:00,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-16 16:54:00,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-16 16:54:00,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-16 16:54:00,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-16 16:54:00,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-16 16:54:00,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-16 16:54:00,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-16 16:54:00,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-16 16:54:00,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-16 16:54:00,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-16 16:54:00,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-16 16:54:00,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-16 16:54:00,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-16 16:54:00,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-16 16:54:00,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-16 16:54:00,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-16 16:54:00,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-16 16:54:00,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-16 16:54:00,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-16 16:54:00,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-16 16:54:00,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-16 16:54:00,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-16 16:54:00,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-16 16:54:00,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-16 16:54:00,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-16 16:54:00,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-16 16:54:00,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-16 16:54:00,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-16 16:54:00,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-16 16:54:00,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-16 16:54:00,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-16 16:54:00,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-16 16:54:00,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-16 16:54:00,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-16 16:54:00,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-16 16:54:00,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-16 16:54:00,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-16 16:54:00,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-16 16:54:00,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-16 16:54:00,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-16 16:54:00,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-16 16:54:00,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-16 16:54:00,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-16 16:54:00,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-16 16:54:00,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-16 16:54:00,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-16 16:54:00,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-16 16:54:00,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-16 16:54:00,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-16 16:54:00,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-16 16:54:01,105 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.7633434741294947E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 16:54:01,105 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1166309597:83 , srv145088493:83 , srv1537110994:83 , srv769190726:83 , srv789204188:83 , srv82459977:83 , srv939287518:83 , srv948242320:83 , srv1087195354:84 , srv118371573:84 , srv1279409791:84 , srv128876955:84 , srv1293355275:84 , srv1330034346:84 , srv1400047303:84 , srv1448684350:84 , srv151029645:84 , srv1636975821:84 , srv1664703088:84 , srv17164774:84 , srv1772109042:84 , srv1791901636:84 , srv1906865536:84 , srv1908890994:84 , srv1923730841:84 , srv2054902041:84 , srv2065107805:84 , srv2115074907:84 , srv227726524:84 , srv375534768:84 , srv386679443:84 , srv472675870:84 , srv516273189:84 , srv525546643:84 , srv543216919:84 , srv577763746:84 , srv608874234:84 , srv669653747:84 , srv702175633:84 , srv704945237:84 , srv720344089:84 , srv726855384:84 , srv761023379:84 , srv800119893:84 , srv858532042:84 , srv900230017:84 , srv103668902:85 , srv1123951373:85 , srv1197340013:85 , srv1214029134:85 , srv12466798:85 , srv1261094494:85 , srv1386693349:85 , srv1433657675:85 , srv1435065375:85 , srv1439009911:85 , srv1483377930:85 , srv1492081335:85 , srv1509650501:85 , srv1570429535:85 , srv1591343837:85 , srv1683269992:85 , srv1700747164:85 , srv1702393967:85 , srv1703888263:85 , srv1721612176:85 , srv1746856027:85 , srv1804987538:85 , srv1818537260:85 , srv1837550902:85 , srv1846311037:85 , srv1974537658:85 , srv2016650758:85 , srv2067126174:85 , srv207017730:85 , srv208136066:85 , srv2143547004:85 , srv215490106:85 , srv21785387:85 , srv269907061:85 , srv28519384:85 , srv331743160:85 , srv380405265:85 , srv383992620:85 , srv395582397:85 , srv421173086:85 , srv451140721:85 , srv46348177:85 , srv502236679:85 , srv558344755:85 , srv646513565:85 , srv684262003:85 , srv736664967:85 , srv869801345:85 , srv90158031:85 , srv974951708:85 , srv1620128431:86 , srv1955656973:86 , srv726043560:86 , srv1587006062:636 }
2023-03-16 16:54:01,144 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=340 (was 340), ProcessCount=169 (was 169), AvailableMemoryMB=5035 (was 5523)
2023-03-16 16:54:01,152 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=340, ProcessCount=169, AvailableMemoryMB=5034
2023-03-16 16:54:01,153 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-16 16:54:01,154 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-03-16 16:54:01,166 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1048438747:84 , srv1091976368:84 , srv1331191340:84 , srv1391168016:84 , srv143614194:84 , srv1451405111:84 , srv1471290382:84 , srv1565194228:84 , srv1610138044:84 , srv1735865956:84 , srv1858313790:84 , srv1871888155:84 , srv1899618383:84 , srv1921045841:84 , srv1921884555:84 , srv202341356:84 , srv20605050:84 , srv2086986882:84 , srv211922820:84 , srv255578995:84 , srv382070212:84 , srv400470055:84 , srv400827996:84 , srv4686969:84 , srv520431885:84 , srv766599939:84 , srv921978499:84 , srv94813660:84 , srv992468472:84 , srv1260954175:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-03-16 16:54:01,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-16 16:54:01,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1391168016=4, srv202341356=16, srv1871888155=12, srv1260954175=2, srv1735865956=10, srv1921884555=15, srv255578995=20, srv520431885=25, srv143614194=5, srv94813660=28, srv1091976368=1, srv1921045841=14, srv1565194228=8, srv2086986882=18, srv1048438747=0, srv382070212=21, srv20605050=17, srv400470055=22, srv400827996=23, srv1331191340=3, srv1471290382=7, srv1451405111=6, srv1899618383=13, srv1858313790=11, srv992468472=29, srv921978499=27, srv766599939=26, srv211922820=19, srv1610138044=9, srv4686969=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-16 16:54:01,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-16 16:54:01,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-16 16:54:01,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-16 16:54:01,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-16 16:54:01,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-16 16:54:01,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-16 16:54:01,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-16 16:54:01,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-16 16:54:01,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-16 16:54:01,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-16 16:54:01,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-16 16:54:01,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-16 16:54:01,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-16 16:54:01,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-16 16:54:01,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-16 16:54:01,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-16 16:54:01,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-16 16:54:01,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-16 16:54:01,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-16 16:54:01,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-16 16:54:01,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-16 16:54:01,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-16 16:54:01,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-16 16:54:01,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-16 16:54:01,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-16 16:54:01,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-16 16:54:01,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-16 16:54:01,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-16 16:54:01,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-16 16:54:01,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-16 16:54:01,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-16 16:54:01,202 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.2086834896767964, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 16:54:01,203 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912850598126107, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.2086834896767964, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-03-16 16:56:01,187 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6007894 different iterations.  Found a solution that moves 1866 regions; Going from a computed imbalance of 0.9912850598126107 to a new imbalance of 4.3553370884861656E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6911111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 16:56:01,192 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1048438747:90 , srv1091976368:90 , srv1260954175:90 , srv1331191340:90 , srv1391168016:90 , srv143614194:90 , srv1451405111:90 , srv1471290382:90 , srv1565194228:90 , srv1610138044:90 , srv1735865956:90 , srv1858313790:90 , srv1871888155:90 , srv1899618383:90 , srv1921045841:90 , srv1921884555:90 , srv202341356:90 , srv20605050:90 , srv2086986882:90 , srv211922820:90 , srv255578995:90 , srv382070212:90 , srv400470055:90 , srv400827996:90 , srv4686969:90 , srv520431885:90 , srv766599939:90 , srv921978499:90 , srv94813660:90 , srv992468472:90 }
2023-03-16 16:56:01,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-16 16:56:01,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1391168016=4, srv202341356=16, srv1871888155=12, srv1260954175=2, srv1735865956=10, srv1921884555=15, srv255578995=20, srv520431885=25, srv143614194=5, srv94813660=28, srv1091976368=1, srv1921045841=14, srv1565194228=8, srv2086986882=18, srv1048438747=0, srv382070212=21, srv20605050=17, srv400470055=22, srv400827996=23, srv1331191340=3, srv1471290382=7, srv1451405111=6, srv1899618383=13, srv1858313790=11, srv992468472=29, srv921978499=27, srv766599939=26, srv211922820=19, srv1610138044=9, srv4686969=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-16 16:56:01,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-16 16:56:01,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-16 16:56:01,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-16 16:56:01,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-16 16:56:01,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-16 16:56:01,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-16 16:56:01,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-16 16:56:01,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-16 16:56:01,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-16 16:56:01,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-16 16:56:01,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-16 16:56:01,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-16 16:56:01,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-16 16:56:01,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-16 16:56:01,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-16 16:56:01,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-16 16:56:01,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-16 16:56:01,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-16 16:56:01,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-16 16:56:01,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-16 16:56:01,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-16 16:56:01,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-16 16:56:01,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-16 16:56:01,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-16 16:56:01,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-16 16:56:01,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-16 16:56:01,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-16 16:56:01,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-16 16:56:01,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-16 16:56:01,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-16 16:56:01,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-16 16:56:01,211 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 16:56:01,211 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1048438747:90 , srv1091976368:90 , srv1260954175:90 , srv1331191340:90 , srv1391168016:90 , srv143614194:90 , srv1451405111:90 , srv1471290382:90 , srv1565194228:90 , srv1610138044:90 , srv1735865956:90 , srv1858313790:90 , srv1871888155:90 , srv1899618383:90 , srv1921045841:90 , srv1921884555:90 , srv202341356:90 , srv20605050:90 , srv2086986882:90 , srv211922820:90 , srv255578995:90 , srv382070212:90 , srv400470055:90 , srv400827996:90 , srv4686969:90 , srv520431885:90 , srv766599939:90 , srv921978499:90 , srv94813660:90 , srv992468472:90 }
2023-03-16 16:56:01,225 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=160 (was 340), ProcessCount=171 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=4675 (was 5034)