2023-05-06 06:53:37,351 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-05-06 06:53:37,692 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-05-06 06:53:37,815 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-05-06 06:53:37,881 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-06 06:53:37,889 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-05-06 06:53:37,906 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=308, ProcessCount=170, AvailableMemoryMB=7990
2023-05-06 06:53:37,908 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-06 06:53:37,909 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-05-06 06:53:38,033 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1005974628:84 , srv1093474111:84 , srv1101894705:84 , srv1123369348:84 , srv1142950224:84 , srv1145531912:84 , srv1155571854:84 , srv1160359875:84 , srv1211440066:84 , srv1215659110:84 , srv1221842450:84 , srv1236285832:84 , srv1245771946:84 , srv1253435477:84 , srv1273538736:84 , srv1286429734:84 , srv1289971554:84 , srv1291278902:84 , srv1324290407:84 , srv136384246:84 , srv1403564880:84 , srv1424649866:84 , srv1438577700:84 , srv1512758595:84 , srv1548100971:84 , srv1552818254:84 , srv1570616866:84 , srv1571785414:84 , srv158458918:84 , srv1634112852:84 , srv1657723673:84 , srv1674306524:84 , srv168693101:84 , srv1717356931:84 , srv1725538067:84 , srv1729696044:84 , srv1733243302:84 , srv174337603:84 , srv1759074194:84 , srv1763925074:84 , srv1790866148:84 , srv1810268602:84 , srv1822466838:84 , srv1838392926:84 , srv1867059654:84 , srv1886114293:84 , srv1938067709:84 , srv1943565271:84 , srv1968581886:84 , srv1990379416:84 , srv1994410918:84 , srv2009421048:84 , srv2011027553:84 , srv2056521096:84 , srv2083793718:84 , srv2116150834:84 , srv2122127519:84 , srv214255845:84 , srv226197838:84 , srv229865535:84 , srv231271109:84 , srv232745635:84 , srv253967200:84 , srv283196351:84 , srv28434284:84 , srv287836745:84 , srv301574390:84 , srv332779991:84 , srv336248297:84 , srv345511646:84 , srv37520101:84 , srv430902771:84 , srv464569882:84 , srv467843285:84 , srv503511617:84 , srv506927175:84 , srv532477618:84 , srv557239515:84 , srv558031642:84 , srv605896259:84 , srv667519050:84 , srv671332037:84 , srv678138865:84 , srv688594592:84 , srv7229803:84 , srv74284502:84 , srv761949344:84 , srv783833725:84 , srv845573201:84 , srv856000960:84 , srv868098620:84 , srv868895296:84 , srv903409715:84 , srv913805760:84 , srv920969153:84 , srv939742344:84 , srv946015722:84 , srv952894393:84 , srv979083227:84 , srv432783215:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-05-06 06:53:38,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-06 06:53:38,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv979083227=99, srv1717356931=33, srv920969153=95, srv1571785414=27, srv1968581886=48, srv1211440066=8, srv1273538736=14, srv1810268602=41, srv1938067709=46, srv678138865=83, srv558031642=79, srv174337603=37, srv432783215=72, srv345511646=69, srv688594592=84, srv7229803=85, srv1867059654=44, srv301574390=66, srv1548100971=24, srv158458918=28, srv1763925074=39, srv214255845=57, srv783833725=88, srv1943565271=47, srv74284502=86, srv1990379416=49, srv1994410918=50, srv28434284=64, srv868895296=92, srv1733243302=36, srv952894393=98, srv1552818254=25, srv845573201=89, srv1215659110=9, srv939742344=96, srv1403564880=20, srv2083793718=54, srv287836745=65, srv430902771=71, srv1674306524=31, srv253967200=62, srv1286429734=15, srv1634112852=29, srv913805760=94, srv467843285=74, srv903409715=93, srv1093474111=1, srv1160359875=7, srv503511617=75, srv2116150834=55, srv1101894705=2, srv2009421048=51, srv1005974628=0, srv667519050=81, srv1245771946=12, srv1291278902=17, srv1725538067=34, srv229865535=59, srv136384246=19, srv1438577700=22, srv332779991=67, srv1123369348=3, srv1729696044=35, srv1790866148=40, srv1570616866=26, srv168693101=32, srv37520101=70, srv761949344=87, srv1324290407=18, srv1142950224=4, srv671332037=82, srv1886114293=45, srv1145531912=5, srv1236285832=11, srv946015722=97, srv557239515=78, srv283196351=63, srv1424649866=21, srv856000960=90, srv1289971554=16, srv226197838=58, srv336248297=68, srv868098620=91, srv1838392926=43, srv605896259=80, srv1253435477=13, srv232745635=61, srv532477618=77, srv231271109=60, srv2122127519=56, srv506927175=76, srv2056521096=53, srv1759074194=38, srv1221842450=10, srv1822466838=42, srv2011027553=52, srv1657723673=30, srv464569882=73, srv1155571854=6, srv1512758595=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-06 06:53:38,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-06 06:53:38,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-06 06:53:38,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-06 06:53:38,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-06 06:53:38,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-06 06:53:38,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-06 06:53:38,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-06 06:53:38,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-06 06:53:38,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-06 06:53:38,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-06 06:53:38,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-06 06:53:38,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-06 06:53:38,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-06 06:53:38,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-06 06:53:38,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-06 06:53:38,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-06 06:53:38,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-06 06:53:38,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-06 06:53:38,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-06 06:53:38,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-06 06:53:38,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-06 06:53:38,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-06 06:53:38,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-06 06:53:38,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-06 06:53:38,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-06 06:53:38,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-06 06:53:38,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-06 06:53:38,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-06 06:53:38,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-06 06:53:38,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-06 06:53:38,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-06 06:53:38,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-06 06:53:38,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-06 06:53:38,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-06 06:53:38,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-06 06:53:38,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-06 06:53:38,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-06 06:53:38,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-06 06:53:38,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-06 06:53:38,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-06 06:53:38,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-06 06:53:38,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-06 06:53:38,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-06 06:53:38,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-06 06:53:38,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-06 06:53:38,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-06 06:53:38,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-06 06:53:38,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-06 06:53:38,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-06 06:53:38,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-06 06:53:38,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-06 06:53:38,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-06 06:53:38,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-06 06:53:38,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-06 06:53:38,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-06 06:53:38,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-06 06:53:38,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-06 06:53:38,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-06 06:53:38,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-06 06:53:38,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-06 06:53:38,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-06 06:53:38,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-06 06:53:38,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-06 06:53:38,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-06 06:53:38,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-06 06:53:38,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-06 06:53:38,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-06 06:53:38,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-06 06:53:38,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-06 06:53:38,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-06 06:53:38,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-06 06:53:38,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-06 06:53:38,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-06 06:53:38,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-06 06:53:38,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-06 06:53:38,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-06 06:53:38,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-06 06:53:38,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-06 06:53:38,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-06 06:53:38,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-06 06:53:38,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-06 06:53:38,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-06 06:53:38,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-06 06:53:38,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-06 06:53:38,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-06 06:53:38,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-06 06:53:38,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-06 06:53:38,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-06 06:53:38,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-06 06:53:38,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-06 06:53:38,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-06 06:53:38,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-06 06:53:38,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-06 06:53:38,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-06 06:53:38,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-06 06:53:38,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-06 06:53:38,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-06 06:53:38,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-06 06:53:38,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-06 06:53:38,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-06 06:53:38,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-06 06:53:38,632 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 06:53:38,632 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-06 06:53:38,633 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-06 06:53:39,385 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1206 ms to try 5000 different iterations.  Found a solution that moves 4708 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17996465281222834. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04076338519756036, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.060593769797493244, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5231111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04076338519756036, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.175, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24341666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 06:53:39,396 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1286429734:79 , srv1160359875:81 , srv1221842450:82 , srv1291278902:82 , srv1886114293:82 , srv229865535:82 , srv868895296:82 , srv952894393:82 , srv1123369348:83 , srv1657723673:83 , srv1867059654:83 , srv1990379416:83 , srv2116150834:83 , srv678138865:83 , srv856000960:83 , srv920969153:83 , srv1253435477:84 , srv1273538736:84 , srv1289971554:84 , srv1571785414:84 , srv174337603:84 , srv2083793718:84 , srv503511617:84 , srv506927175:84 , srv558031642:84 , srv671332037:84 , srv913805760:84 , srv979083227:84 , srv1211440066:85 , srv1215659110:85 , srv1424649866:85 , srv1512758595:85 , srv1943565271:85 , srv2009421048:85 , srv253967200:85 , srv283196351:85 , srv287836745:85 , srv332779991:85 , srv430902771:85 , srv467843285:85 , srv1101894705:86 , srv1245771946:86 , srv1438577700:86 , srv1570616866:86 , srv158458918:86 , srv1674306524:86 , srv1729696044:86 , srv1810268602:86 , srv1994410918:86 , srv214255845:86 , srv74284502:86 , srv845573201:86 , srv868098620:86 , srv946015722:86 , srv1093474111:87 , srv1142950224:87 , srv1155571854:87 , srv1552818254:87 , srv1717356931:87 , srv1733243302:87 , srv1759074194:87 , srv1763925074:87 , srv1838392926:87 , srv1938067709:87 , srv2122127519:87 , srv232745635:87 , srv28434284:87 , srv557239515:87 , srv1145531912:88 , srv1403564880:88 , srv1634112852:88 , srv1725538067:88 , srv2011027553:88 , srv226197838:88 , srv231271109:88 , srv336248297:88 , srv464569882:88 , srv532477618:88 , srv667519050:88 , srv903409715:88 , srv1236285832:89 , srv1324290407:89 , srv136384246:89 , srv2056521096:89 , srv301574390:89 , srv345511646:89 , srv605896259:89 , srv1548100971:90 , srv1822466838:90 , srv1968581886:90 , srv761949344:90 , srv1005974628:91 , srv688594592:91 , srv168693101:92 , srv1790866148:92 , srv37520101:93 , srv783833725:93 , srv7229803:95 , srv939742344:96 , srv432783215:452 }
2023-05-06 06:53:39,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-06 06:53:39,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv979083227=99, srv1717356931=33, srv920969153=95, srv1571785414=27, srv1968581886=48, srv1211440066=8, srv1273538736=14, srv1810268602=41, srv1938067709=46, srv678138865=83, srv558031642=79, srv174337603=37, srv432783215=72, srv345511646=69, srv688594592=84, srv7229803=85, srv1867059654=44, srv301574390=66, srv1548100971=24, srv158458918=28, srv1763925074=39, srv214255845=57, srv783833725=88, srv1943565271=47, srv74284502=86, srv1990379416=49, srv1994410918=50, srv28434284=64, srv868895296=92, srv1733243302=36, srv952894393=98, srv1552818254=25, srv845573201=89, srv1215659110=9, srv939742344=96, srv1403564880=20, srv2083793718=54, srv287836745=65, srv430902771=71, srv1674306524=31, srv253967200=62, srv1286429734=15, srv1634112852=29, srv913805760=94, srv467843285=74, srv903409715=93, srv1093474111=1, srv1160359875=7, srv503511617=75, srv2116150834=55, srv1101894705=2, srv2009421048=51, srv1005974628=0, srv667519050=81, srv1245771946=12, srv1291278902=17, srv1725538067=34, srv229865535=59, srv136384246=19, srv1438577700=22, srv332779991=67, srv1123369348=3, srv1729696044=35, srv1790866148=40, srv1570616866=26, srv168693101=32, srv37520101=70, srv761949344=87, srv1324290407=18, srv1142950224=4, srv671332037=82, srv1886114293=45, srv1145531912=5, srv1236285832=11, srv946015722=97, srv557239515=78, srv283196351=63, srv1424649866=21, srv856000960=90, srv1289971554=16, srv226197838=58, srv336248297=68, srv868098620=91, srv1838392926=43, srv605896259=80, srv1253435477=13, srv232745635=61, srv532477618=77, srv231271109=60, srv2122127519=56, srv506927175=76, srv2056521096=53, srv1759074194=38, srv1221842450=10, srv1822466838=42, srv2011027553=52, srv1657723673=30, srv464569882=73, srv1155571854=6, srv1512758595=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-06 06:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-06 06:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-06 06:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-06 06:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-06 06:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-06 06:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-06 06:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-06 06:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-06 06:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-06 06:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-06 06:53:39,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-06 06:53:39,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-06 06:53:39,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-06 06:53:39,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-06 06:53:39,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-06 06:53:39,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-06 06:53:39,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-06 06:53:39,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-06 06:53:39,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-06 06:53:39,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-06 06:53:39,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-06 06:53:39,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-06 06:53:39,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-06 06:53:39,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-06 06:53:39,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-06 06:53:39,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-06 06:53:39,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-06 06:53:39,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-06 06:53:39,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-06 06:53:39,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-06 06:53:39,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-06 06:53:39,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-06 06:53:39,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-06 06:53:39,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-06 06:53:39,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-06 06:53:39,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-06 06:53:39,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-06 06:53:39,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-06 06:53:39,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-06 06:53:39,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-06 06:53:39,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-06 06:53:39,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-06 06:53:39,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-06 06:53:39,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-06 06:53:39,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-06 06:53:39,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-06 06:53:39,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-06 06:53:39,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-06 06:53:39,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-06 06:53:39,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-06 06:53:39,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-06 06:53:39,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-06 06:53:39,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-06 06:53:39,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-06 06:53:39,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-06 06:53:39,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-06 06:53:39,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-06 06:53:39,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-06 06:53:39,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-06 06:53:39,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-06 06:53:39,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-06 06:53:39,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-06 06:53:39,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-06 06:53:39,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-06 06:53:39,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-06 06:53:39,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-06 06:53:39,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-06 06:53:39,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-06 06:53:39,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-06 06:53:39,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-06 06:53:39,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-06 06:53:39,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-06 06:53:39,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-06 06:53:39,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-06 06:53:39,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-06 06:53:39,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-06 06:53:39,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-06 06:53:39,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-06 06:53:39,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-06 06:53:39,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-06 06:53:39,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-06 06:53:39,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-06 06:53:39,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-06 06:53:39,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-06 06:53:39,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-06 06:53:39,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-06 06:53:39,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-06 06:53:39,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-06 06:53:39,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-06 06:53:39,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-06 06:53:39,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-06 06:53:39,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-06 06:53:39,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-06 06:53:39,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-06 06:53:39,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-06 06:53:39,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-06 06:53:39,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-06 06:53:39,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-06 06:53:39,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-06 06:53:39,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-06 06:53:39,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-06 06:53:39,585 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04076338519756036, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.060593769797493244, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04076338519756036, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.175, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24341666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 06:53:39,585 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-06 06:53:39,585 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1799316866916293, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04076338519756036, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.060593769797493244, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04076338519756036, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.175, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24341666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-06 06:53:39,899 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 455 ms to try 5000 different iterations.  Found a solution that moves 2597 regions; Going from a computed imbalance of 0.1799316866916293 to a new imbalance of 0.015601232022643644. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021829298410871147); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040145638347633045, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28855555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021829298410871147); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04658333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 06:53:39,912 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv939742344:75 , srv37520101:78 , srv7229803:78 , srv1548100971:80 , srv168693101:80 , srv688594592:80 , srv783833725:80 , srv1005974628:81 , srv1236285832:82 , srv1552818254:82 , srv1790866148:82 , srv2056521096:82 , srv301574390:82 , srv605896259:82 , srv761949344:82 , srv1145531912:83 , srv1324290407:83 , srv1725538067:83 , srv1759074194:83 , srv1822466838:83 , srv1968581886:83 , srv2122127519:83 , srv336248297:83 , srv345511646:83 , srv1093474111:84 , srv1245771946:84 , srv1403564880:84 , srv1570616866:84 , srv1717356931:84 , srv1729696044:84 , srv1938067709:84 , srv28434284:84 , srv464569882:84 , srv532477618:84 , srv667519050:84 , srv903409715:84 , srv1101894705:85 , srv2011027553:85 , srv226197838:85 , srv231271109:85 , srv287836745:85 , srv1142950224:86 , srv1211440066:86 , srv136384246:86 , srv1438577700:86 , srv158458918:86 , srv1634112852:86 , srv1943565271:86 , srv1994410918:86 , srv214255845:86 , srv506927175:86 , srv671332037:86 , srv74284502:86 , srv845573201:86 , srv946015722:86 , srv1155571854:87 , srv1215659110:87 , srv1253435477:87 , srv1512758595:87 , srv1674306524:87 , srv1763925074:87 , srv1838392926:87 , srv2116150834:87 , srv430902771:87 , srv467843285:87 , srv557239515:87 , srv1273538736:88 , srv1424649866:88 , srv1571785414:88 , srv1657723673:88 , srv1733243302:88 , srv1810268602:88 , srv1867059654:88 , srv1886114293:88 , srv1990379416:88 , srv2009421048:88 , srv2083793718:88 , srv232745635:88 , srv253967200:88 , srv332779991:88 , srv503511617:88 , srv558031642:88 , srv678138865:88 , srv856000960:88 , srv913805760:88 , srv920969153:88 , srv1123369348:89 , srv1221842450:89 , srv1289971554:89 , srv174337603:89 , srv868098620:89 , srv1160359875:90 , srv868895296:90 , srv979083227:90 , srv1291278902:91 , srv283196351:91 , srv229865535:92 , srv952894393:92 , srv1286429734:96 , srv432783215:516 }
2023-05-06 06:53:39,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-06 06:53:39,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv979083227=99, srv1717356931=33, srv920969153=95, srv1571785414=27, srv1968581886=48, srv1211440066=8, srv1273538736=14, srv1810268602=41, srv1938067709=46, srv678138865=83, srv558031642=79, srv174337603=37, srv432783215=72, srv345511646=69, srv688594592=84, srv7229803=85, srv1867059654=44, srv301574390=66, srv1548100971=24, srv158458918=28, srv1763925074=39, srv214255845=57, srv783833725=88, srv1943565271=47, srv74284502=86, srv1990379416=49, srv1994410918=50, srv28434284=64, srv868895296=92, srv1733243302=36, srv952894393=98, srv1552818254=25, srv845573201=89, srv1215659110=9, srv939742344=96, srv1403564880=20, srv2083793718=54, srv287836745=65, srv430902771=71, srv1674306524=31, srv253967200=62, srv1286429734=15, srv1634112852=29, srv913805760=94, srv467843285=74, srv903409715=93, srv1093474111=1, srv1160359875=7, srv503511617=75, srv2116150834=55, srv1101894705=2, srv2009421048=51, srv1005974628=0, srv667519050=81, srv1245771946=12, srv1291278902=17, srv1725538067=34, srv229865535=59, srv136384246=19, srv1438577700=22, srv332779991=67, srv1123369348=3, srv1729696044=35, srv1790866148=40, srv1570616866=26, srv168693101=32, srv37520101=70, srv761949344=87, srv1324290407=18, srv1142950224=4, srv671332037=82, srv1886114293=45, srv1145531912=5, srv1236285832=11, srv946015722=97, srv557239515=78, srv283196351=63, srv1424649866=21, srv856000960=90, srv1289971554=16, srv226197838=58, srv336248297=68, srv868098620=91, srv1838392926=43, srv605896259=80, srv1253435477=13, srv232745635=61, srv532477618=77, srv231271109=60, srv2122127519=56, srv506927175=76, srv2056521096=53, srv1759074194=38, srv1221842450=10, srv1822466838=42, srv2011027553=52, srv1657723673=30, srv464569882=73, srv1155571854=6, srv1512758595=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-06 06:53:39,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-06 06:53:39,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-06 06:53:39,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-06 06:53:39,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-06 06:53:39,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-06 06:53:39,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-06 06:53:39,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-06 06:53:39,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-06 06:53:39,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-06 06:53:39,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-06 06:53:39,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-06 06:53:39,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-06 06:53:39,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-06 06:53:39,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-06 06:53:39,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-06 06:53:39,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-06 06:53:39,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-06 06:53:39,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-06 06:53:39,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-06 06:53:39,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-06 06:53:39,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-06 06:53:39,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-06 06:53:39,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-06 06:53:39,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-06 06:53:39,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-06 06:53:39,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-06 06:53:39,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-06 06:53:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-06 06:53:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-06 06:53:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-06 06:53:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-06 06:53:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-06 06:53:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-06 06:53:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-06 06:53:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-06 06:53:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-06 06:53:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-06 06:53:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-06 06:53:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-06 06:53:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-06 06:53:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-06 06:53:39,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-06 06:53:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-06 06:53:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-06 06:53:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-06 06:53:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-06 06:53:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-06 06:53:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-06 06:53:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-06 06:53:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-06 06:53:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-06 06:53:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-06 06:53:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-06 06:53:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-06 06:53:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-06 06:53:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-06 06:53:39,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-06 06:53:39,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-06 06:53:39,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-06 06:53:39,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-06 06:53:39,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-06 06:53:39,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-06 06:53:39,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-06 06:53:39,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-06 06:53:39,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-06 06:53:39,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-06 06:53:39,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-06 06:53:39,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-06 06:53:39,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-06 06:53:39,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-06 06:53:39,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-06 06:53:39,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-06 06:53:39,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-06 06:53:39,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-06 06:53:39,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-06 06:53:39,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-06 06:53:39,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-06 06:53:39,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-06 06:53:39,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-06 06:53:39,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-06 06:53:39,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-06 06:53:39,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-06 06:53:39,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-06 06:53:39,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-06 06:53:39,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-06 06:53:39,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-06 06:53:39,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-06 06:53:39,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-06 06:53:39,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-06 06:53:39,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-06 06:53:39,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-06 06:53:39,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-06 06:53:39,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-06 06:53:39,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-06 06:53:39,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-06 06:53:39,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-06 06:53:39,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-06 06:53:39,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-06 06:53:39,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-06 06:53:39,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-06 06:53:39,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-06 06:53:40,082 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021829298410871147); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040145638347633045, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021829298410871147); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04658333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 06:53:40,083 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-06 06:53:40,083 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015583047439976764, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021829298410871147); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040145638347633045, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021829298410871147); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04658333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-06 06:53:40,407 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 436 ms to try 5000 different iterations.  Found a solution that moves 1053 regions; Going from a computed imbalance of 0.015583047439976764 to a new imbalance of 6.799831415205878E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005105203241514896); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007293740686277048); MoveCostFunction : (multiplier=7.0, imbalance=0.117, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005105203241514896); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 06:53:40,409 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1286429734:82 , srv1733243302:82 , srv232745635:82 , srv283196351:82 , srv136384246:83 , srv1634112852:83 , srv1838392926:83 , srv868098620:83 , srv1142950224:84 , srv1155571854:84 , srv1424649866:84 , srv1674306524:84 , srv174337603:84 , srv1763925074:84 , srv1790866148:84 , srv1810268602:84 , srv1968581886:84 , srv2011027553:84 , srv226197838:84 , srv231271109:84 , srv7229803:84 , srv952894393:84 , srv979083227:84 , srv1005974628:85 , srv1215659110:85 , srv1273538736:85 , srv1289971554:85 , srv1291278902:85 , srv1438577700:85 , srv1512758595:85 , srv1571785414:85 , srv168693101:85 , srv1822466838:85 , srv2009421048:85 , srv214255845:85 , srv229865535:85 , srv253967200:85 , srv345511646:85 , srv430902771:85 , srv464569882:85 , srv467843285:85 , srv532477618:85 , srv557239515:85 , srv558031642:85 , srv667519050:85 , srv74284502:85 , srv783833725:85 , srv845573201:85 , srv913805760:85 , srv1101894705:86 , srv1123369348:86 , srv1145531912:86 , srv1211440066:86 , srv1403564880:86 , srv158458918:86 , srv1717356931:86 , srv1725538067:86 , srv1990379416:86 , srv1994410918:86 , srv2083793718:86 , srv301574390:86 , srv332779991:86 , srv336248297:86 , srv37520101:86 , srv503511617:86 , srv605896259:86 , srv761949344:86 , srv856000960:86 , srv868895296:86 , srv903409715:86 , srv920969153:86 , srv939742344:86 , srv946015722:86 , srv1093474111:87 , srv1160359875:87 , srv1236285832:87 , srv1245771946:87 , srv1253435477:87 , srv1548100971:87 , srv1759074194:87 , srv1867059654:87 , srv1886114293:87 , srv1938067709:87 , srv1943565271:87 , srv2056521096:87 , srv2116150834:87 , srv2122127519:87 , srv28434284:87 , srv506927175:87 , srv678138865:87 , srv688594592:87 , srv1221842450:88 , srv1324290407:88 , srv1552818254:88 , srv1570616866:88 , srv1729696044:88 , srv287836745:88 , srv671332037:88 , srv1657723673:89 , srv432783215:535 }
2023-05-06 06:53:40,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-06 06:53:40,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv979083227=99, srv1717356931=33, srv920969153=95, srv1571785414=27, srv1968581886=48, srv1211440066=8, srv1273538736=14, srv1810268602=41, srv1938067709=46, srv678138865=83, srv558031642=79, srv174337603=37, srv432783215=72, srv345511646=69, srv688594592=84, srv7229803=85, srv1867059654=44, srv301574390=66, srv1548100971=24, srv158458918=28, srv1763925074=39, srv214255845=57, srv783833725=88, srv1943565271=47, srv74284502=86, srv1990379416=49, srv1994410918=50, srv28434284=64, srv868895296=92, srv1733243302=36, srv952894393=98, srv1552818254=25, srv845573201=89, srv1215659110=9, srv939742344=96, srv1403564880=20, srv2083793718=54, srv287836745=65, srv430902771=71, srv1674306524=31, srv253967200=62, srv1286429734=15, srv1634112852=29, srv913805760=94, srv467843285=74, srv903409715=93, srv1093474111=1, srv1160359875=7, srv503511617=75, srv2116150834=55, srv1101894705=2, srv2009421048=51, srv1005974628=0, srv667519050=81, srv1245771946=12, srv1291278902=17, srv1725538067=34, srv229865535=59, srv136384246=19, srv1438577700=22, srv332779991=67, srv1123369348=3, srv1729696044=35, srv1790866148=40, srv1570616866=26, srv168693101=32, srv37520101=70, srv761949344=87, srv1324290407=18, srv1142950224=4, srv671332037=82, srv1886114293=45, srv1145531912=5, srv1236285832=11, srv946015722=97, srv557239515=78, srv283196351=63, srv1424649866=21, srv856000960=90, srv1289971554=16, srv226197838=58, srv336248297=68, srv868098620=91, srv1838392926=43, srv605896259=80, srv1253435477=13, srv232745635=61, srv532477618=77, srv231271109=60, srv2122127519=56, srv506927175=76, srv2056521096=53, srv1759074194=38, srv1221842450=10, srv1822466838=42, srv2011027553=52, srv1657723673=30, srv464569882=73, srv1155571854=6, srv1512758595=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-06 06:53:40,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-06 06:53:40,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-06 06:53:40,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-06 06:53:40,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-06 06:53:40,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-06 06:53:40,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-06 06:53:40,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-06 06:53:40,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-06 06:53:40,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-06 06:53:40,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-06 06:53:40,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-06 06:53:40,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-06 06:53:40,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-06 06:53:40,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-06 06:53:40,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-06 06:53:40,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-06 06:53:40,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-06 06:53:40,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-06 06:53:40,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-06 06:53:40,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-06 06:53:40,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-06 06:53:40,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-06 06:53:40,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-06 06:53:40,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-06 06:53:40,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-06 06:53:40,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-06 06:53:40,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-06 06:53:40,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-06 06:53:40,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-06 06:53:40,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-06 06:53:40,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-06 06:53:40,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-06 06:53:40,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-06 06:53:40,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-06 06:53:40,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-06 06:53:40,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-06 06:53:40,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-06 06:53:40,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-06 06:53:40,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-06 06:53:40,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-06 06:53:40,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-06 06:53:40,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-06 06:53:40,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-06 06:53:40,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-06 06:53:40,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-06 06:53:40,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-06 06:53:40,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-06 06:53:40,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-06 06:53:40,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-06 06:53:40,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-06 06:53:40,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-06 06:53:40,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-06 06:53:40,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-06 06:53:40,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-06 06:53:40,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-06 06:53:40,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-06 06:53:40,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-06 06:53:40,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-06 06:53:40,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-06 06:53:40,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-06 06:53:40,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-06 06:53:40,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-06 06:53:40,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-06 06:53:40,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-06 06:53:40,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-06 06:53:40,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-06 06:53:40,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-06 06:53:40,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-06 06:53:40,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-06 06:53:40,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-06 06:53:40,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-06 06:53:40,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-06 06:53:40,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-06 06:53:40,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-06 06:53:40,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-06 06:53:40,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-06 06:53:40,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-06 06:53:40,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-06 06:53:40,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-06 06:53:40,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-06 06:53:40,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-06 06:53:40,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-06 06:53:40,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-06 06:53:40,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-06 06:53:40,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-06 06:53:40,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-06 06:53:40,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-06 06:53:40,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-06 06:53:40,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-06 06:53:40,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-06 06:53:40,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-06 06:53:40,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-06 06:53:40,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-06 06:53:40,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-06 06:53:40,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-06 06:53:40,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-06 06:53:40,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-06 06:53:40,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-06 06:53:40,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-06 06:53:40,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-06 06:53:40,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-06 06:53:40,561 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005105203241514896); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007293740686277048); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005105203241514896); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 06:53:40,561 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-06 06:53:40,561 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.72609877928665E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005105203241514896); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007293740686277048); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005105203241514896); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-06 06:53:40,987 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 542 ms to try 5000 different iterations.  Found a solution that moves 232 regions; Going from a computed imbalance of 6.72609877928665E-4 to a new imbalance of 1.9257933462402327E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.025777777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 06:53:40,988 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1324290407:82 , srv1657723673:82 , srv332779991:83 , srv557239515:83 , srv1093474111:84 , srv1123369348:84 , srv1155571854:84 , srv1160359875:84 , srv1221842450:84 , srv1236285832:84 , srv1245771946:84 , srv1253435477:84 , srv1289971554:84 , srv1291278902:84 , srv1403564880:84 , srv1570616866:84 , srv158458918:84 , srv1729696044:84 , srv1733243302:84 , srv1763925074:84 , srv1790866148:84 , srv1810268602:84 , srv1822466838:84 , srv1867059654:84 , srv1938067709:84 , srv1943565271:84 , srv1968581886:84 , srv1994410918:84 , srv2009421048:84 , srv2083793718:84 , srv2116150834:84 , srv229865535:84 , srv231271109:84 , srv253967200:84 , srv283196351:84 , srv28434284:84 , srv287836745:84 , srv37520101:84 , srv503511617:84 , srv506927175:84 , srv671332037:84 , srv678138865:84 , srv688594592:84 , srv783833725:84 , srv868098620:84 , srv868895296:84 , srv903409715:84 , srv920969153:84 , srv946015722:84 , srv952894393:84 , srv979083227:84 , srv1005974628:85 , srv1142950224:85 , srv1145531912:85 , srv1211440066:85 , srv1215659110:85 , srv1286429734:85 , srv136384246:85 , srv1424649866:85 , srv1438577700:85 , srv1512758595:85 , srv1548100971:85 , srv1552818254:85 , srv1571785414:85 , srv1634112852:85 , srv1674306524:85 , srv168693101:85 , srv1717356931:85 , srv1725538067:85 , srv174337603:85 , srv1759074194:85 , srv1838392926:85 , srv1886114293:85 , srv1990379416:85 , srv2011027553:85 , srv2056521096:85 , srv2122127519:85 , srv214255845:85 , srv226197838:85 , srv232745635:85 , srv301574390:85 , srv336248297:85 , srv345511646:85 , srv430902771:85 , srv464569882:85 , srv467843285:85 , srv532477618:85 , srv558031642:85 , srv605896259:85 , srv667519050:85 , srv7229803:85 , srv74284502:85 , srv761949344:85 , srv845573201:85 , srv913805760:85 , srv939742344:85 , srv1101894705:86 , srv1273538736:86 , srv856000960:86 , srv432783215:639 }
2023-05-06 06:53:40,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-06 06:53:40,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv979083227=99, srv1717356931=33, srv920969153=95, srv1571785414=27, srv1968581886=48, srv1211440066=8, srv1273538736=14, srv1810268602=41, srv1938067709=46, srv678138865=83, srv558031642=79, srv174337603=37, srv432783215=72, srv345511646=69, srv688594592=84, srv7229803=85, srv1867059654=44, srv301574390=66, srv1548100971=24, srv158458918=28, srv1763925074=39, srv214255845=57, srv783833725=88, srv1943565271=47, srv74284502=86, srv1990379416=49, srv1994410918=50, srv28434284=64, srv868895296=92, srv1733243302=36, srv952894393=98, srv1552818254=25, srv845573201=89, srv1215659110=9, srv939742344=96, srv1403564880=20, srv2083793718=54, srv287836745=65, srv430902771=71, srv1674306524=31, srv253967200=62, srv1286429734=15, srv1634112852=29, srv913805760=94, srv467843285=74, srv903409715=93, srv1093474111=1, srv1160359875=7, srv503511617=75, srv2116150834=55, srv1101894705=2, srv2009421048=51, srv1005974628=0, srv667519050=81, srv1245771946=12, srv1291278902=17, srv1725538067=34, srv229865535=59, srv136384246=19, srv1438577700=22, srv332779991=67, srv1123369348=3, srv1729696044=35, srv1790866148=40, srv1570616866=26, srv168693101=32, srv37520101=70, srv761949344=87, srv1324290407=18, srv1142950224=4, srv671332037=82, srv1886114293=45, srv1145531912=5, srv1236285832=11, srv946015722=97, srv557239515=78, srv283196351=63, srv1424649866=21, srv856000960=90, srv1289971554=16, srv226197838=58, srv336248297=68, srv868098620=91, srv1838392926=43, srv605896259=80, srv1253435477=13, srv232745635=61, srv532477618=77, srv231271109=60, srv2122127519=56, srv506927175=76, srv2056521096=53, srv1759074194=38, srv1221842450=10, srv1822466838=42, srv2011027553=52, srv1657723673=30, srv464569882=73, srv1155571854=6, srv1512758595=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-06 06:53:41,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-06 06:53:41,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-06 06:53:41,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-06 06:53:41,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-06 06:53:41,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-06 06:53:41,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-06 06:53:41,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-06 06:53:41,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-06 06:53:41,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-06 06:53:41,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-06 06:53:41,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-06 06:53:41,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-06 06:53:41,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-06 06:53:41,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-06 06:53:41,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-06 06:53:41,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-06 06:53:41,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-06 06:53:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-06 06:53:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-06 06:53:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-06 06:53:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-06 06:53:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-06 06:53:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-06 06:53:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-06 06:53:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-06 06:53:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-06 06:53:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-06 06:53:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-06 06:53:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-06 06:53:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-06 06:53:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-06 06:53:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-06 06:53:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-06 06:53:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-06 06:53:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-06 06:53:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-06 06:53:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-06 06:53:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-06 06:53:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-06 06:53:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-06 06:53:41,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-06 06:53:41,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-06 06:53:41,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-06 06:53:41,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-06 06:53:41,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-06 06:53:41,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-06 06:53:41,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-06 06:53:41,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-06 06:53:41,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-06 06:53:41,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-06 06:53:41,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-06 06:53:41,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-06 06:53:41,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-06 06:53:41,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-06 06:53:41,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-06 06:53:41,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-06 06:53:41,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-06 06:53:41,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-06 06:53:41,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-06 06:53:41,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-06 06:53:41,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-06 06:53:41,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-06 06:53:41,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-06 06:53:41,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-06 06:53:41,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-06 06:53:41,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-06 06:53:41,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-06 06:53:41,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-06 06:53:41,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-06 06:53:41,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-06 06:53:41,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-06 06:53:41,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-06 06:53:41,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-06 06:53:41,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-06 06:53:41,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-06 06:53:41,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-06 06:53:41,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-06 06:53:41,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-06 06:53:41,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-06 06:53:41,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-06 06:53:41,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-06 06:53:41,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-06 06:53:41,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-06 06:53:41,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-06 06:53:41,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-06 06:53:41,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-06 06:53:41,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-06 06:53:41,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-06 06:53:41,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-06 06:53:41,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-06 06:53:41,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-06 06:53:41,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-06 06:53:41,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-06 06:53:41,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-06 06:53:41,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-06 06:53:41,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-06 06:53:41,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-06 06:53:41,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-06 06:53:41,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-06 06:53:41,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-06 06:53:41,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-06 06:53:41,144 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.7633434741294947E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 06:53:41,144 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1324290407:82 , srv1657723673:82 , srv332779991:83 , srv557239515:83 , srv1093474111:84 , srv1123369348:84 , srv1155571854:84 , srv1160359875:84 , srv1221842450:84 , srv1236285832:84 , srv1245771946:84 , srv1253435477:84 , srv1289971554:84 , srv1291278902:84 , srv1403564880:84 , srv1570616866:84 , srv158458918:84 , srv1729696044:84 , srv1733243302:84 , srv1763925074:84 , srv1790866148:84 , srv1810268602:84 , srv1822466838:84 , srv1867059654:84 , srv1938067709:84 , srv1943565271:84 , srv1968581886:84 , srv1994410918:84 , srv2009421048:84 , srv2083793718:84 , srv2116150834:84 , srv229865535:84 , srv231271109:84 , srv253967200:84 , srv283196351:84 , srv28434284:84 , srv287836745:84 , srv37520101:84 , srv503511617:84 , srv506927175:84 , srv671332037:84 , srv678138865:84 , srv688594592:84 , srv783833725:84 , srv868098620:84 , srv868895296:84 , srv903409715:84 , srv920969153:84 , srv946015722:84 , srv952894393:84 , srv979083227:84 , srv1005974628:85 , srv1142950224:85 , srv1145531912:85 , srv1211440066:85 , srv1215659110:85 , srv1286429734:85 , srv136384246:85 , srv1424649866:85 , srv1438577700:85 , srv1512758595:85 , srv1548100971:85 , srv1552818254:85 , srv1571785414:85 , srv1634112852:85 , srv1674306524:85 , srv168693101:85 , srv1717356931:85 , srv1725538067:85 , srv174337603:85 , srv1759074194:85 , srv1838392926:85 , srv1886114293:85 , srv1990379416:85 , srv2011027553:85 , srv2056521096:85 , srv2122127519:85 , srv214255845:85 , srv226197838:85 , srv232745635:85 , srv301574390:85 , srv336248297:85 , srv345511646:85 , srv430902771:85 , srv464569882:85 , srv467843285:85 , srv532477618:85 , srv558031642:85 , srv605896259:85 , srv667519050:85 , srv7229803:85 , srv74284502:85 , srv761949344:85 , srv845573201:85 , srv913805760:85 , srv939742344:85 , srv1101894705:86 , srv1273538736:86 , srv856000960:86 , srv432783215:639 }
2023-05-06 06:53:41,180 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=308 (was 308), ProcessCount=170 (was 170), AvailableMemoryMB=7219 (was 7990)
2023-05-06 06:53:41,196 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=308, ProcessCount=170, AvailableMemoryMB=7218
2023-05-06 06:53:41,197 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-06 06:53:41,198 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-05-06 06:53:41,212 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1046256597:84 , srv105374550:84 , srv1129996474:84 , srv1186640289:84 , srv1186685350:84 , srv1224872539:84 , srv1258565864:84 , srv1419981828:84 , srv1497078828:84 , srv1508391430:84 , srv1517249131:84 , srv155087426:84 , srv1630684262:84 , srv1706997092:84 , srv1899145807:84 , srv2010460622:84 , srv2120049605:84 , srv261103952:84 , srv2648790:84 , srv327885913:84 , srv331524106:84 , srv379192494:84 , srv430773738:84 , srv570556329:84 , srv721294167:84 , srv804389654:84 , srv913962274:84 , srv914350595:84 , srv925572750:84 , srv1077342537:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-05-06 06:53:41,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-06 06:53:41,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv331524106=21, srv2010460622=16, srv2120049605=17, srv430773738=23, srv1517249131=11, srv1630684262=13, srv1046256597=0, srv1129996474=3, srv925572750=29, srv155087426=12, srv1706997092=14, srv379192494=22, srv1186640289=4, srv1077342537=2, srv721294167=25, srv1899145807=15, srv1419981828=8, srv914350595=28, srv804389654=26, srv1258565864=7, srv570556329=24, srv327885913=20, srv1224872539=6, srv2648790=19, srv913962274=27, srv1508391430=10, srv105374550=1, srv1186685350=5, srv1497078828=9, srv261103952=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-06 06:53:41,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-06 06:53:41,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-06 06:53:41,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-06 06:53:41,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-06 06:53:41,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-06 06:53:41,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-06 06:53:41,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-06 06:53:41,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-06 06:53:41,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-06 06:53:41,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-06 06:53:41,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-06 06:53:41,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-06 06:53:41,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-06 06:53:41,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-06 06:53:41,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-06 06:53:41,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-06 06:53:41,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-06 06:53:41,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-06 06:53:41,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-06 06:53:41,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-06 06:53:41,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-06 06:53:41,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-06 06:53:41,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-06 06:53:41,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-06 06:53:41,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-06 06:53:41,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-06 06:53:41,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-06 06:53:41,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-06 06:53:41,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-06 06:53:41,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-06 06:53:41,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-05-06 06:53:41,271 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.123949989943795, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 06:53:41,271 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912583605635253, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.123949989943795, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-05-06 06:55:41,251 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6656559 different iterations.  Found a solution that moves 1839 regions; Going from a computed imbalance of 0.9912583605635253 to a new imbalance of 4.292317741546655E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6811111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 06:55:41,259 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1046256597:90 , srv105374550:90 , srv1077342537:90 , srv1129996474:90 , srv1186640289:90 , srv1186685350:90 , srv1224872539:90 , srv1258565864:90 , srv1419981828:90 , srv1497078828:90 , srv1508391430:90 , srv1517249131:90 , srv155087426:90 , srv1630684262:90 , srv1706997092:90 , srv1899145807:90 , srv2010460622:90 , srv2120049605:90 , srv261103952:90 , srv2648790:90 , srv327885913:90 , srv331524106:90 , srv379192494:90 , srv430773738:90 , srv570556329:90 , srv721294167:90 , srv804389654:90 , srv913962274:90 , srv914350595:90 , srv925572750:90 }
2023-05-06 06:55:41,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-06 06:55:41,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv331524106=21, srv2010460622=16, srv2120049605=17, srv430773738=23, srv1517249131=11, srv1630684262=13, srv1046256597=0, srv1129996474=3, srv925572750=29, srv155087426=12, srv1706997092=14, srv379192494=22, srv1186640289=4, srv1077342537=2, srv721294167=25, srv1899145807=15, srv1419981828=8, srv914350595=28, srv804389654=26, srv1258565864=7, srv570556329=24, srv327885913=20, srv1224872539=6, srv2648790=19, srv913962274=27, srv1508391430=10, srv105374550=1, srv1186685350=5, srv1497078828=9, srv261103952=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-06 06:55:41,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-06 06:55:41,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-06 06:55:41,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-06 06:55:41,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-06 06:55:41,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-06 06:55:41,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-06 06:55:41,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-06 06:55:41,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-06 06:55:41,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-06 06:55:41,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-06 06:55:41,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-06 06:55:41,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-06 06:55:41,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-06 06:55:41,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-06 06:55:41,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-06 06:55:41,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-06 06:55:41,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-06 06:55:41,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-06 06:55:41,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-06 06:55:41,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-06 06:55:41,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-06 06:55:41,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-06 06:55:41,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-06 06:55:41,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-06 06:55:41,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-06 06:55:41,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-06 06:55:41,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-06 06:55:41,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-06 06:55:41,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-06 06:55:41,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-06 06:55:41,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-05-06 06:55:41,286 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 06:55:41,286 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1046256597:90 , srv105374550:90 , srv1077342537:90 , srv1129996474:90 , srv1186640289:90 , srv1186685350:90 , srv1224872539:90 , srv1258565864:90 , srv1419981828:90 , srv1497078828:90 , srv1508391430:90 , srv1517249131:90 , srv155087426:90 , srv1630684262:90 , srv1706997092:90 , srv1899145807:90 , srv2010460622:90 , srv2120049605:90 , srv261103952:90 , srv2648790:90 , srv327885913:90 , srv331524106:90 , srv379192494:90 , srv430773738:90 , srv570556329:90 , srv721294167:90 , srv804389654:90 , srv913962274:90 , srv914350595:90 , srv925572750:90 }
2023-05-06 06:55:41,303 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=165 (was 308), ProcessCount=170 (was 170), AvailableMemoryMB=7030 (was 7218)