2023-03-18 07:54:00,433 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-03-18 07:54:00,757 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-03-18 07:54:00,865 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-03-18 07:54:00,927 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-18 07:54:00,935 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-18 07:54:00,947 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=296, ProcessCount=169, AvailableMemoryMB=3068
2023-03-18 07:54:00,949 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-18 07:54:00,950 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-18 07:54:01,066 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1016754283:84 , srv1020711531:84 , srv1046901972:84 , srv1098870312:84 , srv1113747588:84 , srv1153196312:84 , srv1161769940:84 , srv1183240545:84 , srv1183655602:84 , srv1237212889:84 , srv1254855227:84 , srv1311792158:84 , srv1319726202:84 , srv1323444676:84 , srv1333832408:84 , srv1345033528:84 , srv1363004531:84 , srv1369216238:84 , srv1371369099:84 , srv1407047366:84 , srv1426795489:84 , srv1438658869:84 , srv1477399255:84 , srv1479251516:84 , srv1524959111:84 , srv1533967540:84 , srv1541926422:84 , srv1551562579:84 , srv1555861664:84 , srv1591757690:84 , srv1641785499:84 , srv1646991039:84 , srv1650549745:84 , srv1655649475:84 , srv1663378995:84 , srv1680811109:84 , srv1691427780:84 , srv1730685884:84 , srv1756152159:84 , srv1773283224:84 , srv1831133701:84 , srv1867793071:84 , srv1877742834:84 , srv1924023760:84 , srv19370142:84 , srv1952604103:84 , srv1957705868:84 , srv1962163691:84 , srv1972562781:84 , srv2019291305:84 , srv202777617:84 , srv2028837286:84 , srv2086774045:84 , srv2092217214:84 , srv2100860361:84 , srv2122284239:84 , srv2137615904:84 , srv2141714373:84 , srv2143266155:84 , srv219349732:84 , srv224829778:84 , srv25192434:84 , srv279626930:84 , srv288186254:84 , srv296567015:84 , srv349525960:84 , srv354416211:84 , srv355225618:84 , srv369731994:84 , srv405216210:84 , srv411856483:84 , srv419017150:84 , srv42402706:84 , srv501923053:84 , srv557903151:84 , srv577738804:84 , srv619342779:84 , srv620532406:84 , srv631258744:84 , srv638987012:84 , srv67213330:84 , srv722110888:84 , srv722963094:84 , srv732121427:84 , srv743104946:84 , srv758691574:84 , srv818482571:84 , srv847582057:84 , srv855177691:84 , srv878745592:84 , srv879265849:84 , srv880596251:84 , srv886249568:84 , srv910728910:84 , srv928386345:84 , srv976908714:84 , srv985519872:84 , srv991015851:84 , srv99844791:84 , srv144613611:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-03-18 07:54:01,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-18 07:54:01,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv722110888=82, srv99844791=99, srv2092217214=54, srv355225618=68, srv1952604103=46, srv1254855227=10, srv976908714=96, srv1831133701=41, srv219349732=60, srv279626930=63, srv369731994=69, srv1555861664=29, srv1407047366=19, srv1957705868=47, srv2122284239=56, srv1591757690=30, srv620532406=78, srv886249568=93, srv1311792158=11, srv349525960=66, srv25192434=62, srv818482571=87, srv296567015=65, srv2143266155=59, srv619342779=77, srv758691574=86, srv1867793071=42, srv631258744=79, srv1369216238=17, srv722963094=83, srv224829778=61, srv879265849=91, srv1680811109=36, srv42402706=73, srv288186254=64, srv878745592=90, srv1016754283=0, srv354416211=67, srv1046901972=2, srv1691427780=37, srv855177691=89, srv1183655602=8, srv1098870312=3, srv1477399255=23, srv1924023760=44, srv2019291305=50, srv1183240545=7, srv1756152159=39, srv67213330=81, srv1020711531=1, srv1237212889=9, srv1773283224=40, srv928386345=95, srv880596251=92, srv1479251516=24, srv638987012=80, srv910728910=94, srv2086774045=53, srv1345033528=15, srv144613611=22, srv2028837286=52, srv1371369099=18, srv1426795489=20, srv405216210=70, srv577738804=76, srv1533967540=26, srv991015851=98, srv1972562781=49, srv501923053=74, srv1323444676=13, srv1541926422=27, srv2100860361=55, srv1962163691=48, srv2137615904=57, srv847582057=88, srv1551562579=28, srv1655649475=34, srv1319726202=12, srv1438658869=21, srv1333832408=14, srv19370142=45, srv1161769940=6, srv202777617=51, srv419017150=72, srv1877742834=43, srv1641785499=31, srv985519872=97, srv1524959111=25, srv411856483=71, srv1363004531=16, srv1153196312=5, srv1650549745=33, srv1730685884=38, srv1113747588=4, srv743104946=85, srv557903151=75, srv1646991039=32, srv2141714373=58, srv732121427=84, srv1663378995=35} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-18 07:54:01,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-18 07:54:01,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-18 07:54:01,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-18 07:54:01,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-18 07:54:01,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-18 07:54:01,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-18 07:54:01,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-18 07:54:01,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-18 07:54:01,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-18 07:54:01,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-18 07:54:01,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-18 07:54:01,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-18 07:54:01,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-18 07:54:01,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-18 07:54:01,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-18 07:54:01,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-18 07:54:01,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-18 07:54:01,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-18 07:54:01,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-18 07:54:01,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-18 07:54:01,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-18 07:54:01,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-18 07:54:01,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-18 07:54:01,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-18 07:54:01,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-18 07:54:01,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-18 07:54:01,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-18 07:54:01,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-18 07:54:01,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-18 07:54:01,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-18 07:54:01,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-18 07:54:01,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-18 07:54:01,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-18 07:54:01,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-18 07:54:01,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-18 07:54:01,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-18 07:54:01,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-18 07:54:01,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-18 07:54:01,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-18 07:54:01,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-18 07:54:01,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-18 07:54:01,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-18 07:54:01,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-18 07:54:01,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-18 07:54:01,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-18 07:54:01,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-18 07:54:01,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-18 07:54:01,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-18 07:54:01,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-18 07:54:01,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-18 07:54:01,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-18 07:54:01,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-18 07:54:01,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-18 07:54:01,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-18 07:54:01,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-18 07:54:01,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-18 07:54:01,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-18 07:54:01,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-18 07:54:01,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-18 07:54:01,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-18 07:54:01,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-18 07:54:01,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-18 07:54:01,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-18 07:54:01,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-18 07:54:01,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-18 07:54:01,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-18 07:54:01,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-18 07:54:01,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-18 07:54:01,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-18 07:54:01,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-18 07:54:01,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-18 07:54:01,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-18 07:54:01,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-18 07:54:01,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-18 07:54:01,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-18 07:54:01,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-18 07:54:01,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-18 07:54:01,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-18 07:54:01,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-18 07:54:01,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-18 07:54:01,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-18 07:54:01,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-18 07:54:01,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-18 07:54:01,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-18 07:54:01,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-18 07:54:01,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-18 07:54:01,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-18 07:54:01,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-18 07:54:01,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-18 07:54:01,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-18 07:54:01,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-18 07:54:01,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-18 07:54:01,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-18 07:54:01,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-18 07:54:01,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-18 07:54:01,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-18 07:54:01,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-18 07:54:01,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-18 07:54:01,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-18 07:54:01,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-18 07:54:01,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-18 07:54:01,319 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 07:54:01,319 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-18 07:54:01,319 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-18 07:54:01,623 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 438 ms to try 5000 different iterations.  Found a solution that moves 4682 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17757235433159968. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041086989010128784, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05716261483024723, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5202222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041086989010128784, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.173, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.237, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 07:54:01,636 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1663378995:81 , srv1680811109:81 , srv279626930:81 , srv855177691:81 , srv1046901972:82 , srv1237212889:82 , srv1773283224:82 , srv288186254:82 , srv355225618:82 , srv818482571:82 , srv1311792158:83 , srv1555861664:83 , srv1756152159:83 , srv1924023760:83 , srv202777617:83 , srv354416211:83 , srv1319726202:84 , srv1407047366:84 , srv1831133701:84 , srv1962163691:84 , srv2122284239:84 , srv758691574:84 , srv878745592:84 , srv880596251:84 , srv886249568:84 , srv1098870312:85 , srv1161769940:85 , srv1183655602:85 , srv1254855227:85 , srv1323444676:85 , srv1363004531:85 , srv1551562579:85 , srv2143266155:85 , srv349525960:85 , srv405216210:85 , srv501923053:85 , srv577738804:85 , srv67213330:85 , srv732121427:85 , srv743104946:85 , srv879265849:85 , srv1016754283:86 , srv1113747588:86 , srv1153196312:86 , srv1952604103:86 , srv1957705868:86 , srv2137615904:86 , srv219349732:86 , srv411856483:86 , srv619342779:86 , srv620532406:86 , srv638987012:86 , srv1020711531:87 , srv1345033528:87 , srv1371369099:87 , srv1426795489:87 , srv1533967540:87 , srv1541926422:87 , srv1641785499:87 , srv1646991039:87 , srv2019291305:87 , srv2028837286:87 , srv2092217214:87 , srv2100860361:87 , srv2141714373:87 , srv296567015:87 , srv976908714:87 , srv991015851:87 , srv1183240545:88 , srv1650549745:88 , srv1655649475:88 , srv1972562781:88 , srv369731994:88 , srv419017150:88 , srv42402706:88 , srv557903151:88 , srv722110888:88 , srv847582057:88 , srv928386345:88 , srv1438658869:89 , srv1877742834:89 , srv2086774045:89 , srv224829778:89 , srv631258744:89 , srv722963094:89 , srv985519872:89 , srv1479251516:90 , srv1591757690:90 , srv1691427780:90 , srv1867793071:90 , srv19370142:90 , srv1333832408:91 , srv1477399255:91 , srv1730685884:91 , srv25192434:91 , srv99844791:91 , srv1369216238:92 , srv1524959111:94 , srv910728910:95 , srv144613611:455 }
2023-03-18 07:54:01,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-18 07:54:01,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv722110888=82, srv99844791=99, srv2092217214=54, srv355225618=68, srv1952604103=46, srv1254855227=10, srv976908714=96, srv1831133701=41, srv219349732=60, srv279626930=63, srv369731994=69, srv1555861664=29, srv1407047366=19, srv1957705868=47, srv2122284239=56, srv1591757690=30, srv620532406=78, srv886249568=93, srv1311792158=11, srv349525960=66, srv25192434=62, srv818482571=87, srv296567015=65, srv2143266155=59, srv619342779=77, srv758691574=86, srv1867793071=42, srv631258744=79, srv1369216238=17, srv722963094=83, srv224829778=61, srv879265849=91, srv1680811109=36, srv42402706=73, srv288186254=64, srv878745592=90, srv1016754283=0, srv354416211=67, srv1046901972=2, srv1691427780=37, srv855177691=89, srv1183655602=8, srv1098870312=3, srv1477399255=23, srv1924023760=44, srv2019291305=50, srv1183240545=7, srv1756152159=39, srv67213330=81, srv1020711531=1, srv1237212889=9, srv1773283224=40, srv928386345=95, srv880596251=92, srv1479251516=24, srv638987012=80, srv910728910=94, srv2086774045=53, srv1345033528=15, srv144613611=22, srv2028837286=52, srv1371369099=18, srv1426795489=20, srv405216210=70, srv577738804=76, srv1533967540=26, srv991015851=98, srv1972562781=49, srv501923053=74, srv1323444676=13, srv1541926422=27, srv2100860361=55, srv1962163691=48, srv2137615904=57, srv847582057=88, srv1551562579=28, srv1655649475=34, srv1319726202=12, srv1438658869=21, srv1333832408=14, srv19370142=45, srv1161769940=6, srv202777617=51, srv419017150=72, srv1877742834=43, srv1641785499=31, srv985519872=97, srv1524959111=25, srv411856483=71, srv1363004531=16, srv1153196312=5, srv1650549745=33, srv1730685884=38, srv1113747588=4, srv743104946=85, srv557903151=75, srv1646991039=32, srv2141714373=58, srv732121427=84, srv1663378995=35} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-18 07:54:01,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-18 07:54:01,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-18 07:54:01,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-18 07:54:01,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-18 07:54:01,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-18 07:54:01,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-18 07:54:01,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-18 07:54:01,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-18 07:54:01,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-18 07:54:01,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-18 07:54:01,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-18 07:54:01,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-18 07:54:01,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-18 07:54:01,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-18 07:54:01,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-18 07:54:01,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-18 07:54:01,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-18 07:54:01,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-18 07:54:01,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-18 07:54:01,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-18 07:54:01,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-18 07:54:01,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-18 07:54:01,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-18 07:54:01,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-18 07:54:01,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-18 07:54:01,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-18 07:54:01,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-18 07:54:01,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-18 07:54:01,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-18 07:54:01,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-18 07:54:01,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-18 07:54:01,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-18 07:54:01,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-18 07:54:01,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-18 07:54:01,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-18 07:54:01,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-18 07:54:01,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-18 07:54:01,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-18 07:54:01,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-18 07:54:01,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-18 07:54:01,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-18 07:54:01,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-18 07:54:01,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-18 07:54:01,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-18 07:54:01,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-18 07:54:01,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-18 07:54:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-18 07:54:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-18 07:54:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-18 07:54:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-18 07:54:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-18 07:54:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-18 07:54:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-18 07:54:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-18 07:54:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-18 07:54:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-18 07:54:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-18 07:54:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-18 07:54:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-18 07:54:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-18 07:54:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-18 07:54:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-18 07:54:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-18 07:54:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-18 07:54:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-18 07:54:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-18 07:54:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-18 07:54:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-18 07:54:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-18 07:54:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-18 07:54:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-18 07:54:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-18 07:54:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-18 07:54:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-18 07:54:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-18 07:54:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-18 07:54:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-18 07:54:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-18 07:54:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-18 07:54:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-18 07:54:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-18 07:54:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-18 07:54:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-18 07:54:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-18 07:54:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-18 07:54:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-18 07:54:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-18 07:54:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-18 07:54:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-18 07:54:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-18 07:54:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-18 07:54:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-18 07:54:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-18 07:54:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-18 07:54:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-18 07:54:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-18 07:54:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-18 07:54:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-18 07:54:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-18 07:54:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-18 07:54:01,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-18 07:54:01,790 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041086989010128784, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05716261483024723, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041086989010128784, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.173, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.237, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 07:54:01,790 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-18 07:54:01,791 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17753957026689182, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041086989010128784, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05716261483024723, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041086989010128784, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.173, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.237, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-18 07:54:02,084 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 401 ms to try 5000 different iterations.  Found a solution that moves 2667 regions; Going from a computed imbalance of 0.17753957026689182 to a new imbalance of 0.014786438061320037. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02160824088914879); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030930209989769277, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29633333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02160824088914879); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042166666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 07:54:02,093 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1524959111:77 , srv1369216238:80 , srv1333832408:81 , srv1479251516:81 , srv1591757690:81 , srv1730685884:81 , srv19370142:81 , srv224829778:81 , srv910728910:81 , srv1183240545:82 , srv1477399255:82 , srv25192434:82 , srv631258744:82 , srv99844791:82 , srv1020711531:83 , srv1438658869:83 , srv1541926422:83 , srv1655649475:83 , srv1867793071:83 , srv2086774045:83 , srv557903151:83 , srv722963094:83 , srv1345033528:84 , srv1533967540:84 , srv1641785499:84 , srv2019291305:84 , srv42402706:84 , srv620532406:84 , srv722110888:84 , srv847582057:84 , srv928386345:84 , srv1016754283:85 , srv1113747588:85 , srv1426795489:85 , srv1650549745:85 , srv1877742834:85 , srv1972562781:85 , srv2092217214:85 , srv2100860361:85 , srv2141714373:85 , srv219349732:85 , srv296567015:85 , srv349525960:85 , srv369731994:85 , srv976908714:85 , srv985519872:85 , srv1254855227:86 , srv1371369099:86 , srv1551562579:86 , srv1646991039:86 , srv2028837286:86 , srv2137615904:86 , srv411856483:86 , srv419017150:86 , srv743104946:86 , srv1161769940:87 , srv1311792158:87 , srv1319726202:87 , srv1407047366:87 , srv1831133701:87 , srv1924023760:87 , srv1957705868:87 , srv2122284239:87 , srv2143266155:87 , srv405216210:87 , srv501923053:87 , srv577738804:87 , srv619342779:87 , srv67213330:87 , srv732121427:87 , srv879265849:87 , srv880596251:87 , srv886249568:87 , srv1046901972:88 , srv1098870312:88 , srv1183655602:88 , srv1363004531:88 , srv1952604103:88 , srv1962163691:88 , srv202777617:88 , srv355225618:88 , srv638987012:88 , srv878745592:88 , srv1153196312:89 , srv1323444676:89 , srv1555861664:89 , srv1691427780:89 , srv1756152159:89 , srv1773283224:89 , srv288186254:89 , srv991015851:89 , srv758691574:90 , srv818482571:90 , srv1237212889:91 , srv1680811109:91 , srv279626930:91 , srv354416211:91 , srv1663378995:92 , srv855177691:92 , srv144613611:511 }
2023-03-18 07:54:02,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-18 07:54:02,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv722110888=82, srv99844791=99, srv2092217214=54, srv355225618=68, srv1952604103=46, srv1254855227=10, srv976908714=96, srv1831133701=41, srv219349732=60, srv279626930=63, srv369731994=69, srv1555861664=29, srv1407047366=19, srv1957705868=47, srv2122284239=56, srv1591757690=30, srv620532406=78, srv886249568=93, srv1311792158=11, srv349525960=66, srv25192434=62, srv818482571=87, srv296567015=65, srv2143266155=59, srv619342779=77, srv758691574=86, srv1867793071=42, srv631258744=79, srv1369216238=17, srv722963094=83, srv224829778=61, srv879265849=91, srv1680811109=36, srv42402706=73, srv288186254=64, srv878745592=90, srv1016754283=0, srv354416211=67, srv1046901972=2, srv1691427780=37, srv855177691=89, srv1183655602=8, srv1098870312=3, srv1477399255=23, srv1924023760=44, srv2019291305=50, srv1183240545=7, srv1756152159=39, srv67213330=81, srv1020711531=1, srv1237212889=9, srv1773283224=40, srv928386345=95, srv880596251=92, srv1479251516=24, srv638987012=80, srv910728910=94, srv2086774045=53, srv1345033528=15, srv144613611=22, srv2028837286=52, srv1371369099=18, srv1426795489=20, srv405216210=70, srv577738804=76, srv1533967540=26, srv991015851=98, srv1972562781=49, srv501923053=74, srv1323444676=13, srv1541926422=27, srv2100860361=55, srv1962163691=48, srv2137615904=57, srv847582057=88, srv1551562579=28, srv1655649475=34, srv1319726202=12, srv1438658869=21, srv1333832408=14, srv19370142=45, srv1161769940=6, srv202777617=51, srv419017150=72, srv1877742834=43, srv1641785499=31, srv985519872=97, srv1524959111=25, srv411856483=71, srv1363004531=16, srv1153196312=5, srv1650549745=33, srv1730685884=38, srv1113747588=4, srv743104946=85, srv557903151=75, srv1646991039=32, srv2141714373=58, srv732121427=84, srv1663378995=35} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-18 07:54:02,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-18 07:54:02,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-18 07:54:02,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-18 07:54:02,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-18 07:54:02,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-18 07:54:02,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-18 07:54:02,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-18 07:54:02,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-18 07:54:02,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-18 07:54:02,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-18 07:54:02,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-18 07:54:02,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-18 07:54:02,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-18 07:54:02,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-18 07:54:02,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-18 07:54:02,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-18 07:54:02,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-18 07:54:02,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-18 07:54:02,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-18 07:54:02,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-18 07:54:02,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-18 07:54:02,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-18 07:54:02,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-18 07:54:02,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-18 07:54:02,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-18 07:54:02,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-18 07:54:02,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-18 07:54:02,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-18 07:54:02,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-18 07:54:02,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-18 07:54:02,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-18 07:54:02,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-18 07:54:02,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-18 07:54:02,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-18 07:54:02,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-18 07:54:02,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-18 07:54:02,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-18 07:54:02,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-18 07:54:02,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-18 07:54:02,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-18 07:54:02,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-18 07:54:02,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-18 07:54:02,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-18 07:54:02,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-18 07:54:02,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-18 07:54:02,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-18 07:54:02,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-18 07:54:02,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-18 07:54:02,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-18 07:54:02,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-18 07:54:02,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-18 07:54:02,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-18 07:54:02,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-18 07:54:02,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-18 07:54:02,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-18 07:54:02,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-18 07:54:02,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-18 07:54:02,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-18 07:54:02,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-18 07:54:02,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-18 07:54:02,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-18 07:54:02,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-18 07:54:02,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-18 07:54:02,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-18 07:54:02,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-18 07:54:02,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-18 07:54:02,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-18 07:54:02,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-18 07:54:02,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-18 07:54:02,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-18 07:54:02,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-18 07:54:02,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-18 07:54:02,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-18 07:54:02,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-18 07:54:02,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-18 07:54:02,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-18 07:54:02,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-18 07:54:02,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-18 07:54:02,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-18 07:54:02,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-18 07:54:02,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-18 07:54:02,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-18 07:54:02,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-18 07:54:02,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-18 07:54:02,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-18 07:54:02,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-18 07:54:02,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-18 07:54:02,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-18 07:54:02,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-18 07:54:02,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-18 07:54:02,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-18 07:54:02,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-18 07:54:02,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-18 07:54:02,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-18 07:54:02,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-18 07:54:02,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-18 07:54:02,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-18 07:54:02,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-18 07:54:02,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-18 07:54:02,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-18 07:54:02,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-18 07:54:02,223 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02160824088914879); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030930209989769277, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02160824088914879); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042166666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 07:54:02,223 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-18 07:54:02,223 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014767763328176964, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02160824088914879); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030930209989769277, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02160824088914879); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042166666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-18 07:54:02,485 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 359 ms to try 5000 different iterations.  Found a solution that moves 948 regions; Going from a computed imbalance of 0.014767763328176964 to a new imbalance of 6.000792711560348E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0036528495232634388); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004593461414505888); MoveCostFunction : (multiplier=7.0, imbalance=0.10533333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0036528495232634388); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 07:54:02,487 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1691427780:79 , srv910728910:81 , srv991015851:81 , srv1153196312:82 , srv1323444676:83 , srv1952604103:83 , srv638987012:83 , srv1183655602:84 , srv1646991039:84 , srv1877742834:84 , srv25192434:84 , srv354416211:84 , srv419017150:84 , srv758691574:84 , srv985519872:84 , srv99844791:84 , srv1098870312:85 , srv1161769940:85 , srv1237212889:85 , srv1363004531:85 , srv1369216238:85 , srv1371369099:85 , srv1438658869:85 , srv1477399255:85 , srv1663378995:85 , srv1680811109:85 , srv1730685884:85 , srv1756152159:85 , srv1867793071:85 , srv2028837286:85 , srv2143266155:85 , srv296567015:85 , srv369731994:85 , srv405216210:85 , srv501923053:85 , srv619342779:85 , srv722110888:85 , srv732121427:85 , srv847582057:85 , srv855177691:85 , srv878745592:85 , srv1016754283:86 , srv1113747588:86 , srv1254855227:86 , srv1333832408:86 , srv1426795489:86 , srv1479251516:86 , srv1524959111:86 , srv1533967540:86 , srv1555861664:86 , srv1591757690:86 , srv1650549745:86 , srv1655649475:86 , srv1773283224:86 , srv1957705868:86 , srv1962163691:86 , srv1972562781:86 , srv2019291305:86 , srv202777617:86 , srv2086774045:86 , srv2092217214:86 , srv2100860361:86 , srv2122284239:86 , srv2137615904:86 , srv2141714373:86 , srv219349732:86 , srv279626930:86 , srv411856483:86 , srv42402706:86 , srv577738804:86 , srv67213330:86 , srv722963094:86 , srv743104946:86 , srv818482571:86 , srv928386345:86 , srv976908714:86 , srv1046901972:87 , srv1319726202:87 , srv1345033528:87 , srv1407047366:87 , srv1551562579:87 , srv1641785499:87 , srv1831133701:87 , srv19370142:87 , srv557903151:87 , srv631258744:87 , srv879265849:87 , srv880596251:87 , srv886249568:87 , srv1020711531:88 , srv1183240545:88 , srv1311792158:88 , srv1541926422:88 , srv224829778:88 , srv288186254:88 , srv355225618:88 , srv620532406:88 , srv1924023760:89 , srv349525960:89 , srv144613611:524 }
2023-03-18 07:54:02,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-18 07:54:02,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv722110888=82, srv99844791=99, srv2092217214=54, srv355225618=68, srv1952604103=46, srv1254855227=10, srv976908714=96, srv1831133701=41, srv219349732=60, srv279626930=63, srv369731994=69, srv1555861664=29, srv1407047366=19, srv1957705868=47, srv2122284239=56, srv1591757690=30, srv620532406=78, srv886249568=93, srv1311792158=11, srv349525960=66, srv25192434=62, srv818482571=87, srv296567015=65, srv2143266155=59, srv619342779=77, srv758691574=86, srv1867793071=42, srv631258744=79, srv1369216238=17, srv722963094=83, srv224829778=61, srv879265849=91, srv1680811109=36, srv42402706=73, srv288186254=64, srv878745592=90, srv1016754283=0, srv354416211=67, srv1046901972=2, srv1691427780=37, srv855177691=89, srv1183655602=8, srv1098870312=3, srv1477399255=23, srv1924023760=44, srv2019291305=50, srv1183240545=7, srv1756152159=39, srv67213330=81, srv1020711531=1, srv1237212889=9, srv1773283224=40, srv928386345=95, srv880596251=92, srv1479251516=24, srv638987012=80, srv910728910=94, srv2086774045=53, srv1345033528=15, srv144613611=22, srv2028837286=52, srv1371369099=18, srv1426795489=20, srv405216210=70, srv577738804=76, srv1533967540=26, srv991015851=98, srv1972562781=49, srv501923053=74, srv1323444676=13, srv1541926422=27, srv2100860361=55, srv1962163691=48, srv2137615904=57, srv847582057=88, srv1551562579=28, srv1655649475=34, srv1319726202=12, srv1438658869=21, srv1333832408=14, srv19370142=45, srv1161769940=6, srv202777617=51, srv419017150=72, srv1877742834=43, srv1641785499=31, srv985519872=97, srv1524959111=25, srv411856483=71, srv1363004531=16, srv1153196312=5, srv1650549745=33, srv1730685884=38, srv1113747588=4, srv743104946=85, srv557903151=75, srv1646991039=32, srv2141714373=58, srv732121427=84, srv1663378995=35} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-18 07:54:02,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-18 07:54:02,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-18 07:54:02,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-18 07:54:02,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-18 07:54:02,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-18 07:54:02,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-18 07:54:02,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-18 07:54:02,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-18 07:54:02,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-18 07:54:02,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-18 07:54:02,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-18 07:54:02,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-18 07:54:02,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-18 07:54:02,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-18 07:54:02,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-18 07:54:02,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-18 07:54:02,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-18 07:54:02,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-18 07:54:02,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-18 07:54:02,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-18 07:54:02,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-18 07:54:02,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-18 07:54:02,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-18 07:54:02,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-18 07:54:02,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-18 07:54:02,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-18 07:54:02,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-18 07:54:02,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-18 07:54:02,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-18 07:54:02,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-18 07:54:02,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-18 07:54:02,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-18 07:54:02,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-18 07:54:02,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-18 07:54:02,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-18 07:54:02,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-18 07:54:02,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-18 07:54:02,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-18 07:54:02,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-18 07:54:02,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-18 07:54:02,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-18 07:54:02,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-18 07:54:02,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-18 07:54:02,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-18 07:54:02,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-18 07:54:02,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-18 07:54:02,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-18 07:54:02,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-18 07:54:02,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-18 07:54:02,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-18 07:54:02,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-18 07:54:02,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-18 07:54:02,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-18 07:54:02,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-18 07:54:02,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-18 07:54:02,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-18 07:54:02,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-18 07:54:02,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-18 07:54:02,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-18 07:54:02,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-18 07:54:02,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-18 07:54:02,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-18 07:54:02,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-18 07:54:02,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-18 07:54:02,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-18 07:54:02,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-18 07:54:02,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-18 07:54:02,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-18 07:54:02,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-18 07:54:02,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-18 07:54:02,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-18 07:54:02,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-18 07:54:02,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-18 07:54:02,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-18 07:54:02,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-18 07:54:02,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-18 07:54:02,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-18 07:54:02,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-18 07:54:02,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-18 07:54:02,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-18 07:54:02,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-18 07:54:02,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-18 07:54:02,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-18 07:54:02,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-18 07:54:02,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-18 07:54:02,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-18 07:54:02,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-18 07:54:02,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-18 07:54:02,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-18 07:54:02,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-18 07:54:02,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-18 07:54:02,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-18 07:54:02,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-18 07:54:02,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-18 07:54:02,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-18 07:54:02,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-18 07:54:02,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-18 07:54:02,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-18 07:54:02,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-18 07:54:02,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-18 07:54:02,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-18 07:54:02,644 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0036528495232634388); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004593461414505888); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0036528495232634388); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 07:54:02,644 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-18 07:54:02,645 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.934412332784064E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0036528495232634388); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004593461414505888); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0036528495232634388); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-18 07:54:02,941 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 410 ms to try 5000 different iterations.  Found a solution that moves 203 regions; Going from a computed imbalance of 5.934412332784064E-4 to a new imbalance of 2.543294801637796E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=2.735366466207394E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.022555555555555554); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=2.735366466207394E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 07:54:02,942 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1650549745:83 , srv1867793071:83 , srv1924023760:83 , srv19370142:83 , srv1957705868:83 , srv1972562781:83 , srv2092217214:83 , srv288186254:83 , srv349525960:83 , srv879265849:83 , srv1020711531:84 , srv1098870312:84 , srv1183240545:84 , srv1237212889:84 , srv1311792158:84 , srv1319726202:84 , srv1333832408:84 , srv1345033528:84 , srv1363004531:84 , srv1371369099:84 , srv1407047366:84 , srv1426795489:84 , srv1477399255:84 , srv1541926422:84 , srv1551562579:84 , srv1555861664:84 , srv1641785499:84 , srv1663378995:84 , srv1691427780:84 , srv1831133701:84 , srv1877742834:84 , srv1962163691:84 , srv2028837286:84 , srv2086774045:84 , srv2100860361:84 , srv2137615904:84 , srv2141714373:84 , srv224829778:84 , srv279626930:84 , srv354416211:84 , srv355225618:84 , srv369731994:84 , srv411856483:84 , srv557903151:84 , srv577738804:84 , srv619342779:84 , srv620532406:84 , srv631258744:84 , srv67213330:84 , srv722963094:84 , srv758691574:84 , srv818482571:84 , srv855177691:84 , srv880596251:84 , srv886249568:84 , srv928386345:84 , srv976908714:84 , srv985519872:84 , srv1016754283:85 , srv1046901972:85 , srv1113747588:85 , srv1153196312:85 , srv1161769940:85 , srv1183655602:85 , srv1254855227:85 , srv1323444676:85 , srv1369216238:85 , srv1438658869:85 , srv1479251516:85 , srv1524959111:85 , srv1533967540:85 , srv1591757690:85 , srv1646991039:85 , srv1655649475:85 , srv1680811109:85 , srv1730685884:85 , srv1756152159:85 , srv1773283224:85 , srv1952604103:85 , srv2019291305:85 , srv202777617:85 , srv2122284239:85 , srv2143266155:85 , srv219349732:85 , srv25192434:85 , srv296567015:85 , srv405216210:85 , srv419017150:85 , srv42402706:85 , srv501923053:85 , srv638987012:85 , srv722110888:85 , srv732121427:85 , srv743104946:85 , srv847582057:85 , srv878745592:85 , srv910728910:85 , srv991015851:85 , srv99844791:86 , srv144613611:652 }
2023-03-18 07:54:02,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-18 07:54:02,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv722110888=82, srv99844791=99, srv2092217214=54, srv355225618=68, srv1952604103=46, srv1254855227=10, srv976908714=96, srv1831133701=41, srv219349732=60, srv279626930=63, srv369731994=69, srv1555861664=29, srv1407047366=19, srv1957705868=47, srv2122284239=56, srv1591757690=30, srv620532406=78, srv886249568=93, srv1311792158=11, srv349525960=66, srv25192434=62, srv818482571=87, srv296567015=65, srv2143266155=59, srv619342779=77, srv758691574=86, srv1867793071=42, srv631258744=79, srv1369216238=17, srv722963094=83, srv224829778=61, srv879265849=91, srv1680811109=36, srv42402706=73, srv288186254=64, srv878745592=90, srv1016754283=0, srv354416211=67, srv1046901972=2, srv1691427780=37, srv855177691=89, srv1183655602=8, srv1098870312=3, srv1477399255=23, srv1924023760=44, srv2019291305=50, srv1183240545=7, srv1756152159=39, srv67213330=81, srv1020711531=1, srv1237212889=9, srv1773283224=40, srv928386345=95, srv880596251=92, srv1479251516=24, srv638987012=80, srv910728910=94, srv2086774045=53, srv1345033528=15, srv144613611=22, srv2028837286=52, srv1371369099=18, srv1426795489=20, srv405216210=70, srv577738804=76, srv1533967540=26, srv991015851=98, srv1972562781=49, srv501923053=74, srv1323444676=13, srv1541926422=27, srv2100860361=55, srv1962163691=48, srv2137615904=57, srv847582057=88, srv1551562579=28, srv1655649475=34, srv1319726202=12, srv1438658869=21, srv1333832408=14, srv19370142=45, srv1161769940=6, srv202777617=51, srv419017150=72, srv1877742834=43, srv1641785499=31, srv985519872=97, srv1524959111=25, srv411856483=71, srv1363004531=16, srv1153196312=5, srv1650549745=33, srv1730685884=38, srv1113747588=4, srv743104946=85, srv557903151=75, srv1646991039=32, srv2141714373=58, srv732121427=84, srv1663378995=35} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-18 07:54:02,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-18 07:54:02,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-18 07:54:02,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-18 07:54:02,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-18 07:54:02,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-18 07:54:02,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-18 07:54:02,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-18 07:54:02,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-18 07:54:02,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-18 07:54:02,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-18 07:54:02,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-18 07:54:02,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-18 07:54:02,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-18 07:54:02,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-18 07:54:02,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-18 07:54:02,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-18 07:54:02,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-18 07:54:02,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-18 07:54:02,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-18 07:54:02,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-18 07:54:02,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-18 07:54:02,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-18 07:54:02,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-18 07:54:02,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-18 07:54:02,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-18 07:54:02,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-18 07:54:02,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-18 07:54:02,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-18 07:54:02,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-18 07:54:02,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-18 07:54:02,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-18 07:54:02,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-18 07:54:02,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-18 07:54:02,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-18 07:54:02,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-18 07:54:02,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-18 07:54:02,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-18 07:54:02,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-18 07:54:02,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-18 07:54:02,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-18 07:54:02,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-18 07:54:02,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-18 07:54:02,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-18 07:54:02,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-18 07:54:02,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-18 07:54:02,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-18 07:54:02,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-18 07:54:02,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-18 07:54:02,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-18 07:54:02,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-18 07:54:02,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-18 07:54:02,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-18 07:54:02,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-18 07:54:02,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-18 07:54:02,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-18 07:54:02,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-18 07:54:02,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-18 07:54:02,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-18 07:54:02,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-18 07:54:02,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-18 07:54:02,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-18 07:54:02,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-18 07:54:02,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-18 07:54:02,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-18 07:54:02,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-18 07:54:02,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-18 07:54:02,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-18 07:54:02,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-18 07:54:02,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-18 07:54:02,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-18 07:54:02,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-18 07:54:02,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-18 07:54:02,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-18 07:54:02,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-18 07:54:02,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-18 07:54:02,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-18 07:54:02,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-18 07:54:02,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-18 07:54:02,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-18 07:54:02,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-18 07:54:02,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-18 07:54:02,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-18 07:54:02,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-18 07:54:02,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-18 07:54:02,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-18 07:54:02,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-18 07:54:02,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-18 07:54:02,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-18 07:54:02,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-18 07:54:02,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-18 07:54:02,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-18 07:54:02,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-18 07:54:02,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-18 07:54:02,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-18 07:54:02,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-18 07:54:02,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-18 07:54:02,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-18 07:54:02,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-18 07:54:02,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-18 07:54:02,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-18 07:54:02,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-18 07:54:03,079 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=2.735366466207394E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=2.735366466207394E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 07:54:03,079 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-18 07:54:03,079 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.4011511635409005E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=2.735366466207394E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=2.735366466207394E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-18 07:54:03,313 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 330 ms to try 5000 different iterations.  Found a solution that moves 11 regions; Going from a computed imbalance of 2.4011511635409005E-5 to a new imbalance of 7.46478114657196E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0012222222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 07:54:03,313 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv419017150:83 , srv42402706:83 , srv99844791:83 , srv1016754283:84 , srv1020711531:84 , srv1046901972:84 , srv1098870312:84 , srv1113747588:84 , srv1153196312:84 , srv1161769940:84 , srv1183240545:84 , srv1183655602:84 , srv1237212889:84 , srv1254855227:84 , srv1311792158:84 , srv1319726202:84 , srv1323444676:84 , srv1333832408:84 , srv1345033528:84 , srv1363004531:84 , srv1369216238:84 , srv1371369099:84 , srv1407047366:84 , srv1426795489:84 , srv1438658869:84 , srv1477399255:84 , srv1479251516:84 , srv1524959111:84 , srv1533967540:84 , srv1541926422:84 , srv1551562579:84 , srv1555861664:84 , srv1591757690:84 , srv1641785499:84 , srv1646991039:84 , srv1650549745:84 , srv1655649475:84 , srv1663378995:84 , srv1680811109:84 , srv1691427780:84 , srv1730685884:84 , srv1756152159:84 , srv1773283224:84 , srv1831133701:84 , srv1877742834:84 , srv1924023760:84 , srv1952604103:84 , srv1957705868:84 , srv1962163691:84 , srv1972562781:84 , srv2019291305:84 , srv202777617:84 , srv2028837286:84 , srv2086774045:84 , srv2100860361:84 , srv2122284239:84 , srv2137615904:84 , srv2141714373:84 , srv2143266155:84 , srv219349732:84 , srv224829778:84 , srv25192434:84 , srv279626930:84 , srv288186254:84 , srv296567015:84 , srv349525960:84 , srv354416211:84 , srv355225618:84 , srv369731994:84 , srv405216210:84 , srv411856483:84 , srv501923053:84 , srv557903151:84 , srv577738804:84 , srv619342779:84 , srv620532406:84 , srv631258744:84 , srv638987012:84 , srv67213330:84 , srv722110888:84 , srv722963094:84 , srv732121427:84 , srv743104946:84 , srv758691574:84 , srv818482571:84 , srv847582057:84 , srv855177691:84 , srv878745592:84 , srv879265849:84 , srv880596251:84 , srv886249568:84 , srv910728910:84 , srv928386345:84 , srv976908714:84 , srv985519872:84 , srv991015851:84 , srv1867793071:85 , srv19370142:85 , srv2092217214:85 , srv144613611:684 }
2023-03-18 07:54:03,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-18 07:54:03,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv722110888=82, srv99844791=99, srv2092217214=54, srv355225618=68, srv1952604103=46, srv1254855227=10, srv976908714=96, srv1831133701=41, srv219349732=60, srv279626930=63, srv369731994=69, srv1555861664=29, srv1407047366=19, srv1957705868=47, srv2122284239=56, srv1591757690=30, srv620532406=78, srv886249568=93, srv1311792158=11, srv349525960=66, srv25192434=62, srv818482571=87, srv296567015=65, srv2143266155=59, srv619342779=77, srv758691574=86, srv1867793071=42, srv631258744=79, srv1369216238=17, srv722963094=83, srv224829778=61, srv879265849=91, srv1680811109=36, srv42402706=73, srv288186254=64, srv878745592=90, srv1016754283=0, srv354416211=67, srv1046901972=2, srv1691427780=37, srv855177691=89, srv1183655602=8, srv1098870312=3, srv1477399255=23, srv1924023760=44, srv2019291305=50, srv1183240545=7, srv1756152159=39, srv67213330=81, srv1020711531=1, srv1237212889=9, srv1773283224=40, srv928386345=95, srv880596251=92, srv1479251516=24, srv638987012=80, srv910728910=94, srv2086774045=53, srv1345033528=15, srv144613611=22, srv2028837286=52, srv1371369099=18, srv1426795489=20, srv405216210=70, srv577738804=76, srv1533967540=26, srv991015851=98, srv1972562781=49, srv501923053=74, srv1323444676=13, srv1541926422=27, srv2100860361=55, srv1962163691=48, srv2137615904=57, srv847582057=88, srv1551562579=28, srv1655649475=34, srv1319726202=12, srv1438658869=21, srv1333832408=14, srv19370142=45, srv1161769940=6, srv202777617=51, srv419017150=72, srv1877742834=43, srv1641785499=31, srv985519872=97, srv1524959111=25, srv411856483=71, srv1363004531=16, srv1153196312=5, srv1650549745=33, srv1730685884=38, srv1113747588=4, srv743104946=85, srv557903151=75, srv1646991039=32, srv2141714373=58, srv732121427=84, srv1663378995=35} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-18 07:54:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-18 07:54:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-18 07:54:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-18 07:54:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-18 07:54:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-18 07:54:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-18 07:54:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-18 07:54:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-18 07:54:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-18 07:54:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-18 07:54:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-18 07:54:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-18 07:54:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-18 07:54:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-18 07:54:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-18 07:54:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-18 07:54:03,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-18 07:54:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-18 07:54:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-18 07:54:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-18 07:54:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-18 07:54:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-18 07:54:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-18 07:54:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-18 07:54:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-18 07:54:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-18 07:54:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-18 07:54:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-18 07:54:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-18 07:54:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-18 07:54:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-18 07:54:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-18 07:54:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-18 07:54:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-18 07:54:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-18 07:54:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-18 07:54:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-18 07:54:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-18 07:54:03,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-18 07:54:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-18 07:54:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-18 07:54:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-18 07:54:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-18 07:54:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-18 07:54:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-18 07:54:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-18 07:54:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-18 07:54:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-18 07:54:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-18 07:54:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-18 07:54:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-18 07:54:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-18 07:54:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-18 07:54:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-18 07:54:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-18 07:54:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-18 07:54:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-18 07:54:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-18 07:54:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-18 07:54:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-18 07:54:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-18 07:54:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-18 07:54:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-18 07:54:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-18 07:54:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-18 07:54:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-18 07:54:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-18 07:54:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-18 07:54:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-18 07:54:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-18 07:54:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-18 07:54:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-18 07:54:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-18 07:54:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-18 07:54:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-18 07:54:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-18 07:54:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-18 07:54:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-18 07:54:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-18 07:54:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-18 07:54:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-18 07:54:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-18 07:54:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-18 07:54:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-18 07:54:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-18 07:54:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-18 07:54:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-18 07:54:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-18 07:54:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-18 07:54:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-18 07:54:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-18 07:54:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-18 07:54:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-18 07:54:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-18 07:54:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-18 07:54:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-18 07:54:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-18 07:54:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-18 07:54:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-18 07:54:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-18 07:54:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-18 07:54:03,446 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 07:54:03,446 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv419017150:83 , srv42402706:83 , srv99844791:83 , srv1016754283:84 , srv1020711531:84 , srv1046901972:84 , srv1098870312:84 , srv1113747588:84 , srv1153196312:84 , srv1161769940:84 , srv1183240545:84 , srv1183655602:84 , srv1237212889:84 , srv1254855227:84 , srv1311792158:84 , srv1319726202:84 , srv1323444676:84 , srv1333832408:84 , srv1345033528:84 , srv1363004531:84 , srv1369216238:84 , srv1371369099:84 , srv1407047366:84 , srv1426795489:84 , srv1438658869:84 , srv1477399255:84 , srv1479251516:84 , srv1524959111:84 , srv1533967540:84 , srv1541926422:84 , srv1551562579:84 , srv1555861664:84 , srv1591757690:84 , srv1641785499:84 , srv1646991039:84 , srv1650549745:84 , srv1655649475:84 , srv1663378995:84 , srv1680811109:84 , srv1691427780:84 , srv1730685884:84 , srv1756152159:84 , srv1773283224:84 , srv1831133701:84 , srv1877742834:84 , srv1924023760:84 , srv1952604103:84 , srv1957705868:84 , srv1962163691:84 , srv1972562781:84 , srv2019291305:84 , srv202777617:84 , srv2028837286:84 , srv2086774045:84 , srv2100860361:84 , srv2122284239:84 , srv2137615904:84 , srv2141714373:84 , srv2143266155:84 , srv219349732:84 , srv224829778:84 , srv25192434:84 , srv279626930:84 , srv288186254:84 , srv296567015:84 , srv349525960:84 , srv354416211:84 , srv355225618:84 , srv369731994:84 , srv405216210:84 , srv411856483:84 , srv501923053:84 , srv557903151:84 , srv577738804:84 , srv619342779:84 , srv620532406:84 , srv631258744:84 , srv638987012:84 , srv67213330:84 , srv722110888:84 , srv722963094:84 , srv732121427:84 , srv743104946:84 , srv758691574:84 , srv818482571:84 , srv847582057:84 , srv855177691:84 , srv878745592:84 , srv879265849:84 , srv880596251:84 , srv886249568:84 , srv910728910:84 , srv928386345:84 , srv976908714:84 , srv985519872:84 , srv991015851:84 , srv1867793071:85 , srv19370142:85 , srv2092217214:85 , srv144613611:684 }
2023-03-18 07:54:03,474 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=296 (was 296), ProcessCount=169 (was 169), AvailableMemoryMB=2559 (was 3068)
2023-03-18 07:54:03,484 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=296, ProcessCount=169, AvailableMemoryMB=2566
2023-03-18 07:54:03,484 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-18 07:54:03,485 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-03-18 07:54:03,493 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv105608302:84 , srv1096722228:84 , srv1097472526:84 , srv1263282002:84 , srv1265303600:84 , srv1268924819:84 , srv1360542209:84 , srv1370961572:84 , srv1473911565:84 , srv1515034203:84 , srv1676395629:84 , srv1711533868:84 , srv1762326155:84 , srv1795943190:84 , srv2016040570:84 , srv2145107164:84 , srv222991989:84 , srv22847496:84 , srv383151204:84 , srv38496683:84 , srv404702025:84 , srv420795701:84 , srv527691482:84 , srv536256960:84 , srv559387272:84 , srv604617674:84 , srv613308916:84 , srv688428038:84 , srv863596881:84 , srv1817978931:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-03-18 07:54:03,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-18 07:54:03,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv383151204=19, srv420795701=22, srv404702025=21, srv604617674=26, srv1097472526=2, srv105608302=0, srv1473911565=8, srv1268924819=5, srv1360542209=6, srv22847496=18, srv688428038=28, srv559387272=25, srv1676395629=10, srv1817978931=14, srv1795943190=13, srv1711533868=11, srv2016040570=15, srv222991989=17, srv1265303600=4, srv1515034203=9, srv536256960=24, srv1762326155=12, srv613308916=27, srv1263282002=3, srv38496683=20, srv2145107164=16, srv527691482=23, srv1370961572=7, srv1096722228=1, srv863596881=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-18 07:54:03,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-18 07:54:03,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-18 07:54:03,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-18 07:54:03,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-18 07:54:03,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-18 07:54:03,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-18 07:54:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-18 07:54:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-18 07:54:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-18 07:54:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-18 07:54:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-18 07:54:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-18 07:54:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-18 07:54:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-18 07:54:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-18 07:54:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-18 07:54:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-18 07:54:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-18 07:54:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-18 07:54:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-18 07:54:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-18 07:54:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-18 07:54:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-18 07:54:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-18 07:54:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-18 07:54:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-18 07:54:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-18 07:54:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-18 07:54:03,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-18 07:54:03,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-18 07:54:03,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-18 07:54:03,518 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1264401916562035, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 07:54:03,518 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912591452179536, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1264401916562035, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-03-18 07:56:03,507 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6347353 different iterations.  Found a solution that moves 1843 regions; Going from a computed imbalance of 0.9912591452179536 to a new imbalance of 4.3016539410932495E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6825925925925926, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 07:56:03,513 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv105608302:90 , srv1096722228:90 , srv1097472526:90 , srv1263282002:90 , srv1265303600:90 , srv1268924819:90 , srv1360542209:90 , srv1370961572:90 , srv1473911565:90 , srv1515034203:90 , srv1676395629:90 , srv1711533868:90 , srv1762326155:90 , srv1795943190:90 , srv1817978931:90 , srv2016040570:90 , srv2145107164:90 , srv222991989:90 , srv22847496:90 , srv383151204:90 , srv38496683:90 , srv404702025:90 , srv420795701:90 , srv527691482:90 , srv536256960:90 , srv559387272:90 , srv604617674:90 , srv613308916:90 , srv688428038:90 , srv863596881:90 }
2023-03-18 07:56:03,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-18 07:56:03,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv383151204=19, srv420795701=22, srv404702025=21, srv604617674=26, srv1097472526=2, srv105608302=0, srv1473911565=8, srv1268924819=5, srv1360542209=6, srv22847496=18, srv688428038=28, srv559387272=25, srv1676395629=10, srv1817978931=14, srv1795943190=13, srv1711533868=11, srv2016040570=15, srv222991989=17, srv1265303600=4, srv1515034203=9, srv536256960=24, srv1762326155=12, srv613308916=27, srv1263282002=3, srv38496683=20, srv2145107164=16, srv527691482=23, srv1370961572=7, srv1096722228=1, srv863596881=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-18 07:56:03,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-18 07:56:03,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-18 07:56:03,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-18 07:56:03,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-18 07:56:03,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-18 07:56:03,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-18 07:56:03,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-18 07:56:03,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-18 07:56:03,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-18 07:56:03,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-18 07:56:03,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-18 07:56:03,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-18 07:56:03,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-18 07:56:03,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-18 07:56:03,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-18 07:56:03,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-18 07:56:03,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-18 07:56:03,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-18 07:56:03,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-18 07:56:03,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-18 07:56:03,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-18 07:56:03,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-18 07:56:03,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-18 07:56:03,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-18 07:56:03,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-18 07:56:03,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-18 07:56:03,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-18 07:56:03,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-18 07:56:03,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-18 07:56:03,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-18 07:56:03,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-18 07:56:03,539 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 07:56:03,539 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv105608302:90 , srv1096722228:90 , srv1097472526:90 , srv1263282002:90 , srv1265303600:90 , srv1268924819:90 , srv1360542209:90 , srv1370961572:90 , srv1473911565:90 , srv1515034203:90 , srv1676395629:90 , srv1711533868:90 , srv1762326155:90 , srv1795943190:90 , srv1817978931:90 , srv2016040570:90 , srv2145107164:90 , srv222991989:90 , srv22847496:90 , srv383151204:90 , srv38496683:90 , srv404702025:90 , srv420795701:90 , srv527691482:90 , srv536256960:90 , srv559387272:90 , srv604617674:90 , srv613308916:90 , srv688428038:90 , srv863596881:90 }
2023-03-18 07:56:03,555 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=170 (was 296), ProcessCount=169 (was 169), AvailableMemoryMB=2189 (was 2566)