2023-03-02 06:54:02,743 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-03-02 06:54:03,108 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-03-02 06:54:03,239 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-03-02 06:54:03,302 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-02 06:54:03,311 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-02 06:54:03,326 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=353, ProcessCount=170, AvailableMemoryMB=8795
2023-03-02 06:54:03,328 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-02 06:54:03,328 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-02 06:54:03,451 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1023401736:84 , srv102853419:84 , srv1045992190:84 , srv1056091279:84 , srv1122202495:84 , srv1139808788:84 , srv1163989015:84 , srv1182423196:84 , srv1211423322:84 , srv1218387511:84 , srv1266749425:84 , srv1292328952:84 , srv1293339782:84 , srv1310811643:84 , srv1314094578:84 , srv1330344375:84 , srv1349612688:84 , srv1356931271:84 , srv1360847605:84 , srv13999281:84 , srv1412444801:84 , srv1435113568:84 , srv144538735:84 , srv1481427122:84 , srv1520617214:84 , srv153835081:84 , srv1550870180:84 , srv1561578683:84 , srv1564456632:84 , srv1564572168:84 , srv1576231643:84 , srv1648779715:84 , srv1654585733:84 , srv171397381:84 , srv1743204397:84 , srv175271308:84 , srv1827049557:84 , srv1828108065:84 , srv1838993021:84 , srv1843376671:84 , srv1866736530:84 , srv1942221561:84 , srv1962388831:84 , srv1981072705:84 , srv1981110084:84 , srv2064533959:84 , srv2071460356:84 , srv2084928497:84 , srv2094403433:84 , srv2133702070:84 , srv2136170109:84 , srv218454766:84 , srv246757460:84 , srv249577674:84 , srv271154198:84 , srv29879291:84 , srv359887312:84 , srv370009148:84 , srv385718400:84 , srv386878123:84 , srv401451372:84 , srv407521721:84 , srv409380361:84 , srv413165302:84 , srv432906161:84 , srv452856186:84 , srv46664932:84 , srv473707561:84 , srv474582038:84 , srv500717637:84 , srv525287323:84 , srv533148712:84 , srv537707506:84 , srv556825955:84 , srv577119144:84 , srv579008298:84 , srv583000095:84 , srv598317352:84 , srv600599360:84 , srv605968930:84 , srv618126362:84 , srv629338668:84 , srv652018763:84 , srv658610992:84 , srv675563541:84 , srv68782012:84 , srv712901534:84 , srv776943048:84 , srv816414106:84 , srv837667379:84 , srv841091695:84 , srv86002833:84 , srv869526486:84 , srv87491041:84 , srv898432092:84 , srv966071304:84 , srv981463743:84 , srv981679665:84 , srv98350441:84 , srv1888522896:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-03-02 06:54:03,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-02 06:54:03,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv841091695=91, srv837667379=90, srv1962388831=43, srv153835081=25, srv1888522896=41, srv249577674=54, srv1564572168=29, srv1056091279=3, srv600599360=79, srv271154198=55, srv1211423322=8, srv981463743=97, srv1182423196=7, srv1827049557=36, srv1520617214=24, srv1292328952=11, srv1564456632=28, srv598317352=78, srv1981072705=44, srv1163989015=6, srv605968930=80, srv102853419=1, srv1122202495=4, srv618126362=81, srv87491041=94, srv98350441=99, srv652018763=83, srv1435113568=21, srv658610992=84, srv1561578683=27, srv898432092=95, srv525287323=71, srv1293339782=12, srv473707561=68, srv246757460=53, srv409380361=63, srv577119144=75, srv1838993021=38, srv1330344375=15, srv1356931271=17, srv1828108065=37, srv1866736530=40, srv816414106=89, srv981679665=98, srv2136170109=51, srv1648779715=31, srv1743204397=34, srv533148712=72, srv869526486=93, srv556825955=74, srv359887312=57, srv1654585733=32, srv1139808788=5, srv29879291=56, srv370009148=58, srv675563541=85, srv413165302=64, srv218454766=52, srv2084928497=48, srv583000095=77, srv385718400=59, srv1349612688=16, srv1266749425=10, srv2094403433=49, srv1310811643=13, srv68782012=86, srv474582038=69, srv776943048=88, srv144538735=22, srv537707506=73, srv1843376671=39, srv401451372=61, srv432906161=65, srv386878123=60, srv407521721=62, srv1023401736=0, srv500717637=70, srv1981110084=45, srv1218387511=9, srv1412444801=20, srv629338668=82, srv1576231643=30, srv46664932=67, srv966071304=96, srv712901534=87, srv171397381=33, srv2133702070=50, srv1045992190=2, srv86002833=92, srv1481427122=23, srv579008298=76, srv1360847605=18, srv2064533959=46, srv1314094578=14, srv175271308=35, srv1942221561=42, srv2071460356=47, srv13999281=19, srv452856186=66, srv1550870180=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-02 06:54:03,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-02 06:54:03,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-02 06:54:03,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-02 06:54:03,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-02 06:54:03,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-02 06:54:03,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-02 06:54:03,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-02 06:54:03,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-02 06:54:03,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-02 06:54:03,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-02 06:54:03,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-02 06:54:03,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-02 06:54:03,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-02 06:54:03,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-02 06:54:03,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-02 06:54:03,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-02 06:54:03,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-02 06:54:03,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-02 06:54:03,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-02 06:54:03,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-02 06:54:03,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-02 06:54:03,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-02 06:54:03,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-02 06:54:03,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-02 06:54:03,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-02 06:54:03,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-02 06:54:03,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-02 06:54:03,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-02 06:54:03,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-02 06:54:03,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-02 06:54:03,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-02 06:54:03,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-02 06:54:03,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-02 06:54:03,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-02 06:54:03,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-02 06:54:03,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-02 06:54:03,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-02 06:54:03,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-02 06:54:03,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-02 06:54:03,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-02 06:54:03,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-02 06:54:03,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-02 06:54:03,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-02 06:54:03,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-02 06:54:03,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-02 06:54:03,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-02 06:54:03,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-02 06:54:03,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-02 06:54:03,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-02 06:54:03,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-02 06:54:03,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-02 06:54:03,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-02 06:54:03,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-02 06:54:03,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-02 06:54:03,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-02 06:54:03,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-02 06:54:03,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-02 06:54:03,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-02 06:54:03,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-02 06:54:03,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-02 06:54:03,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-02 06:54:03,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-02 06:54:03,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-02 06:54:03,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-02 06:54:03,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-02 06:54:03,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-02 06:54:03,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-02 06:54:03,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-02 06:54:03,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-02 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-02 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-02 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-02 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-02 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-02 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-02 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-02 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-02 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-02 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-02 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-02 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-02 06:54:03,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-02 06:54:03,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-02 06:54:03,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-02 06:54:03,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-02 06:54:03,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-02 06:54:03,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-02 06:54:03,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-02 06:54:03,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-02 06:54:03,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-02 06:54:03,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-02 06:54:03,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-02 06:54:03,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-02 06:54:03,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-02 06:54:03,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-02 06:54:03,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-02 06:54:03,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-02 06:54:03,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-02 06:54:03,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-02 06:54:03,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-02 06:54:03,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-02 06:54:03,907 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-02 06:54:03,908 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-02 06:54:03,908 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-02 06:54:04,690 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1070 ms to try 5000 different iterations.  Found a solution that moves 4701 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17743670854255456. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04001153344137432, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05814954211372532, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5223333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04001153344137432, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17233333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24216666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-02 06:54:04,717 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1828108065:83 , srv1962388831:83 , srv2136170109:83 , srv401451372:83 , srv1360847605:84 , srv13999281:84 , srv153835081:84 , srv1564456632:84 , srv1564572168:84 , srv1648779715:84 , srv1654585733:84 , srv473707561:84 , srv525287323:84 , srv579008298:84 , srv869526486:84 , srv87491041:84 , srv1023401736:85 , srv1435113568:85 , srv1481427122:85 , srv1743204397:85 , srv1843376671:85 , srv2064533959:85 , srv2084928497:85 , srv246757460:85 , srv249577674:85 , srv29879291:85 , srv370009148:85 , srv407521721:85 , srv452856186:85 , srv46664932:85 , srv500717637:85 , srv577119144:85 , srv583000095:85 , srv600599360:85 , srv658610992:85 , srv776943048:85 , srv966071304:85 , srv981679665:85 , srv1056091279:86 , srv1163989015:86 , srv1182423196:86 , srv1310811643:86 , srv1349612688:86 , srv171397381:86 , srv1942221561:86 , srv271154198:86 , srv359887312:86 , srv386878123:86 , srv629338668:86 , srv675563541:86 , srv68782012:86 , srv981463743:86 , srv1211423322:87 , srv1266749425:87 , srv1293339782:87 , srv1330344375:87 , srv1356931271:87 , srv1550870180:87 , srv1561578683:87 , srv175271308:87 , srv1838993021:87 , srv1981072705:87 , srv1981110084:87 , srv2071460356:87 , srv2094403433:87 , srv218454766:87 , srv385718400:87 , srv409380361:87 , srv556825955:87 , srv837667379:87 , srv86002833:87 , srv898432092:87 , srv1045992190:88 , srv1122202495:88 , srv1218387511:88 , srv1292328952:88 , srv144538735:88 , srv1576231643:88 , srv1866736530:88 , srv413165302:88 , srv474582038:88 , srv537707506:88 , srv605968930:88 , srv841091695:88 , srv1314094578:89 , srv1827049557:89 , srv2133702070:89 , srv432906161:89 , srv533148712:89 , srv598317352:89 , srv618126362:89 , srv652018763:89 , srv712901534:89 , srv1139808788:90 , srv1412444801:90 , srv1520617214:90 , srv102853419:91 , srv816414106:91 , srv98350441:91 , srv1888522896:446 }
2023-03-02 06:54:04,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-02 06:54:04,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv841091695=91, srv837667379=90, srv1962388831=43, srv153835081=25, srv1888522896=41, srv249577674=54, srv1564572168=29, srv1056091279=3, srv600599360=79, srv271154198=55, srv1211423322=8, srv981463743=97, srv1182423196=7, srv1827049557=36, srv1520617214=24, srv1292328952=11, srv1564456632=28, srv598317352=78, srv1981072705=44, srv1163989015=6, srv605968930=80, srv102853419=1, srv1122202495=4, srv618126362=81, srv87491041=94, srv98350441=99, srv652018763=83, srv1435113568=21, srv658610992=84, srv1561578683=27, srv898432092=95, srv525287323=71, srv1293339782=12, srv473707561=68, srv246757460=53, srv409380361=63, srv577119144=75, srv1838993021=38, srv1330344375=15, srv1356931271=17, srv1828108065=37, srv1866736530=40, srv816414106=89, srv981679665=98, srv2136170109=51, srv1648779715=31, srv1743204397=34, srv533148712=72, srv869526486=93, srv556825955=74, srv359887312=57, srv1654585733=32, srv1139808788=5, srv29879291=56, srv370009148=58, srv675563541=85, srv413165302=64, srv218454766=52, srv2084928497=48, srv583000095=77, srv385718400=59, srv1349612688=16, srv1266749425=10, srv2094403433=49, srv1310811643=13, srv68782012=86, srv474582038=69, srv776943048=88, srv144538735=22, srv537707506=73, srv1843376671=39, srv401451372=61, srv432906161=65, srv386878123=60, srv407521721=62, srv1023401736=0, srv500717637=70, srv1981110084=45, srv1218387511=9, srv1412444801=20, srv629338668=82, srv1576231643=30, srv46664932=67, srv966071304=96, srv712901534=87, srv171397381=33, srv2133702070=50, srv1045992190=2, srv86002833=92, srv1481427122=23, srv579008298=76, srv1360847605=18, srv2064533959=46, srv1314094578=14, srv175271308=35, srv1942221561=42, srv2071460356=47, srv13999281=19, srv452856186=66, srv1550870180=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-02 06:54:04,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-02 06:54:04,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-02 06:54:04,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-02 06:54:04,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-02 06:54:04,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-02 06:54:04,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-02 06:54:04,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-02 06:54:04,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-02 06:54:04,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-02 06:54:04,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-02 06:54:04,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-02 06:54:04,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-02 06:54:04,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-02 06:54:04,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-02 06:54:04,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-02 06:54:04,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-02 06:54:04,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-02 06:54:04,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-02 06:54:04,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-02 06:54:04,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-02 06:54:04,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-02 06:54:04,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-02 06:54:04,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-02 06:54:04,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-02 06:54:04,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-02 06:54:04,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-02 06:54:04,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-02 06:54:04,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-02 06:54:04,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-02 06:54:04,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-02 06:54:04,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-02 06:54:04,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-02 06:54:04,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-02 06:54:04,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-02 06:54:04,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-02 06:54:04,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-02 06:54:04,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-02 06:54:04,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-02 06:54:04,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-02 06:54:04,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-02 06:54:04,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-02 06:54:04,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-02 06:54:04,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-02 06:54:04,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-02 06:54:04,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-02 06:54:04,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-02 06:54:04,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-02 06:54:04,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-02 06:54:04,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-02 06:54:04,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-02 06:54:04,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-02 06:54:04,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-02 06:54:04,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-02 06:54:04,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-02 06:54:04,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-02 06:54:04,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-02 06:54:04,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-02 06:54:04,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-02 06:54:04,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-02 06:54:04,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-02 06:54:04,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-02 06:54:04,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-02 06:54:04,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-02 06:54:04,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-02 06:54:04,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-02 06:54:04,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-02 06:54:04,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-02 06:54:04,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-02 06:54:04,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-02 06:54:04,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-02 06:54:04,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-02 06:54:04,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-02 06:54:04,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-02 06:54:04,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-02 06:54:04,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-02 06:54:04,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-02 06:54:04,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-02 06:54:04,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-02 06:54:04,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-02 06:54:04,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-02 06:54:04,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-02 06:54:04,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-02 06:54:04,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-02 06:54:04,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-02 06:54:04,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-02 06:54:04,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-02 06:54:04,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-02 06:54:04,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-02 06:54:04,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-02 06:54:04,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-02 06:54:04,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-02 06:54:04,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-02 06:54:04,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-02 06:54:04,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-02 06:54:04,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-02 06:54:04,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-02 06:54:04,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-02 06:54:04,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-02 06:54:04,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-02 06:54:04,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-02 06:54:04,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-02 06:54:04,957 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04001153344137432, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05814954211372532, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04001153344137432, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17233333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24216666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-02 06:54:04,957 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-02 06:54:04,957 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17740379143700313, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04001153344137432, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05814954211372532, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04001153344137432, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17233333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24216666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-02 06:54:05,264 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 479 ms to try 5000 different iterations.  Found a solution that moves 2676 regions; Going from a computed imbalance of 0.17740379143700313 to a new imbalance of 0.014321828678039142. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020705838839928593); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032000280582383814, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29733333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020705838839928593); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04616666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-02 06:54:05,274 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv102853419:78 , srv1314094578:81 , srv98350441:81 , srv1139808788:82 , srv537707506:82 , srv1045992190:83 , srv1412444801:83 , srv1981072705:83 , srv385718400:83 , srv598317352:83 , srv605968930:83 , srv618126362:83 , srv1218387511:84 , srv1310811643:84 , srv1356931271:84 , srv1520617214:84 , srv1550870180:84 , srv1576231643:84 , srv1827049557:84 , srv2133702070:84 , srv432906161:84 , srv712901534:84 , srv816414106:84 , srv841091695:84 , srv966071304:84 , srv1056091279:85 , srv1122202495:85 , srv1266749425:85 , srv1292328952:85 , srv1293339782:85 , srv1330344375:85 , srv1561578683:85 , srv175271308:85 , srv1843376671:85 , srv2094403433:85 , srv370009148:85 , srv407521721:85 , srv413165302:85 , srv46664932:85 , srv474582038:85 , srv556825955:85 , srv629338668:85 , srv1349612688:86 , srv1481427122:86 , srv1866736530:86 , srv1981110084:86 , srv2064533959:86 , srv2071460356:86 , srv2084928497:86 , srv218454766:86 , srv271154198:86 , srv359887312:86 , srv386878123:86 , srv409380361:86 , srv500717637:86 , srv583000095:86 , srv652018763:86 , srv658610992:86 , srv675563541:86 , srv837667379:86 , srv898432092:86 , srv981463743:86 , srv1023401736:87 , srv1211423322:87 , srv1360847605:87 , srv13999281:87 , srv144538735:87 , srv1648779715:87 , srv171397381:87 , srv1743204397:87 , srv246757460:87 , srv452856186:87 , srv525287323:87 , srv533148712:87 , srv577119144:87 , srv579008298:87 , srv68782012:87 , srv86002833:87 , srv869526486:87 , srv1163989015:88 , srv1182423196:88 , srv1435113568:88 , srv153835081:88 , srv1838993021:88 , srv1942221561:88 , srv1962388831:88 , srv2136170109:88 , srv249577674:88 , srv29879291:88 , srv600599360:88 , srv776943048:88 , srv87491041:88 , srv981679665:88 , srv1564456632:89 , srv1564572168:89 , srv1654585733:89 , srv1828108065:89 , srv401451372:89 , srv473707561:90 , srv1888522896:512 }
2023-03-02 06:54:05,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-02 06:54:05,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv841091695=91, srv837667379=90, srv1962388831=43, srv153835081=25, srv1888522896=41, srv249577674=54, srv1564572168=29, srv1056091279=3, srv600599360=79, srv271154198=55, srv1211423322=8, srv981463743=97, srv1182423196=7, srv1827049557=36, srv1520617214=24, srv1292328952=11, srv1564456632=28, srv598317352=78, srv1981072705=44, srv1163989015=6, srv605968930=80, srv102853419=1, srv1122202495=4, srv618126362=81, srv87491041=94, srv98350441=99, srv652018763=83, srv1435113568=21, srv658610992=84, srv1561578683=27, srv898432092=95, srv525287323=71, srv1293339782=12, srv473707561=68, srv246757460=53, srv409380361=63, srv577119144=75, srv1838993021=38, srv1330344375=15, srv1356931271=17, srv1828108065=37, srv1866736530=40, srv816414106=89, srv981679665=98, srv2136170109=51, srv1648779715=31, srv1743204397=34, srv533148712=72, srv869526486=93, srv556825955=74, srv359887312=57, srv1654585733=32, srv1139808788=5, srv29879291=56, srv370009148=58, srv675563541=85, srv413165302=64, srv218454766=52, srv2084928497=48, srv583000095=77, srv385718400=59, srv1349612688=16, srv1266749425=10, srv2094403433=49, srv1310811643=13, srv68782012=86, srv474582038=69, srv776943048=88, srv144538735=22, srv537707506=73, srv1843376671=39, srv401451372=61, srv432906161=65, srv386878123=60, srv407521721=62, srv1023401736=0, srv500717637=70, srv1981110084=45, srv1218387511=9, srv1412444801=20, srv629338668=82, srv1576231643=30, srv46664932=67, srv966071304=96, srv712901534=87, srv171397381=33, srv2133702070=50, srv1045992190=2, srv86002833=92, srv1481427122=23, srv579008298=76, srv1360847605=18, srv2064533959=46, srv1314094578=14, srv175271308=35, srv1942221561=42, srv2071460356=47, srv13999281=19, srv452856186=66, srv1550870180=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-02 06:54:05,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-02 06:54:05,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-02 06:54:05,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-02 06:54:05,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-02 06:54:05,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-02 06:54:05,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-02 06:54:05,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-02 06:54:05,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-02 06:54:05,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-02 06:54:05,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-02 06:54:05,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-02 06:54:05,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-02 06:54:05,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-02 06:54:05,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-02 06:54:05,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-02 06:54:05,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-02 06:54:05,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-02 06:54:05,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-02 06:54:05,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-02 06:54:05,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-02 06:54:05,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-02 06:54:05,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-02 06:54:05,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-02 06:54:05,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-02 06:54:05,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-02 06:54:05,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-02 06:54:05,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-02 06:54:05,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-02 06:54:05,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-02 06:54:05,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-02 06:54:05,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-02 06:54:05,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-02 06:54:05,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-02 06:54:05,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-02 06:54:05,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-02 06:54:05,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-02 06:54:05,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-02 06:54:05,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-02 06:54:05,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-02 06:54:05,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-02 06:54:05,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-02 06:54:05,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-02 06:54:05,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-02 06:54:05,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-02 06:54:05,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-02 06:54:05,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-02 06:54:05,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-02 06:54:05,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-02 06:54:05,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-02 06:54:05,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-02 06:54:05,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-02 06:54:05,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-02 06:54:05,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-02 06:54:05,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-02 06:54:05,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-02 06:54:05,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-02 06:54:05,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-02 06:54:05,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-02 06:54:05,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-02 06:54:05,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-02 06:54:05,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-02 06:54:05,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-02 06:54:05,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-02 06:54:05,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-02 06:54:05,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-02 06:54:05,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-02 06:54:05,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-02 06:54:05,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-02 06:54:05,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-02 06:54:05,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-02 06:54:05,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-02 06:54:05,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-02 06:54:05,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-02 06:54:05,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-02 06:54:05,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-02 06:54:05,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-02 06:54:05,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-02 06:54:05,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-02 06:54:05,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-02 06:54:05,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-02 06:54:05,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-02 06:54:05,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-02 06:54:05,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-02 06:54:05,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-02 06:54:05,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-02 06:54:05,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-02 06:54:05,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-02 06:54:05,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-02 06:54:05,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-02 06:54:05,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-02 06:54:05,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-02 06:54:05,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-02 06:54:05,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-02 06:54:05,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-02 06:54:05,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-02 06:54:05,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-02 06:54:05,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-02 06:54:05,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-02 06:54:05,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-02 06:54:05,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-02 06:54:05,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-02 06:54:05,483 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020705838839928593); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032000280582383814, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020705838839928593); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04616666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-02 06:54:05,483 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-02 06:54:05,483 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014303090925549128, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020705838839928593); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032000280582383814, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020705838839928593); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04616666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-02 06:54:05,713 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 364 ms to try 5000 different iterations.  Found a solution that moves 1031 regions; Going from a computed imbalance of 0.014303090925549128 to a new imbalance of 4.755562685878678E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00499906463379595); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007027283689263065); MoveCostFunction : (multiplier=7.0, imbalance=0.11455555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00499906463379595); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-02 06:54:05,717 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1838993021:82 , srv533148712:82 , srv816414106:82 , srv1163989015:83 , srv144538735:83 , srv1942221561:83 , srv409380361:83 , srv86002833:83 , srv1139808788:84 , srv1182423196:84 , srv1211423322:84 , srv1292328952:84 , srv1412444801:84 , srv1435113568:84 , srv1520617214:84 , srv1564572168:84 , srv171397381:84 , srv2071460356:84 , srv29879291:84 , srv473707561:84 , srv600599360:84 , srv618126362:84 , srv652018763:84 , srv68782012:84 , srv712901534:84 , srv837667379:84 , srv1045992190:85 , srv1122202495:85 , srv1218387511:85 , srv1330344375:85 , srv1349612688:85 , srv153835081:85 , srv1564456632:85 , srv175271308:85 , srv1827049557:85 , srv1866736530:85 , srv1981110084:85 , srv2094403433:85 , srv2133702070:85 , srv218454766:85 , srv249577674:85 , srv359887312:85 , srv386878123:85 , srv413165302:85 , srv432906161:85 , srv452856186:85 , srv474582038:85 , srv556825955:85 , srv675563541:85 , srv776943048:85 , srv898432092:85 , srv981463743:85 , srv981679665:85 , srv1023401736:86 , srv1056091279:86 , srv1266749425:86 , srv1293339782:86 , srv1356931271:86 , srv1360847605:86 , srv13999281:86 , srv1481427122:86 , srv1550870180:86 , srv1576231643:86 , srv1654585733:86 , srv1743204397:86 , srv1828108065:86 , srv1962388831:86 , srv2064533959:86 , srv246757460:86 , srv271154198:86 , srv401451372:86 , srv577119144:86 , srv579008298:86 , srv583000095:86 , srv598317352:86 , srv605968930:86 , srv841091695:86 , srv87491041:86 , srv98350441:86 , srv1310811643:87 , srv1314094578:87 , srv1561578683:87 , srv1843376671:87 , srv2136170109:87 , srv370009148:87 , srv525287323:87 , srv629338668:87 , srv658610992:87 , srv869526486:87 , srv102853419:88 , srv1648779715:88 , srv1981072705:88 , srv2084928497:88 , srv385718400:88 , srv407521721:88 , srv46664932:88 , srv500717637:88 , srv537707506:88 , srv966071304:90 , srv1888522896:544 }
2023-03-02 06:54:05,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-02 06:54:05,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv841091695=91, srv837667379=90, srv1962388831=43, srv153835081=25, srv1888522896=41, srv249577674=54, srv1564572168=29, srv1056091279=3, srv600599360=79, srv271154198=55, srv1211423322=8, srv981463743=97, srv1182423196=7, srv1827049557=36, srv1520617214=24, srv1292328952=11, srv1564456632=28, srv598317352=78, srv1981072705=44, srv1163989015=6, srv605968930=80, srv102853419=1, srv1122202495=4, srv618126362=81, srv87491041=94, srv98350441=99, srv652018763=83, srv1435113568=21, srv658610992=84, srv1561578683=27, srv898432092=95, srv525287323=71, srv1293339782=12, srv473707561=68, srv246757460=53, srv409380361=63, srv577119144=75, srv1838993021=38, srv1330344375=15, srv1356931271=17, srv1828108065=37, srv1866736530=40, srv816414106=89, srv981679665=98, srv2136170109=51, srv1648779715=31, srv1743204397=34, srv533148712=72, srv869526486=93, srv556825955=74, srv359887312=57, srv1654585733=32, srv1139808788=5, srv29879291=56, srv370009148=58, srv675563541=85, srv413165302=64, srv218454766=52, srv2084928497=48, srv583000095=77, srv385718400=59, srv1349612688=16, srv1266749425=10, srv2094403433=49, srv1310811643=13, srv68782012=86, srv474582038=69, srv776943048=88, srv144538735=22, srv537707506=73, srv1843376671=39, srv401451372=61, srv432906161=65, srv386878123=60, srv407521721=62, srv1023401736=0, srv500717637=70, srv1981110084=45, srv1218387511=9, srv1412444801=20, srv629338668=82, srv1576231643=30, srv46664932=67, srv966071304=96, srv712901534=87, srv171397381=33, srv2133702070=50, srv1045992190=2, srv86002833=92, srv1481427122=23, srv579008298=76, srv1360847605=18, srv2064533959=46, srv1314094578=14, srv175271308=35, srv1942221561=42, srv2071460356=47, srv13999281=19, srv452856186=66, srv1550870180=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-02 06:54:05,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-02 06:54:05,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-02 06:54:05,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-02 06:54:05,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-02 06:54:05,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-02 06:54:05,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-02 06:54:05,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-02 06:54:05,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-02 06:54:05,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-02 06:54:05,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-02 06:54:05,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-02 06:54:05,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-02 06:54:05,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-02 06:54:05,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-02 06:54:05,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-02 06:54:05,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-02 06:54:05,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-02 06:54:05,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-02 06:54:05,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-02 06:54:05,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-02 06:54:05,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-02 06:54:05,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-02 06:54:05,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-02 06:54:05,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-02 06:54:05,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-02 06:54:05,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-02 06:54:05,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-02 06:54:05,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-02 06:54:05,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-02 06:54:05,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-02 06:54:05,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-02 06:54:05,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-02 06:54:05,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-02 06:54:05,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-02 06:54:05,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-02 06:54:05,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-02 06:54:05,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-02 06:54:05,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-02 06:54:05,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-02 06:54:05,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-02 06:54:05,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-02 06:54:05,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-02 06:54:05,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-02 06:54:05,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-02 06:54:05,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-02 06:54:05,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-02 06:54:05,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-02 06:54:05,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-02 06:54:05,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-02 06:54:05,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-02 06:54:05,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-02 06:54:05,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-02 06:54:05,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-02 06:54:05,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-02 06:54:05,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-02 06:54:05,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-02 06:54:05,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-02 06:54:05,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-02 06:54:05,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-02 06:54:05,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-02 06:54:05,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-02 06:54:05,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-02 06:54:05,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-02 06:54:05,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-02 06:54:05,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-02 06:54:05,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-02 06:54:05,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-02 06:54:05,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-02 06:54:05,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-02 06:54:05,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-02 06:54:05,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-02 06:54:05,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-02 06:54:05,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-02 06:54:05,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-02 06:54:05,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-02 06:54:05,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-02 06:54:05,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-02 06:54:05,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-02 06:54:05,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-02 06:54:05,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-02 06:54:05,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-02 06:54:05,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-02 06:54:05,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-02 06:54:05,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-02 06:54:05,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-02 06:54:05,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-02 06:54:05,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-02 06:54:05,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-02 06:54:05,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-02 06:54:05,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-02 06:54:05,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-02 06:54:05,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-02 06:54:05,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-02 06:54:05,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-02 06:54:05,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-02 06:54:05,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-02 06:54:05,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-02 06:54:05,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-02 06:54:05,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-02 06:54:05,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-02 06:54:05,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-02 06:54:05,857 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00499906463379595); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007027283689263065); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00499906463379595); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-02 06:54:05,857 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-02 06:54:05,858 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.683370522884638E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00499906463379595); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007027283689263065); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00499906463379595); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-02 06:54:06,194 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 434 ms to try 5000 different iterations.  Found a solution that moves 271 regions; Going from a computed imbalance of 4.683370522884638E-4 to a new imbalance of 1.098620384082707E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.030111111111111113, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-02 06:54:06,195 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1561578683:83 , srv1648779715:83 , srv1654585733:83 , srv1866736530:83 , srv500717637:83 , srv652018763:83 , srv966071304:83 , srv1023401736:84 , srv1122202495:84 , srv1182423196:84 , srv1211423322:84 , srv1266749425:84 , srv1293339782:84 , srv1310811643:84 , srv1356931271:84 , srv1435113568:84 , srv144538735:84 , srv1520617214:84 , srv1564456632:84 , srv1576231643:84 , srv1743204397:84 , srv1827049557:84 , srv1828108065:84 , srv1981072705:84 , srv1981110084:84 , srv2084928497:84 , srv2136170109:84 , srv218454766:84 , srv246757460:84 , srv271154198:84 , srv385718400:84 , srv401451372:84 , srv407521721:84 , srv413165302:84 , srv432906161:84 , srv46664932:84 , srv473707561:84 , srv474582038:84 , srv525287323:84 , srv533148712:84 , srv537707506:84 , srv577119144:84 , srv598317352:84 , srv629338668:84 , srv658610992:84 , srv675563541:84 , srv776943048:84 , srv869526486:84 , srv87491041:84 , srv898432092:84 , srv981679665:84 , srv98350441:84 , srv102853419:85 , srv1056091279:85 , srv1139808788:85 , srv1163989015:85 , srv1218387511:85 , srv1292328952:85 , srv1314094578:85 , srv1330344375:85 , srv1349612688:85 , srv1360847605:85 , srv13999281:85 , srv1412444801:85 , srv1481427122:85 , srv153835081:85 , srv1550870180:85 , srv1564572168:85 , srv171397381:85 , srv175271308:85 , srv1838993021:85 , srv1843376671:85 , srv1942221561:85 , srv1962388831:85 , srv2064533959:85 , srv2071460356:85 , srv2094403433:85 , srv2133702070:85 , srv249577674:85 , srv29879291:85 , srv359887312:85 , srv370009148:85 , srv386878123:85 , srv452856186:85 , srv556825955:85 , srv579008298:85 , srv583000095:85 , srv600599360:85 , srv605968930:85 , srv68782012:85 , srv712901534:85 , srv816414106:85 , srv837667379:85 , srv841091695:85 , srv981463743:85 , srv1045992190:86 , srv409380361:86 , srv618126362:86 , srv86002833:86 , srv1888522896:640 }
2023-03-02 06:54:06,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-02 06:54:06,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv841091695=91, srv837667379=90, srv1962388831=43, srv153835081=25, srv1888522896=41, srv249577674=54, srv1564572168=29, srv1056091279=3, srv600599360=79, srv271154198=55, srv1211423322=8, srv981463743=97, srv1182423196=7, srv1827049557=36, srv1520617214=24, srv1292328952=11, srv1564456632=28, srv598317352=78, srv1981072705=44, srv1163989015=6, srv605968930=80, srv102853419=1, srv1122202495=4, srv618126362=81, srv87491041=94, srv98350441=99, srv652018763=83, srv1435113568=21, srv658610992=84, srv1561578683=27, srv898432092=95, srv525287323=71, srv1293339782=12, srv473707561=68, srv246757460=53, srv409380361=63, srv577119144=75, srv1838993021=38, srv1330344375=15, srv1356931271=17, srv1828108065=37, srv1866736530=40, srv816414106=89, srv981679665=98, srv2136170109=51, srv1648779715=31, srv1743204397=34, srv533148712=72, srv869526486=93, srv556825955=74, srv359887312=57, srv1654585733=32, srv1139808788=5, srv29879291=56, srv370009148=58, srv675563541=85, srv413165302=64, srv218454766=52, srv2084928497=48, srv583000095=77, srv385718400=59, srv1349612688=16, srv1266749425=10, srv2094403433=49, srv1310811643=13, srv68782012=86, srv474582038=69, srv776943048=88, srv144538735=22, srv537707506=73, srv1843376671=39, srv401451372=61, srv432906161=65, srv386878123=60, srv407521721=62, srv1023401736=0, srv500717637=70, srv1981110084=45, srv1218387511=9, srv1412444801=20, srv629338668=82, srv1576231643=30, srv46664932=67, srv966071304=96, srv712901534=87, srv171397381=33, srv2133702070=50, srv1045992190=2, srv86002833=92, srv1481427122=23, srv579008298=76, srv1360847605=18, srv2064533959=46, srv1314094578=14, srv175271308=35, srv1942221561=42, srv2071460356=47, srv13999281=19, srv452856186=66, srv1550870180=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-02 06:54:06,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-02 06:54:06,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-02 06:54:06,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-02 06:54:06,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-02 06:54:06,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-02 06:54:06,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-02 06:54:06,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-02 06:54:06,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-02 06:54:06,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-02 06:54:06,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-02 06:54:06,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-02 06:54:06,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-02 06:54:06,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-02 06:54:06,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-02 06:54:06,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-02 06:54:06,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-02 06:54:06,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-02 06:54:06,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-02 06:54:06,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-02 06:54:06,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-02 06:54:06,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-02 06:54:06,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-02 06:54:06,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-02 06:54:06,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-02 06:54:06,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-02 06:54:06,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-02 06:54:06,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-02 06:54:06,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-02 06:54:06,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-02 06:54:06,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-02 06:54:06,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-02 06:54:06,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-02 06:54:06,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-02 06:54:06,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-02 06:54:06,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-02 06:54:06,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-02 06:54:06,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-02 06:54:06,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-02 06:54:06,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-02 06:54:06,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-02 06:54:06,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-02 06:54:06,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-02 06:54:06,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-02 06:54:06,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-02 06:54:06,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-02 06:54:06,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-02 06:54:06,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-02 06:54:06,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-02 06:54:06,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-02 06:54:06,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-02 06:54:06,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-02 06:54:06,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-02 06:54:06,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-02 06:54:06,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-02 06:54:06,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-02 06:54:06,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-02 06:54:06,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-02 06:54:06,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-02 06:54:06,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-02 06:54:06,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-02 06:54:06,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-02 06:54:06,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-02 06:54:06,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-02 06:54:06,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-02 06:54:06,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-02 06:54:06,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-02 06:54:06,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-02 06:54:06,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-02 06:54:06,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-02 06:54:06,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-02 06:54:06,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-02 06:54:06,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-02 06:54:06,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-02 06:54:06,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-02 06:54:06,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-02 06:54:06,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-02 06:54:06,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-02 06:54:06,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-02 06:54:06,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-02 06:54:06,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-02 06:54:06,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-02 06:54:06,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-02 06:54:06,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-02 06:54:06,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-02 06:54:06,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-02 06:54:06,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-02 06:54:06,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-02 06:54:06,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-02 06:54:06,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-02 06:54:06,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-02 06:54:06,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-02 06:54:06,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-02 06:54:06,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-02 06:54:06,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-02 06:54:06,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-02 06:54:06,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-02 06:54:06,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-02 06:54:06,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-02 06:54:06,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-02 06:54:06,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-02 06:54:06,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-02 06:54:06,321 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=9.08862128298181E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-02 06:54:06,322 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1561578683:83 , srv1648779715:83 , srv1654585733:83 , srv1866736530:83 , srv500717637:83 , srv652018763:83 , srv966071304:83 , srv1023401736:84 , srv1122202495:84 , srv1182423196:84 , srv1211423322:84 , srv1266749425:84 , srv1293339782:84 , srv1310811643:84 , srv1356931271:84 , srv1435113568:84 , srv144538735:84 , srv1520617214:84 , srv1564456632:84 , srv1576231643:84 , srv1743204397:84 , srv1827049557:84 , srv1828108065:84 , srv1981072705:84 , srv1981110084:84 , srv2084928497:84 , srv2136170109:84 , srv218454766:84 , srv246757460:84 , srv271154198:84 , srv385718400:84 , srv401451372:84 , srv407521721:84 , srv413165302:84 , srv432906161:84 , srv46664932:84 , srv473707561:84 , srv474582038:84 , srv525287323:84 , srv533148712:84 , srv537707506:84 , srv577119144:84 , srv598317352:84 , srv629338668:84 , srv658610992:84 , srv675563541:84 , srv776943048:84 , srv869526486:84 , srv87491041:84 , srv898432092:84 , srv981679665:84 , srv98350441:84 , srv102853419:85 , srv1056091279:85 , srv1139808788:85 , srv1163989015:85 , srv1218387511:85 , srv1292328952:85 , srv1314094578:85 , srv1330344375:85 , srv1349612688:85 , srv1360847605:85 , srv13999281:85 , srv1412444801:85 , srv1481427122:85 , srv153835081:85 , srv1550870180:85 , srv1564572168:85 , srv171397381:85 , srv175271308:85 , srv1838993021:85 , srv1843376671:85 , srv1942221561:85 , srv1962388831:85 , srv2064533959:85 , srv2071460356:85 , srv2094403433:85 , srv2133702070:85 , srv249577674:85 , srv29879291:85 , srv359887312:85 , srv370009148:85 , srv386878123:85 , srv452856186:85 , srv556825955:85 , srv579008298:85 , srv583000095:85 , srv600599360:85 , srv605968930:85 , srv68782012:85 , srv712901534:85 , srv816414106:85 , srv837667379:85 , srv841091695:85 , srv981463743:85 , srv1045992190:86 , srv409380361:86 , srv618126362:86 , srv86002833:86 , srv1888522896:640 }
2023-03-02 06:54:06,351 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=353 (was 353), ProcessCount=173 (was 170) - ProcessCount LEAK? -, AvailableMemoryMB=8274 (was 8795)
2023-03-02 06:54:06,359 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=353, ProcessCount=173, AvailableMemoryMB=8274
2023-03-02 06:54:06,360 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-02 06:54:06,360 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-03-02 06:54:06,370 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1063428944:84 , srv1146471562:84 , srv1188002290:84 , srv1188702407:84 , srv1206918878:84 , srv1255036854:84 , srv1500559545:84 , srv1575424515:84 , srv1619151766:84 , srv172176481:84 , srv1728825982:84 , srv1860780602:84 , srv18679647:84 , srv1868763968:84 , srv1982489283:84 , srv270810695:84 , srv319972043:84 , srv332307323:84 , srv338710905:84 , srv370218036:84 , srv524946611:84 , srv573904598:84 , srv608456763:84 , srv652680052:84 , srv672010843:84 , srv704190385:84 , srv781156605:84 , srv847402573:84 , srv853753933:84 , srv1974541008:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-03-02 06:54:06,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-02 06:54:06,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv370218036=20, srv1188702407=3, srv18679647=12, srv704190385=26, srv332307323=18, srv1146471562=1, srv652680052=24, srv270810695=16, srv1619151766=8, srv338710905=19, srv1868763968=13, srv1860780602=11, srv573904598=22, srv847402573=28, srv1728825982=10, srv608456763=23, srv1188002290=2, srv1982489283=15, srv781156605=27, srv1575424515=7, srv1063428944=0, srv1500559545=6, srv853753933=29, srv1974541008=14, srv524946611=21, srv672010843=25, srv1255036854=5, srv172176481=9, srv1206918878=4, srv319972043=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-02 06:54:06,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-02 06:54:06,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-02 06:54:06,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-02 06:54:06,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-02 06:54:06,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-02 06:54:06,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-02 06:54:06,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-02 06:54:06,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-02 06:54:06,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-02 06:54:06,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-02 06:54:06,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-02 06:54:06,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-02 06:54:06,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-02 06:54:06,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-02 06:54:06,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-02 06:54:06,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-02 06:54:06,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-02 06:54:06,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-02 06:54:06,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-02 06:54:06,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-02 06:54:06,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-02 06:54:06,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-02 06:54:06,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-02 06:54:06,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-02 06:54:06,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-02 06:54:06,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-02 06:54:06,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-02 06:54:06,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-02 06:54:06,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-02 06:54:06,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-02 06:54:06,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-02 06:54:06,395 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1459079498642026, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-02 06:54:06,395 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912652794449969, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1459079498642026, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-03-02 06:56:06,383 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6703661 different iterations.  Found a solution that moves 1846 regions; Going from a computed imbalance of 0.9912652794449969 to a new imbalance of 4.308656090753194E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6837037037037037, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-02 06:56:06,389 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1063428944:90 , srv1146471562:90 , srv1188002290:90 , srv1188702407:90 , srv1206918878:90 , srv1255036854:90 , srv1500559545:90 , srv1575424515:90 , srv1619151766:90 , srv172176481:90 , srv1728825982:90 , srv1860780602:90 , srv18679647:90 , srv1868763968:90 , srv1974541008:90 , srv1982489283:90 , srv270810695:90 , srv319972043:90 , srv332307323:90 , srv338710905:90 , srv370218036:90 , srv524946611:90 , srv573904598:90 , srv608456763:90 , srv652680052:90 , srv672010843:90 , srv704190385:90 , srv781156605:90 , srv847402573:90 , srv853753933:90 }
2023-03-02 06:56:06,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-02 06:56:06,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv370218036=20, srv1188702407=3, srv18679647=12, srv704190385=26, srv332307323=18, srv1146471562=1, srv652680052=24, srv270810695=16, srv1619151766=8, srv338710905=19, srv1868763968=13, srv1860780602=11, srv573904598=22, srv847402573=28, srv1728825982=10, srv608456763=23, srv1188002290=2, srv1982489283=15, srv781156605=27, srv1575424515=7, srv1063428944=0, srv1500559545=6, srv853753933=29, srv1974541008=14, srv524946611=21, srv672010843=25, srv1255036854=5, srv172176481=9, srv1206918878=4, srv319972043=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-02 06:56:06,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-02 06:56:06,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-02 06:56:06,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-02 06:56:06,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-02 06:56:06,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-02 06:56:06,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-02 06:56:06,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-02 06:56:06,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-02 06:56:06,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-02 06:56:06,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-02 06:56:06,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-02 06:56:06,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-02 06:56:06,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-02 06:56:06,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-02 06:56:06,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-02 06:56:06,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-02 06:56:06,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-02 06:56:06,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-02 06:56:06,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-02 06:56:06,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-02 06:56:06,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-02 06:56:06,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-02 06:56:06,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-02 06:56:06,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-02 06:56:06,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-02 06:56:06,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-02 06:56:06,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-02 06:56:06,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-02 06:56:06,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-02 06:56:06,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-02 06:56:06,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-02 06:56:06,406 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-02 06:56:06,407 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1063428944:90 , srv1146471562:90 , srv1188002290:90 , srv1188702407:90 , srv1206918878:90 , srv1255036854:90 , srv1500559545:90 , srv1575424515:90 , srv1619151766:90 , srv172176481:90 , srv1728825982:90 , srv1860780602:90 , srv18679647:90 , srv1868763968:90 , srv1974541008:90 , srv1982489283:90 , srv270810695:90 , srv319972043:90 , srv332307323:90 , srv338710905:90 , srv370218036:90 , srv524946611:90 , srv573904598:90 , srv608456763:90 , srv652680052:90 , srv672010843:90 , srv704190385:90 , srv781156605:90 , srv847402573:90 , srv853753933:90 }
2023-03-02 06:56:06,424 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=176 (was 353), ProcessCount=171 (was 173), AvailableMemoryMB=8082 (was 8274)