2023-02-21 09:53:59,391 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-02-21 09:53:59,768 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-02-21 09:53:59,874 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-02-21 09:53:59,932 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-02-21 09:53:59,939 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-02-21 09:53:59,952 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=290, ProcessCount=172, AvailableMemoryMB=6498
2023-02-21 09:53:59,954 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-02-21 09:53:59,954 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-02-21 09:54:00,100 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1033127083:84 , srv1040072596:84 , srv1046378343:84 , srv1048022878:84 , srv1067535709:84 , srv108041406:84 , srv1086759037:84 , srv1116734808:84 , srv1141845767:84 , srv1150437171:84 , srv11816253:84 , srv1278398247:84 , srv12880647:84 , srv1305805869:84 , srv1388687618:84 , srv1404623177:84 , srv1419620937:84 , srv1432519491:84 , srv1440902069:84 , srv1442089361:84 , srv1475063614:84 , srv1518081479:84 , srv1525711323:84 , srv1536168191:84 , srv1629118824:84 , srv1642970124:84 , srv1691449627:84 , srv1709272146:84 , srv1714163568:84 , srv1753976863:84 , srv1757327832:84 , srv1794402077:84 , srv1830880229:84 , srv1852923797:84 , srv1862911585:84 , srv188622263:84 , srv1892898021:84 , srv1967211331:84 , srv1969675133:84 , srv1988575584:84 , srv1993841740:84 , srv2011346028:84 , srv2018912865:84 , srv2091712995:84 , srv231305596:84 , srv2369039:84 , srv251369477:84 , srv267891068:84 , srv301430358:84 , srv324690617:84 , srv330927389:84 , srv336806950:84 , srv374837098:84 , srv388704700:84 , srv39747204:84 , srv401361551:84 , srv420769249:84 , srv426268502:84 , srv43029551:84 , srv444006755:84 , srv464649545:84 , srv472277825:84 , srv474380741:84 , srv487140294:84 , srv528431212:84 , srv532019047:84 , srv551358337:84 , srv56095266:84 , srv584466614:84 , srv597923345:84 , srv606044527:84 , srv610764073:84 , srv619061738:84 , srv639764611:84 , srv643996760:84 , srv682437451:84 , srv696264301:84 , srv712676785:84 , srv713795812:84 , srv719829769:84 , srv733105478:84 , srv749269167:84 , srv776277325:84 , srv781139736:84 , srv783647240:84 , srv792671162:84 , srv808116630:84 , srv833405663:84 , srv842104200:84 , srv84495635:84 , srv870482057:84 , srv891990091:84 , srv898138404:84 , srv922658262:84 , srv923297024:84 , srv930995669:84 , srv952750765:84 , srv975194725:84 , srv988717341:84 , srv2041397885:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-02-21 09:54:00,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-02-21 09:54:00,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv584466614=69, srv792671162=86, srv1278398247=11, srv336806950=52, srv922658262=94, srv808116630=87, srv1150437171=9, srv619061738=73, srv231305596=45, srv682437451=76, srv267891068=48, srv1993841740=40, srv1141845767=8, srv2369039=46, srv56095266=68, srv975194725=98, srv426268502=58, srv444006755=60, srv776277325=83, srv1714163568=28, srv842104200=89, srv1892898021=36, srv2011346028=41, srv930995669=96, srv1988575584=39, srv84495635=90, srv1757327832=30, srv528431212=65, srv1536168191=23, srv1629118824=24, srv1852923797=33, srv301430358=49, srv472277825=62, srv988717341=99, srv597923345=70, srv891990091=92, srv719829769=80, srv474380741=63, srv783647240=85, srv898138404=93, srv923297024=95, srv1388687618=14, srv330927389=51, srv11816253=10, srv2091712995=44, srv388704700=54, srv464649545=61, srv781139736=84, srv251369477=47, srv1709272146=27, srv1305805869=13, srv639764611=74, srv108041406=5, srv696264301=77, srv1404623177=15, srv1642970124=25, srv1442089361=19, srv2041397885=43, srv733105478=81, srv1086759037=6, srv749269167=82, srv1033127083=0, srv1753976863=29, srv1967211331=37, srv1048022878=3, srv12880647=12, srv1518081479=21, srv1969675133=38, srv43029551=59, srv610764073=72, srv532019047=66, srv833405663=88, srv487140294=64, srv643996760=75, srv1067535709=4, srv39747204=55, srv606044527=71, srv1862911585=34, srv401361551=56, srv712676785=78, srv1040072596=1, srv1046378343=2, srv952750765=97, srv1116734808=7, srv1440902069=18, srv1691449627=26, srv1794402077=31, srv374837098=53, srv1419620937=16, srv551358337=67, srv1475063614=20, srv1432519491=17, srv420769249=57, srv188622263=35, srv713795812=79, srv2018912865=42, srv1830880229=32, srv324690617=50, srv870482057=91, srv1525711323=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-02-21 09:54:00,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-02-21 09:54:00,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-02-21 09:54:00,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-02-21 09:54:00,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-02-21 09:54:00,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-02-21 09:54:00,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-02-21 09:54:00,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-02-21 09:54:00,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-02-21 09:54:00,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-02-21 09:54:00,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-02-21 09:54:00,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-02-21 09:54:00,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-02-21 09:54:00,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-02-21 09:54:00,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-02-21 09:54:00,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-02-21 09:54:00,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-02-21 09:54:00,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-02-21 09:54:00,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-02-21 09:54:00,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-02-21 09:54:00,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-02-21 09:54:00,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-02-21 09:54:00,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-02-21 09:54:00,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-02-21 09:54:00,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-02-21 09:54:00,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-02-21 09:54:00,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-02-21 09:54:00,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-02-21 09:54:00,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-02-21 09:54:00,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-02-21 09:54:00,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-02-21 09:54:00,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-02-21 09:54:00,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-02-21 09:54:00,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-02-21 09:54:00,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-02-21 09:54:00,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-02-21 09:54:00,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-02-21 09:54:00,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-02-21 09:54:00,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-02-21 09:54:00,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-02-21 09:54:00,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-02-21 09:54:00,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-02-21 09:54:00,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-02-21 09:54:00,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-02-21 09:54:00,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-02-21 09:54:00,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-02-21 09:54:00,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-02-21 09:54:00,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-02-21 09:54:00,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-02-21 09:54:00,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-02-21 09:54:00,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-02-21 09:54:00,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-02-21 09:54:00,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-02-21 09:54:00,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-02-21 09:54:00,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-02-21 09:54:00,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-02-21 09:54:00,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-02-21 09:54:00,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-02-21 09:54:00,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-02-21 09:54:00,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-02-21 09:54:00,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-02-21 09:54:00,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-02-21 09:54:00,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-02-21 09:54:00,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-02-21 09:54:00,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-02-21 09:54:00,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-02-21 09:54:00,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-02-21 09:54:00,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-02-21 09:54:00,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-02-21 09:54:00,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-02-21 09:54:00,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-02-21 09:54:00,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-02-21 09:54:00,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-02-21 09:54:00,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-02-21 09:54:00,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-02-21 09:54:00,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-02-21 09:54:00,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-02-21 09:54:00,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-02-21 09:54:00,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-02-21 09:54:00,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-02-21 09:54:00,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-02-21 09:54:00,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-02-21 09:54:00,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-02-21 09:54:00,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-02-21 09:54:00,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-02-21 09:54:00,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-02-21 09:54:00,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-02-21 09:54:00,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-02-21 09:54:00,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-02-21 09:54:00,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-02-21 09:54:00,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-02-21 09:54:00,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-02-21 09:54:00,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-02-21 09:54:00,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-02-21 09:54:00,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-02-21 09:54:00,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-02-21 09:54:00,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-02-21 09:54:00,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-02-21 09:54:00,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-02-21 09:54:00,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-02-21 09:54:00,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-02-21 09:54:00,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-02-21 09:54:00,567 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-21 09:54:00,568 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-02-21 09:54:00,568 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-02-21 09:54:01,267 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1000 ms to try 5000 different iterations.  Found a solution that moves 4699 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17955346127538335. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039262117931625484, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05419961442911629, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5221111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039262117931625484, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.175, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23925, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-21 09:54:01,291 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1388687618:80 , srv1048022878:82 , srv1442089361:82 , srv464649545:82 , srv891990091:82 , srv108041406:83 , srv12880647:83 , srv1404623177:83 , srv1753976863:83 , srv1757327832:83 , srv1892898021:83 , srv1969675133:83 , srv551358337:83 , srv56095266:83 , srv1046378343:84 , srv1629118824:84 , srv1642970124:84 , srv267891068:84 , srv696264301:84 , srv749269167:84 , srv842104200:84 , srv930995669:84 , srv1086759037:85 , srv11816253:85 , srv1830880229:85 , srv2091712995:85 , srv324690617:85 , srv388704700:85 , srv444006755:85 , srv472277825:85 , srv597923345:85 , srv719829769:85 , srv792671162:85 , srv84495635:85 , srv1518081479:86 , srv1525711323:86 , srv1536168191:86 , srv188622263:86 , srv1993841740:86 , srv2018912865:86 , srv2369039:86 , srv401361551:86 , srv420769249:86 , srv43029551:86 , srv474380741:86 , srv682437451:86 , srv733105478:86 , srv833405663:86 , srv1141845767:87 , srv1150437171:87 , srv1432519491:87 , srv1714163568:87 , srv2011346028:87 , srv231305596:87 , srv301430358:87 , srv426268502:87 , srv487140294:87 , srv639764611:87 , srv781139736:87 , srv808116630:87 , srv952750765:87 , srv1040072596:88 , srv1116734808:88 , srv1278398247:88 , srv1305805869:88 , srv1691449627:88 , srv1862911585:88 , srv1967211331:88 , srv336806950:88 , srv528431212:88 , srv532019047:88 , srv584466614:88 , srv643996760:88 , srv713795812:88 , srv783647240:88 , srv870482057:88 , srv898138404:88 , srv923297024:88 , srv988717341:88 , srv1067535709:89 , srv1419620937:89 , srv1794402077:89 , srv374837098:89 , srv39747204:89 , srv606044527:89 , srv610764073:89 , srv776277325:89 , srv922658262:89 , srv1440902069:90 , srv1475063614:90 , srv1709272146:90 , srv1852923797:90 , srv1988575584:90 , srv251369477:90 , srv619061738:90 , srv975194725:90 , srv330927389:91 , srv712676785:91 , srv1033127083:92 , srv2041397885:439 }
2023-02-21 09:54:01,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-02-21 09:54:01,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv584466614=69, srv792671162=86, srv1278398247=11, srv336806950=52, srv922658262=94, srv808116630=87, srv1150437171=9, srv619061738=73, srv231305596=45, srv682437451=76, srv267891068=48, srv1993841740=40, srv1141845767=8, srv2369039=46, srv56095266=68, srv975194725=98, srv426268502=58, srv444006755=60, srv776277325=83, srv1714163568=28, srv842104200=89, srv1892898021=36, srv2011346028=41, srv930995669=96, srv1988575584=39, srv84495635=90, srv1757327832=30, srv528431212=65, srv1536168191=23, srv1629118824=24, srv1852923797=33, srv301430358=49, srv472277825=62, srv988717341=99, srv597923345=70, srv891990091=92, srv719829769=80, srv474380741=63, srv783647240=85, srv898138404=93, srv923297024=95, srv1388687618=14, srv330927389=51, srv11816253=10, srv2091712995=44, srv388704700=54, srv464649545=61, srv781139736=84, srv251369477=47, srv1709272146=27, srv1305805869=13, srv639764611=74, srv108041406=5, srv696264301=77, srv1404623177=15, srv1642970124=25, srv1442089361=19, srv2041397885=43, srv733105478=81, srv1086759037=6, srv749269167=82, srv1033127083=0, srv1753976863=29, srv1967211331=37, srv1048022878=3, srv12880647=12, srv1518081479=21, srv1969675133=38, srv43029551=59, srv610764073=72, srv532019047=66, srv833405663=88, srv487140294=64, srv643996760=75, srv1067535709=4, srv39747204=55, srv606044527=71, srv1862911585=34, srv401361551=56, srv712676785=78, srv1040072596=1, srv1046378343=2, srv952750765=97, srv1116734808=7, srv1440902069=18, srv1691449627=26, srv1794402077=31, srv374837098=53, srv1419620937=16, srv551358337=67, srv1475063614=20, srv1432519491=17, srv420769249=57, srv188622263=35, srv713795812=79, srv2018912865=42, srv1830880229=32, srv324690617=50, srv870482057=91, srv1525711323=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-02-21 09:54:01,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-02-21 09:54:01,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-02-21 09:54:01,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-02-21 09:54:01,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-02-21 09:54:01,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-02-21 09:54:01,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-02-21 09:54:01,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-02-21 09:54:01,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-02-21 09:54:01,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-02-21 09:54:01,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-02-21 09:54:01,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-02-21 09:54:01,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-02-21 09:54:01,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-02-21 09:54:01,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-02-21 09:54:01,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-02-21 09:54:01,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-02-21 09:54:01,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-02-21 09:54:01,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-02-21 09:54:01,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-02-21 09:54:01,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-02-21 09:54:01,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-02-21 09:54:01,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-02-21 09:54:01,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-02-21 09:54:01,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-02-21 09:54:01,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-02-21 09:54:01,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-02-21 09:54:01,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-02-21 09:54:01,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-02-21 09:54:01,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-02-21 09:54:01,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-02-21 09:54:01,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-02-21 09:54:01,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-02-21 09:54:01,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-02-21 09:54:01,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-02-21 09:54:01,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-02-21 09:54:01,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-02-21 09:54:01,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-02-21 09:54:01,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-02-21 09:54:01,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-02-21 09:54:01,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-02-21 09:54:01,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-02-21 09:54:01,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-02-21 09:54:01,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-02-21 09:54:01,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-02-21 09:54:01,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-02-21 09:54:01,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-02-21 09:54:01,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-02-21 09:54:01,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-02-21 09:54:01,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-02-21 09:54:01,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-02-21 09:54:01,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-02-21 09:54:01,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-02-21 09:54:01,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-02-21 09:54:01,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-02-21 09:54:01,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-02-21 09:54:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-02-21 09:54:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-02-21 09:54:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-02-21 09:54:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-02-21 09:54:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-02-21 09:54:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-02-21 09:54:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-02-21 09:54:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-02-21 09:54:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-02-21 09:54:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-02-21 09:54:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-02-21 09:54:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-02-21 09:54:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-02-21 09:54:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-02-21 09:54:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-02-21 09:54:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-02-21 09:54:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-02-21 09:54:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-02-21 09:54:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-02-21 09:54:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-02-21 09:54:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-02-21 09:54:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-02-21 09:54:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-02-21 09:54:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-02-21 09:54:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-02-21 09:54:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-02-21 09:54:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-02-21 09:54:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-02-21 09:54:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-02-21 09:54:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-02-21 09:54:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-02-21 09:54:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-02-21 09:54:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-02-21 09:54:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-02-21 09:54:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-02-21 09:54:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-02-21 09:54:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-02-21 09:54:01,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-02-21 09:54:01,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-02-21 09:54:01,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-02-21 09:54:01,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-02-21 09:54:01,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-02-21 09:54:01,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-02-21 09:54:01,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-02-21 09:54:01,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-02-21 09:54:01,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-02-21 09:54:01,476 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039262117931625484, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05419961442911629, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039262117931625484, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.175, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23925, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-21 09:54:01,476 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-02-21 09:54:01,477 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17952055817413126, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039262117931625484, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05419961442911629, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039262117931625484, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.175, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23925, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-02-21 09:54:01,788 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 447 ms to try 5000 different iterations.  Found a solution that moves 2682 regions; Going from a computed imbalance of 0.17952055817413126 to a new imbalance of 0.015368391417235198. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02114740810206044); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03401656690193016, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.298, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02114740810206044); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044333333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-21 09:54:01,799 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1475063614:80 , srv1852923797:80 , srv1440902069:81 , srv1988575584:81 , srv330927389:81 , srv712676785:81 , srv1033127083:82 , srv1709272146:82 , srv251369477:82 , srv374837098:82 , srv606044527:82 , srv1067535709:83 , srv1116734808:83 , srv1150437171:83 , srv231305596:83 , srv301430358:83 , srv39747204:83 , srv584466614:83 , srv619061738:83 , srv776277325:83 , srv783647240:83 , srv870482057:83 , srv975194725:83 , srv1141845767:84 , srv1278398247:84 , srv1419620937:84 , srv1691449627:84 , srv1714163568:84 , srv1794402077:84 , srv1862911585:84 , srv2011346028:84 , srv336806950:84 , srv528431212:84 , srv532019047:84 , srv713795812:84 , srv781139736:84 , srv808116630:84 , srv833405663:84 , srv922658262:84 , srv1305805869:85 , srv1432519491:85 , srv1518081479:85 , srv1525711323:85 , srv188622263:85 , srv1967211331:85 , srv2018912865:85 , srv426268502:85 , srv487140294:85 , srv597923345:85 , srv610764073:85 , srv639764611:85 , srv898138404:85 , srv2091712995:86 , srv388704700:86 , srv420769249:86 , srv474380741:86 , srv643996760:86 , srv682437451:86 , srv952750765:86 , srv988717341:86 , srv1040072596:87 , srv1046378343:87 , srv1404623177:87 , srv1536168191:87 , srv1642970124:87 , srv1757327832:87 , srv1993841740:87 , srv2369039:87 , srv324690617:87 , srv444006755:87 , srv472277825:87 , srv733105478:87 , srv842104200:87 , srv923297024:87 , srv1086759037:88 , srv11816253:88 , srv1629118824:88 , srv43029551:88 , srv56095266:88 , srv696264301:88 , srv719829769:88 , srv792671162:88 , srv84495635:88 , srv108041406:89 , srv12880647:89 , srv1442089361:89 , srv1830880229:89 , srv1892898021:89 , srv267891068:89 , srv401361551:89 , srv464649545:89 , srv1048022878:90 , srv1969675133:90 , srv749269167:90 , srv891990091:90 , srv1753976863:91 , srv930995669:91 , srv551358337:92 , srv1388687618:94 , srv2041397885:523 }
2023-02-21 09:54:01,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-02-21 09:54:01,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv584466614=69, srv792671162=86, srv1278398247=11, srv336806950=52, srv922658262=94, srv808116630=87, srv1150437171=9, srv619061738=73, srv231305596=45, srv682437451=76, srv267891068=48, srv1993841740=40, srv1141845767=8, srv2369039=46, srv56095266=68, srv975194725=98, srv426268502=58, srv444006755=60, srv776277325=83, srv1714163568=28, srv842104200=89, srv1892898021=36, srv2011346028=41, srv930995669=96, srv1988575584=39, srv84495635=90, srv1757327832=30, srv528431212=65, srv1536168191=23, srv1629118824=24, srv1852923797=33, srv301430358=49, srv472277825=62, srv988717341=99, srv597923345=70, srv891990091=92, srv719829769=80, srv474380741=63, srv783647240=85, srv898138404=93, srv923297024=95, srv1388687618=14, srv330927389=51, srv11816253=10, srv2091712995=44, srv388704700=54, srv464649545=61, srv781139736=84, srv251369477=47, srv1709272146=27, srv1305805869=13, srv639764611=74, srv108041406=5, srv696264301=77, srv1404623177=15, srv1642970124=25, srv1442089361=19, srv2041397885=43, srv733105478=81, srv1086759037=6, srv749269167=82, srv1033127083=0, srv1753976863=29, srv1967211331=37, srv1048022878=3, srv12880647=12, srv1518081479=21, srv1969675133=38, srv43029551=59, srv610764073=72, srv532019047=66, srv833405663=88, srv487140294=64, srv643996760=75, srv1067535709=4, srv39747204=55, srv606044527=71, srv1862911585=34, srv401361551=56, srv712676785=78, srv1040072596=1, srv1046378343=2, srv952750765=97, srv1116734808=7, srv1440902069=18, srv1691449627=26, srv1794402077=31, srv374837098=53, srv1419620937=16, srv551358337=67, srv1475063614=20, srv1432519491=17, srv420769249=57, srv188622263=35, srv713795812=79, srv2018912865=42, srv1830880229=32, srv324690617=50, srv870482057=91, srv1525711323=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-02-21 09:54:01,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-02-21 09:54:01,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-02-21 09:54:01,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-02-21 09:54:01,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-02-21 09:54:01,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-02-21 09:54:01,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-02-21 09:54:01,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-02-21 09:54:01,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-02-21 09:54:01,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-02-21 09:54:01,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-02-21 09:54:01,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-02-21 09:54:01,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-02-21 09:54:01,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-02-21 09:54:01,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-02-21 09:54:01,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-02-21 09:54:01,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-02-21 09:54:01,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-02-21 09:54:01,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-02-21 09:54:01,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-02-21 09:54:01,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-02-21 09:54:01,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-02-21 09:54:01,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-02-21 09:54:01,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-02-21 09:54:01,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-02-21 09:54:01,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-02-21 09:54:01,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-02-21 09:54:01,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-02-21 09:54:01,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-02-21 09:54:01,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-02-21 09:54:01,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-02-21 09:54:01,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-02-21 09:54:01,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-02-21 09:54:01,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-02-21 09:54:01,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-02-21 09:54:01,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-02-21 09:54:01,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-02-21 09:54:01,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-02-21 09:54:01,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-02-21 09:54:01,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-02-21 09:54:01,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-02-21 09:54:01,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-02-21 09:54:01,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-02-21 09:54:01,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-02-21 09:54:01,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-02-21 09:54:01,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-02-21 09:54:01,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-02-21 09:54:01,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-02-21 09:54:01,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-02-21 09:54:01,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-02-21 09:54:01,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-02-21 09:54:01,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-02-21 09:54:01,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-02-21 09:54:01,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-02-21 09:54:01,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-02-21 09:54:01,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-02-21 09:54:01,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-02-21 09:54:01,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-02-21 09:54:01,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-02-21 09:54:01,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-02-21 09:54:01,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-02-21 09:54:01,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-02-21 09:54:01,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-02-21 09:54:01,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-02-21 09:54:01,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-02-21 09:54:01,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-02-21 09:54:01,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-02-21 09:54:01,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-02-21 09:54:01,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-02-21 09:54:01,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-02-21 09:54:01,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-02-21 09:54:01,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-02-21 09:54:01,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-02-21 09:54:01,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-02-21 09:54:01,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-02-21 09:54:01,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-02-21 09:54:01,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-02-21 09:54:01,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-02-21 09:54:01,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-02-21 09:54:01,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-02-21 09:54:01,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-02-21 09:54:01,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-02-21 09:54:01,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-02-21 09:54:01,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-02-21 09:54:01,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-02-21 09:54:01,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-02-21 09:54:01,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-02-21 09:54:01,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-02-21 09:54:01,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-02-21 09:54:01,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-02-21 09:54:01,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-02-21 09:54:01,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-02-21 09:54:01,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-02-21 09:54:01,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-02-21 09:54:01,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-02-21 09:54:01,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-02-21 09:54:01,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-02-21 09:54:01,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-02-21 09:54:01,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-02-21 09:54:01,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-02-21 09:54:01,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-02-21 09:54:01,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-02-21 09:54:01,969 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02114740810206044); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03401656690193016, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02114740810206044); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044333333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-21 09:54:01,969 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-02-21 09:54:01,969 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015349611651847223, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02114740810206044); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03401656690193016, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02114740810206044); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044333333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-02-21 09:54:02,216 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 357 ms to try 5000 different iterations.  Found a solution that moves 953 regions; Going from a computed imbalance of 0.015349611651847223 to a new imbalance of 6.731733552042238E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005554433107709862); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007122465682345129); MoveCostFunction : (multiplier=7.0, imbalance=0.1058888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005554433107709862); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-21 09:54:02,218 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv401361551:82 , srv551358337:82 , srv1033127083:83 , srv1040072596:83 , srv487140294:83 , srv643996760:83 , srv749269167:83 , srv923297024:83 , srv930995669:83 , srv11816253:84 , srv1388687618:84 , srv1419620937:84 , srv1536168191:84 , srv1794402077:84 , srv1830880229:84 , srv1892898021:84 , srv1967211331:84 , srv1993841740:84 , srv610764073:84 , srv952750765:84 , srv975194725:84 , srv988717341:84 , srv1048022878:85 , srv108041406:85 , srv1086759037:85 , srv1278398247:85 , srv12880647:85 , srv1305805869:85 , srv1629118824:85 , srv1709272146:85 , srv1753976863:85 , srv1969675133:85 , srv251369477:85 , srv267891068:85 , srv324690617:85 , srv39747204:85 , srv426268502:85 , srv43029551:85 , srv444006755:85 , srv472277825:85 , srv639764611:85 , srv682437451:85 , srv696264301:85 , srv712676785:85 , srv719829769:85 , srv776277325:85 , srv792671162:85 , srv84495635:85 , srv891990091:85 , srv898138404:85 , srv922658262:85 , srv1046378343:86 , srv1067535709:86 , srv1116734808:86 , srv1141845767:86 , srv1432519491:86 , srv1442089361:86 , srv1525711323:86 , srv1691449627:86 , srv1862911585:86 , srv188622263:86 , srv1988575584:86 , srv2011346028:86 , srv2091712995:86 , srv2369039:86 , srv336806950:86 , srv374837098:86 , srv420769249:86 , srv474380741:86 , srv528431212:86 , srv532019047:86 , srv584466614:86 , srv606044527:86 , srv619061738:86 , srv733105478:86 , srv808116630:86 , srv842104200:86 , srv870482057:86 , srv1404623177:87 , srv1440902069:87 , srv1475063614:87 , srv1518081479:87 , srv1642970124:87 , srv1714163568:87 , srv1852923797:87 , srv2018912865:87 , srv231305596:87 , srv330927389:87 , srv388704700:87 , srv464649545:87 , srv56095266:87 , srv597923345:87 , srv713795812:87 , srv781139736:87 , srv783647240:87 , srv833405663:87 , srv1150437171:88 , srv1757327832:88 , srv301430358:89 , srv2041397885:545 }
2023-02-21 09:54:02,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-02-21 09:54:02,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv584466614=69, srv792671162=86, srv1278398247=11, srv336806950=52, srv922658262=94, srv808116630=87, srv1150437171=9, srv619061738=73, srv231305596=45, srv682437451=76, srv267891068=48, srv1993841740=40, srv1141845767=8, srv2369039=46, srv56095266=68, srv975194725=98, srv426268502=58, srv444006755=60, srv776277325=83, srv1714163568=28, srv842104200=89, srv1892898021=36, srv2011346028=41, srv930995669=96, srv1988575584=39, srv84495635=90, srv1757327832=30, srv528431212=65, srv1536168191=23, srv1629118824=24, srv1852923797=33, srv301430358=49, srv472277825=62, srv988717341=99, srv597923345=70, srv891990091=92, srv719829769=80, srv474380741=63, srv783647240=85, srv898138404=93, srv923297024=95, srv1388687618=14, srv330927389=51, srv11816253=10, srv2091712995=44, srv388704700=54, srv464649545=61, srv781139736=84, srv251369477=47, srv1709272146=27, srv1305805869=13, srv639764611=74, srv108041406=5, srv696264301=77, srv1404623177=15, srv1642970124=25, srv1442089361=19, srv2041397885=43, srv733105478=81, srv1086759037=6, srv749269167=82, srv1033127083=0, srv1753976863=29, srv1967211331=37, srv1048022878=3, srv12880647=12, srv1518081479=21, srv1969675133=38, srv43029551=59, srv610764073=72, srv532019047=66, srv833405663=88, srv487140294=64, srv643996760=75, srv1067535709=4, srv39747204=55, srv606044527=71, srv1862911585=34, srv401361551=56, srv712676785=78, srv1040072596=1, srv1046378343=2, srv952750765=97, srv1116734808=7, srv1440902069=18, srv1691449627=26, srv1794402077=31, srv374837098=53, srv1419620937=16, srv551358337=67, srv1475063614=20, srv1432519491=17, srv420769249=57, srv188622263=35, srv713795812=79, srv2018912865=42, srv1830880229=32, srv324690617=50, srv870482057=91, srv1525711323=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-02-21 09:54:02,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-02-21 09:54:02,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-02-21 09:54:02,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-02-21 09:54:02,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-02-21 09:54:02,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-02-21 09:54:02,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-02-21 09:54:02,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-02-21 09:54:02,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-02-21 09:54:02,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-02-21 09:54:02,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-02-21 09:54:02,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-02-21 09:54:02,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-02-21 09:54:02,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-02-21 09:54:02,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-02-21 09:54:02,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-02-21 09:54:02,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-02-21 09:54:02,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-02-21 09:54:02,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-02-21 09:54:02,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-02-21 09:54:02,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-02-21 09:54:02,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-02-21 09:54:02,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-02-21 09:54:02,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-02-21 09:54:02,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-02-21 09:54:02,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-02-21 09:54:02,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-02-21 09:54:02,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-02-21 09:54:02,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-02-21 09:54:02,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-02-21 09:54:02,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-02-21 09:54:02,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-02-21 09:54:02,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-02-21 09:54:02,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-02-21 09:54:02,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-02-21 09:54:02,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-02-21 09:54:02,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-02-21 09:54:02,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-02-21 09:54:02,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-02-21 09:54:02,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-02-21 09:54:02,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-02-21 09:54:02,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-02-21 09:54:02,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-02-21 09:54:02,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-02-21 09:54:02,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-02-21 09:54:02,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-02-21 09:54:02,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-02-21 09:54:02,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-02-21 09:54:02,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-02-21 09:54:02,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-02-21 09:54:02,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-02-21 09:54:02,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-02-21 09:54:02,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-02-21 09:54:02,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-02-21 09:54:02,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-02-21 09:54:02,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-02-21 09:54:02,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-02-21 09:54:02,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-02-21 09:54:02,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-02-21 09:54:02,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-02-21 09:54:02,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-02-21 09:54:02,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-02-21 09:54:02,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-02-21 09:54:02,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-02-21 09:54:02,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-02-21 09:54:02,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-02-21 09:54:02,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-02-21 09:54:02,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-02-21 09:54:02,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-02-21 09:54:02,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-02-21 09:54:02,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-02-21 09:54:02,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-02-21 09:54:02,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-02-21 09:54:02,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-02-21 09:54:02,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-02-21 09:54:02,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-02-21 09:54:02,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-02-21 09:54:02,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-02-21 09:54:02,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-02-21 09:54:02,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-02-21 09:54:02,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-02-21 09:54:02,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-02-21 09:54:02,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-02-21 09:54:02,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-02-21 09:54:02,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-02-21 09:54:02,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-02-21 09:54:02,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-02-21 09:54:02,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-02-21 09:54:02,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-02-21 09:54:02,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-02-21 09:54:02,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-02-21 09:54:02,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-02-21 09:54:02,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-02-21 09:54:02,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-02-21 09:54:02,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-02-21 09:54:02,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-02-21 09:54:02,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-02-21 09:54:02,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-02-21 09:54:02,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-02-21 09:54:02,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-02-21 09:54:02,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-02-21 09:54:02,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-02-21 09:54:02,369 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005554433107709862); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007122465682345129); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005554433107709862); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-21 09:54:02,370 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-02-21 09:54:02,370 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.665003065782957E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005554433107709862); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007122465682345129); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005554433107709862); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-02-21 09:54:02,766 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 507 ms to try 5000 different iterations.  Found a solution that moves 224 regions; Going from a computed imbalance of 6.665003065782957E-4 to a new imbalance of 1.2374160930837792E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.024888888888888887); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-21 09:54:02,767 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1753976863:83 , srv2369039:83 , srv301430358:83 , srv330927389:83 , srv43029551:83 , srv733105478:83 , srv988717341:83 , srv1040072596:84 , srv1067535709:84 , srv1150437171:84 , srv1305805869:84 , srv1388687618:84 , srv1404623177:84 , srv1432519491:84 , srv1440902069:84 , srv1518081479:84 , srv1525711323:84 , srv1642970124:84 , srv1691449627:84 , srv1714163568:84 , srv1757327832:84 , srv1830880229:84 , srv1862911585:84 , srv1969675133:84 , srv2091712995:84 , srv267891068:84 , srv336806950:84 , srv388704700:84 , srv401361551:84 , srv420769249:84 , srv444006755:84 , srv464649545:84 , srv474380741:84 , srv528431212:84 , srv532019047:84 , srv610764073:84 , srv619061738:84 , srv713795812:84 , srv719829769:84 , srv781139736:84 , srv783647240:84 , srv792671162:84 , srv84495635:84 , srv898138404:84 , srv923297024:84 , srv930995669:84 , srv975194725:84 , srv1033127083:85 , srv1046378343:85 , srv1048022878:85 , srv108041406:85 , srv1086759037:85 , srv1116734808:85 , srv1141845767:85 , srv11816253:85 , srv1278398247:85 , srv12880647:85 , srv1419620937:85 , srv1442089361:85 , srv1475063614:85 , srv1536168191:85 , srv1629118824:85 , srv1709272146:85 , srv1794402077:85 , srv1852923797:85 , srv188622263:85 , srv1967211331:85 , srv1988575584:85 , srv1993841740:85 , srv2011346028:85 , srv2018912865:85 , srv231305596:85 , srv251369477:85 , srv324690617:85 , srv374837098:85 , srv39747204:85 , srv472277825:85 , srv551358337:85 , srv56095266:85 , srv584466614:85 , srv597923345:85 , srv606044527:85 , srv639764611:85 , srv643996760:85 , srv682437451:85 , srv696264301:85 , srv712676785:85 , srv749269167:85 , srv776277325:85 , srv808116630:85 , srv833405663:85 , srv842104200:85 , srv870482057:85 , srv891990091:85 , srv922658262:85 , srv952750765:85 , srv1892898021:86 , srv426268502:86 , srv487140294:87 , srv2041397885:635 }
2023-02-21 09:54:02,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-02-21 09:54:02,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv584466614=69, srv792671162=86, srv1278398247=11, srv336806950=52, srv922658262=94, srv808116630=87, srv1150437171=9, srv619061738=73, srv231305596=45, srv682437451=76, srv267891068=48, srv1993841740=40, srv1141845767=8, srv2369039=46, srv56095266=68, srv975194725=98, srv426268502=58, srv444006755=60, srv776277325=83, srv1714163568=28, srv842104200=89, srv1892898021=36, srv2011346028=41, srv930995669=96, srv1988575584=39, srv84495635=90, srv1757327832=30, srv528431212=65, srv1536168191=23, srv1629118824=24, srv1852923797=33, srv301430358=49, srv472277825=62, srv988717341=99, srv597923345=70, srv891990091=92, srv719829769=80, srv474380741=63, srv783647240=85, srv898138404=93, srv923297024=95, srv1388687618=14, srv330927389=51, srv11816253=10, srv2091712995=44, srv388704700=54, srv464649545=61, srv781139736=84, srv251369477=47, srv1709272146=27, srv1305805869=13, srv639764611=74, srv108041406=5, srv696264301=77, srv1404623177=15, srv1642970124=25, srv1442089361=19, srv2041397885=43, srv733105478=81, srv1086759037=6, srv749269167=82, srv1033127083=0, srv1753976863=29, srv1967211331=37, srv1048022878=3, srv12880647=12, srv1518081479=21, srv1969675133=38, srv43029551=59, srv610764073=72, srv532019047=66, srv833405663=88, srv487140294=64, srv643996760=75, srv1067535709=4, srv39747204=55, srv606044527=71, srv1862911585=34, srv401361551=56, srv712676785=78, srv1040072596=1, srv1046378343=2, srv952750765=97, srv1116734808=7, srv1440902069=18, srv1691449627=26, srv1794402077=31, srv374837098=53, srv1419620937=16, srv551358337=67, srv1475063614=20, srv1432519491=17, srv420769249=57, srv188622263=35, srv713795812=79, srv2018912865=42, srv1830880229=32, srv324690617=50, srv870482057=91, srv1525711323=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-02-21 09:54:02,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-02-21 09:54:02,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-02-21 09:54:02,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-02-21 09:54:02,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-02-21 09:54:02,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-02-21 09:54:02,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-02-21 09:54:02,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-02-21 09:54:02,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-02-21 09:54:02,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-02-21 09:54:02,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-02-21 09:54:02,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-02-21 09:54:02,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-02-21 09:54:02,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-02-21 09:54:02,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-02-21 09:54:02,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-02-21 09:54:02,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-02-21 09:54:02,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-02-21 09:54:02,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-02-21 09:54:02,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-02-21 09:54:02,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-02-21 09:54:02,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-02-21 09:54:02,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-02-21 09:54:02,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-02-21 09:54:02,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-02-21 09:54:02,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-02-21 09:54:02,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-02-21 09:54:02,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-02-21 09:54:02,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-02-21 09:54:02,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-02-21 09:54:02,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-02-21 09:54:02,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-02-21 09:54:02,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-02-21 09:54:02,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-02-21 09:54:02,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-02-21 09:54:02,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-02-21 09:54:02,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-02-21 09:54:02,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-02-21 09:54:02,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-02-21 09:54:02,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-02-21 09:54:02,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-02-21 09:54:02,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-02-21 09:54:02,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-02-21 09:54:02,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-02-21 09:54:02,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-02-21 09:54:02,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-02-21 09:54:02,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-02-21 09:54:02,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-02-21 09:54:02,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-02-21 09:54:02,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-02-21 09:54:02,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-02-21 09:54:02,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-02-21 09:54:02,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-02-21 09:54:02,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-02-21 09:54:02,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-02-21 09:54:02,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-02-21 09:54:02,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-02-21 09:54:02,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-02-21 09:54:02,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-02-21 09:54:02,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-02-21 09:54:02,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-02-21 09:54:02,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-02-21 09:54:02,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-02-21 09:54:02,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-02-21 09:54:02,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-02-21 09:54:02,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-02-21 09:54:02,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-02-21 09:54:02,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-02-21 09:54:02,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-02-21 09:54:02,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-02-21 09:54:02,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-02-21 09:54:02,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-02-21 09:54:02,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-02-21 09:54:02,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-02-21 09:54:02,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-02-21 09:54:02,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-02-21 09:54:02,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-02-21 09:54:02,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-02-21 09:54:02,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-02-21 09:54:02,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-02-21 09:54:02,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-02-21 09:54:02,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-02-21 09:54:02,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-02-21 09:54:02,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-02-21 09:54:02,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-02-21 09:54:02,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-02-21 09:54:02,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-02-21 09:54:02,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-02-21 09:54:02,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-02-21 09:54:02,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-02-21 09:54:02,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-02-21 09:54:02,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-02-21 09:54:02,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-02-21 09:54:02,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-02-21 09:54:02,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-02-21 09:54:02,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-02-21 09:54:02,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-02-21 09:54:02,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-02-21 09:54:02,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-02-21 09:54:02,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-02-21 09:54:02,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-02-21 09:54:02,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-02-21 09:54:02,934 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.0805679407009976E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-21 09:54:02,934 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1753976863:83 , srv2369039:83 , srv301430358:83 , srv330927389:83 , srv43029551:83 , srv733105478:83 , srv988717341:83 , srv1040072596:84 , srv1067535709:84 , srv1150437171:84 , srv1305805869:84 , srv1388687618:84 , srv1404623177:84 , srv1432519491:84 , srv1440902069:84 , srv1518081479:84 , srv1525711323:84 , srv1642970124:84 , srv1691449627:84 , srv1714163568:84 , srv1757327832:84 , srv1830880229:84 , srv1862911585:84 , srv1969675133:84 , srv2091712995:84 , srv267891068:84 , srv336806950:84 , srv388704700:84 , srv401361551:84 , srv420769249:84 , srv444006755:84 , srv464649545:84 , srv474380741:84 , srv528431212:84 , srv532019047:84 , srv610764073:84 , srv619061738:84 , srv713795812:84 , srv719829769:84 , srv781139736:84 , srv783647240:84 , srv792671162:84 , srv84495635:84 , srv898138404:84 , srv923297024:84 , srv930995669:84 , srv975194725:84 , srv1033127083:85 , srv1046378343:85 , srv1048022878:85 , srv108041406:85 , srv1086759037:85 , srv1116734808:85 , srv1141845767:85 , srv11816253:85 , srv1278398247:85 , srv12880647:85 , srv1419620937:85 , srv1442089361:85 , srv1475063614:85 , srv1536168191:85 , srv1629118824:85 , srv1709272146:85 , srv1794402077:85 , srv1852923797:85 , srv188622263:85 , srv1967211331:85 , srv1988575584:85 , srv1993841740:85 , srv2011346028:85 , srv2018912865:85 , srv231305596:85 , srv251369477:85 , srv324690617:85 , srv374837098:85 , srv39747204:85 , srv472277825:85 , srv551358337:85 , srv56095266:85 , srv584466614:85 , srv597923345:85 , srv606044527:85 , srv639764611:85 , srv643996760:85 , srv682437451:85 , srv696264301:85 , srv712676785:85 , srv749269167:85 , srv776277325:85 , srv808116630:85 , srv833405663:85 , srv842104200:85 , srv870482057:85 , srv891990091:85 , srv922658262:85 , srv952750765:85 , srv1892898021:86 , srv426268502:86 , srv487140294:87 , srv2041397885:635 }
2023-02-21 09:54:02,961 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=290 (was 290), ProcessCount=172 (was 172), AvailableMemoryMB=5988 (was 6498)
2023-02-21 09:54:02,970 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=290, ProcessCount=172, AvailableMemoryMB=5987
2023-02-21 09:54:02,971 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-02-21 09:54:02,971 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-02-21 09:54:02,980 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1181616645:84 , srv1201761761:84 , srv1258108543:84 , srv1271862879:84 , srv1301732348:84 , srv1338107503:84 , srv1595327634:84 , srv1671041687:84 , srv1860209363:84 , srv1885955492:84 , srv1954963903:84 , srv2066327384:84 , srv2087242858:84 , srv2088513556:84 , srv2097944623:84 , srv2126438278:84 , srv254079896:84 , srv302114197:84 , srv405747085:84 , srv407101779:84 , srv483344983:84 , srv549369101:84 , srv575912915:84 , srv607911972:84 , srv656029469:84 , srv850850006:84 , srv964017368:84 , srv974801339:84 , srv999547023:84 , srv636428632:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-02-21 09:54:02,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-02-21 09:54:02,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv656029469=25, srv1181616645=0, srv1860209363=8, srv964017368=27, srv405747085=18, srv2087242858=12, srv2097944623=14, srv974801339=28, srv2126438278=15, srv1338107503=5, srv549369101=21, srv1201761761=1, srv1595327634=6, srv1954963903=10, srv302114197=17, srv1301732348=4, srv407101779=19, srv607911972=23, srv2066327384=11, srv636428632=24, srv850850006=26, srv1671041687=7, srv999547023=29, srv1258108543=2, srv1885955492=9, srv254079896=16, srv1271862879=3, srv483344983=20, srv2088513556=13, srv575912915=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-02-21 09:54:02,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-02-21 09:54:02,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-02-21 09:54:02,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-02-21 09:54:02,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-02-21 09:54:02,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-02-21 09:54:02,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-02-21 09:54:02,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-02-21 09:54:02,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-02-21 09:54:02,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-02-21 09:54:02,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-02-21 09:54:02,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-02-21 09:54:02,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-02-21 09:54:02,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-02-21 09:54:02,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-02-21 09:54:02,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-02-21 09:54:02,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-02-21 09:54:02,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-02-21 09:54:02,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-02-21 09:54:02,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-02-21 09:54:02,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-02-21 09:54:02,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-02-21 09:54:02,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-02-21 09:54:02,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-02-21 09:54:02,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-02-21 09:54:02,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-02-21 09:54:02,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-02-21 09:54:02,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-02-21 09:54:02,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-02-21 09:54:02,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-02-21 09:54:02,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-02-21 09:54:02,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-02-21 09:54:03,008 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.157883941321729, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-21 09:54:03,008 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912690530407998, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.157883941321729, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-02-21 09:56:02,995 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6705889 different iterations.  Found a solution that moves 1854 regions; Going from a computed imbalance of 0.9912690530407998 to a new imbalance of 4.3273284898463826E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6866666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-21 09:56:03,004 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1181616645:90 , srv1201761761:90 , srv1258108543:90 , srv1271862879:90 , srv1301732348:90 , srv1338107503:90 , srv1595327634:90 , srv1671041687:90 , srv1860209363:90 , srv1885955492:90 , srv1954963903:90 , srv2066327384:90 , srv2087242858:90 , srv2088513556:90 , srv2097944623:90 , srv2126438278:90 , srv254079896:90 , srv302114197:90 , srv405747085:90 , srv407101779:90 , srv483344983:90 , srv549369101:90 , srv575912915:90 , srv607911972:90 , srv636428632:90 , srv656029469:90 , srv850850006:90 , srv964017368:90 , srv974801339:90 , srv999547023:90 }
2023-02-21 09:56:03,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-02-21 09:56:03,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv656029469=25, srv1181616645=0, srv1860209363=8, srv964017368=27, srv405747085=18, srv2087242858=12, srv2097944623=14, srv974801339=28, srv2126438278=15, srv1338107503=5, srv549369101=21, srv1201761761=1, srv1595327634=6, srv1954963903=10, srv302114197=17, srv1301732348=4, srv407101779=19, srv607911972=23, srv2066327384=11, srv636428632=24, srv850850006=26, srv1671041687=7, srv999547023=29, srv1258108543=2, srv1885955492=9, srv254079896=16, srv1271862879=3, srv483344983=20, srv2088513556=13, srv575912915=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-02-21 09:56:03,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-02-21 09:56:03,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-02-21 09:56:03,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-02-21 09:56:03,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-02-21 09:56:03,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-02-21 09:56:03,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-02-21 09:56:03,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-02-21 09:56:03,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-02-21 09:56:03,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-02-21 09:56:03,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-02-21 09:56:03,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-02-21 09:56:03,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-02-21 09:56:03,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-02-21 09:56:03,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-02-21 09:56:03,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-02-21 09:56:03,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-02-21 09:56:03,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-02-21 09:56:03,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-02-21 09:56:03,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-02-21 09:56:03,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-02-21 09:56:03,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-02-21 09:56:03,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-02-21 09:56:03,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-02-21 09:56:03,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-02-21 09:56:03,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-02-21 09:56:03,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-02-21 09:56:03,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-02-21 09:56:03,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-02-21 09:56:03,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-02-21 09:56:03,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-02-21 09:56:03,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-02-21 09:56:03,033 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-21 09:56:03,033 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1181616645:90 , srv1201761761:90 , srv1258108543:90 , srv1271862879:90 , srv1301732348:90 , srv1338107503:90 , srv1595327634:90 , srv1671041687:90 , srv1860209363:90 , srv1885955492:90 , srv1954963903:90 , srv2066327384:90 , srv2087242858:90 , srv2088513556:90 , srv2097944623:90 , srv2126438278:90 , srv254079896:90 , srv302114197:90 , srv405747085:90 , srv407101779:90 , srv483344983:90 , srv549369101:90 , srv575912915:90 , srv607911972:90 , srv636428632:90 , srv656029469:90 , srv850850006:90 , srv964017368:90 , srv974801339:90 , srv999547023:90 }
2023-02-21 09:56:03,049 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=197 (was 290), ProcessCount=172 (was 172), AvailableMemoryMB=5775 (was 5987)