2023-03-28 05:56:02,943 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-03-28 05:56:03,303 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-03-28 05:56:03,431 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-03-28 05:56:03,513 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-28 05:56:03,523 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-28 05:56:03,539 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=343, ProcessCount=170, AvailableMemoryMB=5109
2023-03-28 05:56:03,542 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-28 05:56:03,542 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-28 05:56:03,705 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1021361571:84 , srv1037219583:84 , srv1054239320:84 , srv1089999841:84 , srv1106765323:84 , srv1116753685:84 , srv1128958542:84 , srv1167492920:84 , srv1167857721:84 , srv1173918122:84 , srv118627256:84 , srv1189172488:84 , srv1200260681:84 , srv1214444747:84 , srv1243178876:84 , srv1260970219:84 , srv1273550020:84 , srv1293936126:84 , srv1310659430:84 , srv1312968161:84 , srv1314380919:84 , srv1316110101:84 , srv1321754755:84 , srv1335309357:84 , srv1381036818:84 , srv1382933807:84 , srv1386165651:84 , srv1396549304:84 , srv1405847637:84 , srv1447204430:84 , srv1457312742:84 , srv1490662399:84 , srv1505234773:84 , srv1520890161:84 , srv1540227482:84 , srv1540820400:84 , srv1574057810:84 , srv1608054614:84 , srv1609322071:84 , srv1626310890:84 , srv1629220348:84 , srv165066687:84 , srv1653772430:84 , srv1658310314:84 , srv1665795864:84 , srv16848368:84 , srv1706802283:84 , srv1747165876:84 , srv1822502759:84 , srv1824904467:84 , srv1855970723:84 , srv1857409856:84 , srv1904458506:84 , srv1924675997:84 , srv193946221:84 , srv2077362636:84 , srv2082862237:84 , srv2093159399:84 , srv2100076963:84 , srv2107366898:84 , srv2141951174:84 , srv2142131442:84 , srv2142705611:84 , srv296595934:84 , srv311666110:84 , srv34659052:84 , srv355915984:84 , srv431240307:84 , srv43270481:84 , srv454986971:84 , srv455623035:84 , srv475480428:84 , srv476598549:84 , srv482588120:84 , srv492722670:84 , srv528312206:84 , srv532718043:84 , srv549315097:84 , srv563912759:84 , srv577532550:84 , srv628985673:84 , srv643242439:84 , srv652592287:84 , srv663286005:84 , srv673556121:84 , srv688835030:84 , srv695084789:84 , srv735374154:84 , srv736057304:84 , srv761421142:84 , srv781399995:84 , srv817163316:84 , srv834299625:84 , srv857111501:84 , srv869145969:84 , srv881423995:84 , srv922154542:84 , srv974242954:84 , srv99266360:84 , srv942671864:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-03-28 05:56:03,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-28 05:56:03,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1747165876=47, srv1857409856=51, srv643242439=81, srv1335309357=23, srv922154542=96, srv34659052=65, srv355915984=66, srv834299625=92, srv1574057810=36, srv475480428=71, srv2107366898=59, srv296595934=63, srv1540227482=34, srv431240307=67, srv1447204430=29, srv736057304=88, srv1609322071=38, srv1626310890=39, srv1037219583=1, srv1658310314=43, srv1116753685=5, srv1653772430=42, srv1629220348=40, srv1706802283=46, srv781399995=90, srv1314380919=20, srv942671864=97, srv1540820400=35, srv549315097=77, srv817163316=91, srv1381036818=24, srv1189172488=11, srv1904458506=52, srv2142705611=62, srv1167492920=7, srv455623035=70, srv99266360=99, srv528312206=75, srv1214444747=13, srv1396549304=27, srv1021361571=0, srv1128958542=6, srv1457312742=30, srv577532550=79, srv869145969=94, srv1321754755=22, srv1273550020=16, srv663286005=83, srv1382933807=25, srv1608054614=37, srv673556121=84, srv118627256=10, srv16848368=45, srv628985673=80, srv857111501=93, srv2142131442=61, srv1665795864=44, srv1260970219=15, srv974242954=98, srv2100076963=58, srv1316110101=21, srv881423995=95, srv1200260681=12, srv482588120=73, srv311666110=64, srv1924675997=53, srv193946221=54, srv1520890161=33, srv1173918122=9, srv735374154=87, srv1405847637=28, srv2077362636=55, srv1106765323=4, srv476598549=72, srv532718043=76, srv2141951174=60, srv1054239320=2, srv1310659430=18, srv1293936126=17, srv1312968161=19, srv1243178876=14, srv1505234773=32, srv1855970723=50, srv454986971=69, srv1490662399=31, srv563912759=78, srv1824904467=49, srv688835030=85, srv695084789=86, srv492722670=74, srv1089999841=3, srv2093159399=57, srv1167857721=8, srv761421142=89, srv165066687=41, srv1822502759=48, srv1386165651=26, srv652592287=82, srv2082862237=56, srv43270481=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-28 05:56:03,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-28 05:56:03,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-28 05:56:03,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-28 05:56:03,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-28 05:56:03,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-28 05:56:03,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-28 05:56:03,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-28 05:56:03,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-28 05:56:03,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-28 05:56:03,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-28 05:56:03,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-28 05:56:03,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-28 05:56:03,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-28 05:56:03,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-28 05:56:03,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-28 05:56:03,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-28 05:56:03,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-28 05:56:03,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-28 05:56:03,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-28 05:56:03,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-28 05:56:03,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-28 05:56:03,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-28 05:56:03,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-28 05:56:03,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-28 05:56:03,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-28 05:56:03,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-28 05:56:03,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-28 05:56:03,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-28 05:56:03,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-28 05:56:03,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-28 05:56:03,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-28 05:56:03,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-28 05:56:03,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-28 05:56:03,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-28 05:56:03,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-28 05:56:03,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-28 05:56:03,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-28 05:56:03,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-28 05:56:03,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-28 05:56:03,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-28 05:56:03,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-28 05:56:03,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-28 05:56:03,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-28 05:56:03,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-28 05:56:03,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-28 05:56:03,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-28 05:56:03,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-28 05:56:03,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-28 05:56:03,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-28 05:56:03,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-28 05:56:03,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-28 05:56:03,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-28 05:56:03,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-28 05:56:03,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-28 05:56:03,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-28 05:56:03,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-28 05:56:03,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-28 05:56:03,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-28 05:56:03,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-28 05:56:03,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-28 05:56:03,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-28 05:56:03,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-28 05:56:03,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-28 05:56:03,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-28 05:56:03,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-28 05:56:03,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-28 05:56:03,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-28 05:56:03,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-28 05:56:03,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-28 05:56:03,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-28 05:56:03,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-28 05:56:03,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-28 05:56:03,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-28 05:56:03,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-28 05:56:03,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-28 05:56:03,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-28 05:56:03,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-28 05:56:03,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-28 05:56:03,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-28 05:56:03,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-28 05:56:03,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-28 05:56:03,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-28 05:56:03,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-28 05:56:03,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-28 05:56:03,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-28 05:56:03,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-28 05:56:03,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-28 05:56:03,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-28 05:56:03,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-28 05:56:03,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-28 05:56:03,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-28 05:56:03,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-28 05:56:03,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-28 05:56:03,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-28 05:56:03,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-28 05:56:03,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-28 05:56:03,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-28 05:56:03,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-28 05:56:03,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-28 05:56:03,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-28 05:56:03,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-28 05:56:04,079 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 05:56:04,080 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-28 05:56:04,080 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-28 05:56:04,520 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 637 ms to try 5000 different iterations.  Found a solution that moves 4728 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17678409818975452. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03674090365989728, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0549604489972952, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5253333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03674090365989728, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17166666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24191666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 05:56:04,540 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1128958542:82 , srv1189172488:82 , srv1381036818:82 , srv2082862237:82 , srv881423995:82 , srv118627256:83 , srv1520890161:83 , srv2141951174:83 , srv577532550:83 , srv1243178876:84 , srv1293936126:84 , srv1312968161:84 , srv1316110101:84 , srv1626310890:84 , srv1747165876:84 , srv2142131442:84 , srv296595934:84 , srv43270481:84 , srv475480428:84 , srv528312206:84 , srv532718043:84 , srv563912759:84 , srv817163316:84 , srv974242954:84 , srv1335309357:85 , srv1457312742:85 , srv1490662399:85 , srv1505234773:85 , srv1540820400:85 , srv165066687:85 , srv1665795864:85 , srv1857409856:85 , srv2142705611:85 , srv482588120:85 , srv492722670:85 , srv735374154:85 , srv1089999841:86 , srv1106765323:86 , srv1167492920:86 , srv1200260681:86 , srv1214444747:86 , srv1382933807:86 , srv1822502759:86 , srv193946221:86 , srv2093159399:86 , srv311666110:86 , srv476598549:86 , srv628985673:86 , srv652592287:86 , srv663286005:86 , srv781399995:86 , srv869145969:86 , srv1021361571:87 , srv1054239320:87 , srv1173918122:87 , srv1310659430:87 , srv1321754755:87 , srv1386165651:87 , srv1396549304:87 , srv1405847637:87 , srv1609322071:87 , srv1706802283:87 , srv2100076963:87 , srv355915984:87 , srv455623035:87 , srv549315097:87 , srv673556121:87 , srv736057304:87 , srv857111501:87 , srv1273550020:88 , srv1314380919:88 , srv1540227482:88 , srv1608054614:88 , srv1653772430:88 , srv16848368:88 , srv2077362636:88 , srv454986971:88 , srv688835030:88 , srv1116753685:89 , srv1167857721:89 , srv1629220348:89 , srv1904458506:89 , srv2107366898:89 , srv695084789:89 , srv1037219583:90 , srv1260970219:90 , srv1447204430:90 , srv1824904467:90 , srv834299625:90 , srv1924675997:91 , srv643242439:91 , srv761421142:91 , srv922154542:91 , srv1658310314:92 , srv1574057810:93 , srv1855970723:93 , srv99266360:93 , srv431240307:97 , srv34659052:99 , srv942671864:416 }
2023-03-28 05:56:04,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-28 05:56:04,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1747165876=47, srv1857409856=51, srv643242439=81, srv1335309357=23, srv922154542=96, srv34659052=65, srv355915984=66, srv834299625=92, srv1574057810=36, srv475480428=71, srv2107366898=59, srv296595934=63, srv1540227482=34, srv431240307=67, srv1447204430=29, srv736057304=88, srv1609322071=38, srv1626310890=39, srv1037219583=1, srv1658310314=43, srv1116753685=5, srv1653772430=42, srv1629220348=40, srv1706802283=46, srv781399995=90, srv1314380919=20, srv942671864=97, srv1540820400=35, srv549315097=77, srv817163316=91, srv1381036818=24, srv1189172488=11, srv1904458506=52, srv2142705611=62, srv1167492920=7, srv455623035=70, srv99266360=99, srv528312206=75, srv1214444747=13, srv1396549304=27, srv1021361571=0, srv1128958542=6, srv1457312742=30, srv577532550=79, srv869145969=94, srv1321754755=22, srv1273550020=16, srv663286005=83, srv1382933807=25, srv1608054614=37, srv673556121=84, srv118627256=10, srv16848368=45, srv628985673=80, srv857111501=93, srv2142131442=61, srv1665795864=44, srv1260970219=15, srv974242954=98, srv2100076963=58, srv1316110101=21, srv881423995=95, srv1200260681=12, srv482588120=73, srv311666110=64, srv1924675997=53, srv193946221=54, srv1520890161=33, srv1173918122=9, srv735374154=87, srv1405847637=28, srv2077362636=55, srv1106765323=4, srv476598549=72, srv532718043=76, srv2141951174=60, srv1054239320=2, srv1310659430=18, srv1293936126=17, srv1312968161=19, srv1243178876=14, srv1505234773=32, srv1855970723=50, srv454986971=69, srv1490662399=31, srv563912759=78, srv1824904467=49, srv688835030=85, srv695084789=86, srv492722670=74, srv1089999841=3, srv2093159399=57, srv1167857721=8, srv761421142=89, srv165066687=41, srv1822502759=48, srv1386165651=26, srv652592287=82, srv2082862237=56, srv43270481=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-28 05:56:04,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-28 05:56:04,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-28 05:56:04,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-28 05:56:04,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-28 05:56:04,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-28 05:56:04,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-28 05:56:04,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-28 05:56:04,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-28 05:56:04,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-28 05:56:04,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-28 05:56:04,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-28 05:56:04,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-28 05:56:04,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-28 05:56:04,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-28 05:56:04,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-28 05:56:04,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-28 05:56:04,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-28 05:56:04,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-28 05:56:04,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-28 05:56:04,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-28 05:56:04,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-28 05:56:04,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-28 05:56:04,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-28 05:56:04,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-28 05:56:04,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-28 05:56:04,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-28 05:56:04,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-28 05:56:04,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-28 05:56:04,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-28 05:56:04,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-28 05:56:04,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-28 05:56:04,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-28 05:56:04,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-28 05:56:04,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-28 05:56:04,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-28 05:56:04,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-28 05:56:04,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-28 05:56:04,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-28 05:56:04,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-28 05:56:04,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-28 05:56:04,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-28 05:56:04,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-28 05:56:04,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-28 05:56:04,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-28 05:56:04,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-28 05:56:04,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-28 05:56:04,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-28 05:56:04,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-28 05:56:04,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-28 05:56:04,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-28 05:56:04,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-28 05:56:04,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-28 05:56:04,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-28 05:56:04,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-28 05:56:04,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-28 05:56:04,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-28 05:56:04,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-28 05:56:04,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-28 05:56:04,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-28 05:56:04,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-28 05:56:04,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-28 05:56:04,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-28 05:56:04,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-28 05:56:04,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-28 05:56:04,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-28 05:56:04,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-28 05:56:04,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-28 05:56:04,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-28 05:56:04,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-28 05:56:04,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-28 05:56:04,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-28 05:56:04,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-28 05:56:04,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-28 05:56:04,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-28 05:56:04,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-28 05:56:04,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-28 05:56:04,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-28 05:56:04,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-28 05:56:04,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-28 05:56:04,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-28 05:56:04,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-28 05:56:04,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-28 05:56:04,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-28 05:56:04,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-28 05:56:04,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-28 05:56:04,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-28 05:56:04,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-28 05:56:04,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-28 05:56:04,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-28 05:56:04,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-28 05:56:04,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-28 05:56:04,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-28 05:56:04,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-28 05:56:04,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-28 05:56:04,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-28 05:56:04,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-28 05:56:04,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-28 05:56:04,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-28 05:56:04,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-28 05:56:04,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-28 05:56:04,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-28 05:56:04,753 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03674090365989728, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0549604489972952, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03674090365989728, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17166666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24191666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 05:56:04,753 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-28 05:56:04,754 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1767509920261623, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03674090365989728, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0549604489972952, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03674090365989728, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17166666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24191666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-28 05:56:05,113 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 510 ms to try 5000 different iterations.  Found a solution that moves 2642 regions; Going from a computed imbalance of 0.1767509920261623 to a new imbalance of 0.016057588834705427. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0210640979268992); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03060922742558966, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29355555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0210640979268992); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04633333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 05:56:05,127 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv34659052:73 , srv431240307:75 , srv1574057810:77 , srv1855970723:79 , srv643242439:80 , srv922154542:80 , srv1447204430:81 , srv1629220348:81 , srv1658310314:81 , srv1924675997:81 , srv1037219583:82 , srv1540227482:82 , srv1653772430:82 , srv1824904467:82 , srv1904458506:82 , srv1167492920:83 , srv1321754755:83 , srv16848368:83 , srv2107366898:83 , srv549315097:83 , srv688835030:83 , srv736057304:83 , srv99266360:83 , srv1054239320:84 , srv1167857721:84 , srv1173918122:84 , srv1214444747:84 , srv1405847637:84 , srv1608054614:84 , srv1706802283:84 , srv2077362636:84 , srv355915984:84 , srv454986971:84 , srv695084789:84 , srv1021361571:85 , srv1116753685:85 , srv1200260681:85 , srv1260970219:85 , srv1273550020:85 , srv1609322071:85 , srv1665795864:85 , srv1857409856:85 , srv193946221:85 , srv2093159399:85 , srv2100076963:85 , srv43270481:85 , srv455623035:85 , srv628985673:85 , srv652592287:85 , srv673556121:85 , srv834299625:85 , srv1310659430:86 , srv1314380919:86 , srv1382933807:86 , srv1386165651:86 , srv1540820400:86 , srv165066687:86 , srv2142131442:86 , srv492722670:86 , srv761421142:86 , srv781399995:86 , srv857111501:86 , srv1316110101:87 , srv1335309357:87 , srv1396549304:87 , srv2082862237:87 , srv476598549:87 , srv532718043:87 , srv663286005:87 , srv735374154:87 , srv1106765323:88 , srv118627256:88 , srv1312968161:88 , srv1457312742:88 , srv311666110:88 , srv475480428:88 , srv563912759:88 , srv577532550:88 , srv817163316:88 , srv869145969:88 , srv1089999841:89 , srv1189172488:89 , srv1293936126:89 , srv1520890161:89 , srv1747165876:89 , srv1822502759:89 , srv2141951174:89 , srv296595934:89 , srv482588120:89 , srv528312206:89 , srv1128958542:90 , srv1381036818:90 , srv1490662399:90 , srv1505234773:90 , srv1626310890:90 , srv2142705611:90 , srv974242954:90 , srv1243178876:91 , srv881423995:91 , srv942671864:545 }
2023-03-28 05:56:05,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-28 05:56:05,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1747165876=47, srv1857409856=51, srv643242439=81, srv1335309357=23, srv922154542=96, srv34659052=65, srv355915984=66, srv834299625=92, srv1574057810=36, srv475480428=71, srv2107366898=59, srv296595934=63, srv1540227482=34, srv431240307=67, srv1447204430=29, srv736057304=88, srv1609322071=38, srv1626310890=39, srv1037219583=1, srv1658310314=43, srv1116753685=5, srv1653772430=42, srv1629220348=40, srv1706802283=46, srv781399995=90, srv1314380919=20, srv942671864=97, srv1540820400=35, srv549315097=77, srv817163316=91, srv1381036818=24, srv1189172488=11, srv1904458506=52, srv2142705611=62, srv1167492920=7, srv455623035=70, srv99266360=99, srv528312206=75, srv1214444747=13, srv1396549304=27, srv1021361571=0, srv1128958542=6, srv1457312742=30, srv577532550=79, srv869145969=94, srv1321754755=22, srv1273550020=16, srv663286005=83, srv1382933807=25, srv1608054614=37, srv673556121=84, srv118627256=10, srv16848368=45, srv628985673=80, srv857111501=93, srv2142131442=61, srv1665795864=44, srv1260970219=15, srv974242954=98, srv2100076963=58, srv1316110101=21, srv881423995=95, srv1200260681=12, srv482588120=73, srv311666110=64, srv1924675997=53, srv193946221=54, srv1520890161=33, srv1173918122=9, srv735374154=87, srv1405847637=28, srv2077362636=55, srv1106765323=4, srv476598549=72, srv532718043=76, srv2141951174=60, srv1054239320=2, srv1310659430=18, srv1293936126=17, srv1312968161=19, srv1243178876=14, srv1505234773=32, srv1855970723=50, srv454986971=69, srv1490662399=31, srv563912759=78, srv1824904467=49, srv688835030=85, srv695084789=86, srv492722670=74, srv1089999841=3, srv2093159399=57, srv1167857721=8, srv761421142=89, srv165066687=41, srv1822502759=48, srv1386165651=26, srv652592287=82, srv2082862237=56, srv43270481=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-28 05:56:05,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-28 05:56:05,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-28 05:56:05,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-28 05:56:05,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-28 05:56:05,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-28 05:56:05,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-28 05:56:05,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-28 05:56:05,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-28 05:56:05,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-28 05:56:05,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-28 05:56:05,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-28 05:56:05,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-28 05:56:05,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-28 05:56:05,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-28 05:56:05,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-28 05:56:05,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-28 05:56:05,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-28 05:56:05,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-28 05:56:05,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-28 05:56:05,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-28 05:56:05,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-28 05:56:05,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-28 05:56:05,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-28 05:56:05,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-28 05:56:05,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-28 05:56:05,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-28 05:56:05,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-28 05:56:05,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-28 05:56:05,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-28 05:56:05,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-28 05:56:05,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-28 05:56:05,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-28 05:56:05,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-28 05:56:05,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-28 05:56:05,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-28 05:56:05,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-28 05:56:05,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-28 05:56:05,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-28 05:56:05,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-28 05:56:05,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-28 05:56:05,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-28 05:56:05,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-28 05:56:05,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-28 05:56:05,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-28 05:56:05,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-28 05:56:05,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-28 05:56:05,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-28 05:56:05,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-28 05:56:05,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-28 05:56:05,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-28 05:56:05,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-28 05:56:05,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-28 05:56:05,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-28 05:56:05,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-28 05:56:05,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-28 05:56:05,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-28 05:56:05,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-28 05:56:05,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-28 05:56:05,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-28 05:56:05,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-28 05:56:05,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-28 05:56:05,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-28 05:56:05,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-28 05:56:05,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-28 05:56:05,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-28 05:56:05,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-28 05:56:05,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-28 05:56:05,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-28 05:56:05,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-28 05:56:05,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-28 05:56:05,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-28 05:56:05,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-28 05:56:05,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-28 05:56:05,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-28 05:56:05,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-28 05:56:05,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-28 05:56:05,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-28 05:56:05,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-28 05:56:05,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-28 05:56:05,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-28 05:56:05,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-28 05:56:05,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-28 05:56:05,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-28 05:56:05,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-28 05:56:05,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-28 05:56:05,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-28 05:56:05,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-28 05:56:05,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-28 05:56:05,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-28 05:56:05,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-28 05:56:05,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-28 05:56:05,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-28 05:56:05,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-28 05:56:05,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-28 05:56:05,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-28 05:56:05,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-28 05:56:05,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-28 05:56:05,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-28 05:56:05,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-28 05:56:05,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-28 05:56:05,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-28 05:56:05,355 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0210640979268992); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03060922742558966, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0210640979268992); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04633333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 05:56:05,355 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-28 05:56:05,355 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016039089155303853, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0210640979268992); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03060922742558966, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0210640979268992); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04633333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-28 05:56:05,641 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 428 ms to try 5000 different iterations.  Found a solution that moves 1050 regions; Going from a computed imbalance of 0.016039089155303853 to a new imbalance of 7.19523959009415E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006324949656174055); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006444096124338012); MoveCostFunction : (multiplier=7.0, imbalance=0.11666666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006324949656174055); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 05:56:05,643 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1089999841:81 , srv761421142:81 , srv1260970219:82 , srv1490662399:82 , srv1822502759:82 , srv99266360:82 , srv1116753685:83 , srv1243178876:83 , srv1505234773:83 , srv2142705611:83 , srv482588120:83 , srv834299625:83 , srv974242954:83 , srv1106765323:84 , srv1293936126:84 , srv1314380919:84 , srv1396549304:84 , srv1457312742:84 , srv1626310890:84 , srv296595934:84 , srv311666110:84 , srv857111501:84 , srv1021361571:85 , srv1037219583:85 , srv1054239320:85 , srv1128958542:85 , srv1167857721:85 , srv1273550020:85 , srv1310659430:85 , srv1312968161:85 , srv1381036818:85 , srv1382933807:85 , srv1520890161:85 , srv1608054614:85 , srv1658310314:85 , srv1747165876:85 , srv1824904467:85 , srv1855970723:85 , srv1924675997:85 , srv2077362636:85 , srv2107366898:85 , srv34659052:85 , srv431240307:85 , srv454986971:85 , srv455623035:85 , srv475480428:85 , srv528312206:85 , srv563912759:85 , srv663286005:85 , srv695084789:85 , srv869145969:85 , srv881423995:85 , srv118627256:86 , srv1189172488:86 , srv1200260681:86 , srv1316110101:86 , srv1321754755:86 , srv1335309357:86 , srv1386165651:86 , srv1447204430:86 , srv1540820400:86 , srv1609322071:86 , srv165066687:86 , srv1665795864:86 , srv16848368:86 , srv1706802283:86 , srv193946221:86 , srv2093159399:86 , srv2100076963:86 , srv2141951174:86 , srv355915984:86 , srv476598549:86 , srv532718043:86 , srv577532550:86 , srv643242439:86 , srv652592287:86 , srv673556121:86 , srv688835030:86 , srv735374154:86 , srv781399995:86 , srv817163316:86 , srv1173918122:87 , srv1214444747:87 , srv1405847637:87 , srv1540227482:87 , srv1629220348:87 , srv1653772430:87 , srv1904458506:87 , srv2142131442:87 , srv492722670:87 , srv549315097:87 , srv628985673:87 , srv922154542:87 , srv1574057810:88 , srv736057304:88 , srv1167492920:89 , srv1857409856:89 , srv2082862237:89 , srv43270481:89 , srv942671864:553 }
2023-03-28 05:56:05,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-28 05:56:05,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1747165876=47, srv1857409856=51, srv643242439=81, srv1335309357=23, srv922154542=96, srv34659052=65, srv355915984=66, srv834299625=92, srv1574057810=36, srv475480428=71, srv2107366898=59, srv296595934=63, srv1540227482=34, srv431240307=67, srv1447204430=29, srv736057304=88, srv1609322071=38, srv1626310890=39, srv1037219583=1, srv1658310314=43, srv1116753685=5, srv1653772430=42, srv1629220348=40, srv1706802283=46, srv781399995=90, srv1314380919=20, srv942671864=97, srv1540820400=35, srv549315097=77, srv817163316=91, srv1381036818=24, srv1189172488=11, srv1904458506=52, srv2142705611=62, srv1167492920=7, srv455623035=70, srv99266360=99, srv528312206=75, srv1214444747=13, srv1396549304=27, srv1021361571=0, srv1128958542=6, srv1457312742=30, srv577532550=79, srv869145969=94, srv1321754755=22, srv1273550020=16, srv663286005=83, srv1382933807=25, srv1608054614=37, srv673556121=84, srv118627256=10, srv16848368=45, srv628985673=80, srv857111501=93, srv2142131442=61, srv1665795864=44, srv1260970219=15, srv974242954=98, srv2100076963=58, srv1316110101=21, srv881423995=95, srv1200260681=12, srv482588120=73, srv311666110=64, srv1924675997=53, srv193946221=54, srv1520890161=33, srv1173918122=9, srv735374154=87, srv1405847637=28, srv2077362636=55, srv1106765323=4, srv476598549=72, srv532718043=76, srv2141951174=60, srv1054239320=2, srv1310659430=18, srv1293936126=17, srv1312968161=19, srv1243178876=14, srv1505234773=32, srv1855970723=50, srv454986971=69, srv1490662399=31, srv563912759=78, srv1824904467=49, srv688835030=85, srv695084789=86, srv492722670=74, srv1089999841=3, srv2093159399=57, srv1167857721=8, srv761421142=89, srv165066687=41, srv1822502759=48, srv1386165651=26, srv652592287=82, srv2082862237=56, srv43270481=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-28 05:56:05,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-28 05:56:05,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-28 05:56:05,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-28 05:56:05,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-28 05:56:05,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-28 05:56:05,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-28 05:56:05,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-28 05:56:05,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-28 05:56:05,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-28 05:56:05,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-28 05:56:05,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-28 05:56:05,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-28 05:56:05,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-28 05:56:05,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-28 05:56:05,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-28 05:56:05,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-28 05:56:05,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-28 05:56:05,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-28 05:56:05,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-28 05:56:05,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-28 05:56:05,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-28 05:56:05,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-28 05:56:05,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-28 05:56:05,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-28 05:56:05,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-28 05:56:05,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-28 05:56:05,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-28 05:56:05,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-28 05:56:05,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-28 05:56:05,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-28 05:56:05,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-28 05:56:05,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-28 05:56:05,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-28 05:56:05,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-28 05:56:05,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-28 05:56:05,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-28 05:56:05,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-28 05:56:05,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-28 05:56:05,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-28 05:56:05,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-28 05:56:05,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-28 05:56:05,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-28 05:56:05,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-28 05:56:05,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-28 05:56:05,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-28 05:56:05,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-28 05:56:05,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-28 05:56:05,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-28 05:56:05,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-28 05:56:05,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-28 05:56:05,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-28 05:56:05,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-28 05:56:05,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-28 05:56:05,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-28 05:56:05,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-28 05:56:05,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-28 05:56:05,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-28 05:56:05,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-28 05:56:05,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-28 05:56:05,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-28 05:56:05,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-28 05:56:05,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-28 05:56:05,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-28 05:56:05,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-28 05:56:05,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-28 05:56:05,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-28 05:56:05,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-28 05:56:05,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-28 05:56:05,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-28 05:56:05,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-28 05:56:05,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-28 05:56:05,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-28 05:56:05,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-28 05:56:05,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-28 05:56:05,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-28 05:56:05,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-28 05:56:05,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-28 05:56:05,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-28 05:56:05,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-28 05:56:05,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-28 05:56:05,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-28 05:56:05,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-28 05:56:05,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-28 05:56:05,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-28 05:56:05,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-28 05:56:05,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-28 05:56:05,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-28 05:56:05,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-28 05:56:05,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-28 05:56:05,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-28 05:56:05,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-28 05:56:05,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-28 05:56:05,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-28 05:56:05,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-28 05:56:05,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-28 05:56:05,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-28 05:56:05,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-28 05:56:05,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-28 05:56:05,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-28 05:56:05,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-28 05:56:05,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-28 05:56:05,784 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006324949656174055); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006444096124338012); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006324949656174055); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 05:56:05,784 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-28 05:56:05,784 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.121717018664721E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006324949656174055); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006444096124338012); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006324949656174055); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-28 05:56:06,110 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 426 ms to try 5000 different iterations.  Found a solution that moves 257 regions; Going from a computed imbalance of 7.121717018664721E-4 to a new imbalance of 3.42170082665298E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.028555555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 05:56:06,111 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1173918122:83 , srv1273550020:83 , srv1386165651:83 , srv1857409856:83 , srv476598549:83 , srv673556121:83 , srv1106765323:84 , srv1167492920:84 , srv1167857721:84 , srv1293936126:84 , srv1310659430:84 , srv1335309357:84 , srv1396549304:84 , srv1405847637:84 , srv1574057810:84 , srv1608054614:84 , srv1609322071:84 , srv1626310890:84 , srv165066687:84 , srv1658310314:84 , srv1747165876:84 , srv1824904467:84 , srv1904458506:84 , srv2082862237:84 , srv2093159399:84 , srv2100076963:84 , srv2141951174:84 , srv2142131442:84 , srv2142705611:84 , srv311666110:84 , srv43270481:84 , srv528312206:84 , srv628985673:84 , srv663286005:84 , srv688835030:84 , srv695084789:84 , srv735374154:84 , srv736057304:84 , srv761421142:84 , srv817163316:84 , srv869145969:84 , srv881423995:84 , srv922154542:84 , srv99266360:84 , srv1021361571:85 , srv1037219583:85 , srv1054239320:85 , srv1116753685:85 , srv1128958542:85 , srv118627256:85 , srv1189172488:85 , srv1200260681:85 , srv1214444747:85 , srv1243178876:85 , srv1260970219:85 , srv1312968161:85 , srv1314380919:85 , srv1316110101:85 , srv1381036818:85 , srv1382933807:85 , srv1447204430:85 , srv1457312742:85 , srv1505234773:85 , srv1520890161:85 , srv1540227482:85 , srv1540820400:85 , srv1629220348:85 , srv1653772430:85 , srv16848368:85 , srv1822502759:85 , srv1855970723:85 , srv1924675997:85 , srv193946221:85 , srv2077362636:85 , srv2107366898:85 , srv34659052:85 , srv355915984:85 , srv431240307:85 , srv454986971:85 , srv475480428:85 , srv482588120:85 , srv492722670:85 , srv532718043:85 , srv549315097:85 , srv563912759:85 , srv577532550:85 , srv643242439:85 , srv652592287:85 , srv781399995:85 , srv834299625:85 , srv857111501:85 , srv974242954:85 , srv1089999841:86 , srv1321754755:86 , srv1490662399:86 , srv1665795864:86 , srv1706802283:86 , srv296595934:86 , srv455623035:86 , srv942671864:628 }
2023-03-28 05:56:06,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-28 05:56:06,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1747165876=47, srv1857409856=51, srv643242439=81, srv1335309357=23, srv922154542=96, srv34659052=65, srv355915984=66, srv834299625=92, srv1574057810=36, srv475480428=71, srv2107366898=59, srv296595934=63, srv1540227482=34, srv431240307=67, srv1447204430=29, srv736057304=88, srv1609322071=38, srv1626310890=39, srv1037219583=1, srv1658310314=43, srv1116753685=5, srv1653772430=42, srv1629220348=40, srv1706802283=46, srv781399995=90, srv1314380919=20, srv942671864=97, srv1540820400=35, srv549315097=77, srv817163316=91, srv1381036818=24, srv1189172488=11, srv1904458506=52, srv2142705611=62, srv1167492920=7, srv455623035=70, srv99266360=99, srv528312206=75, srv1214444747=13, srv1396549304=27, srv1021361571=0, srv1128958542=6, srv1457312742=30, srv577532550=79, srv869145969=94, srv1321754755=22, srv1273550020=16, srv663286005=83, srv1382933807=25, srv1608054614=37, srv673556121=84, srv118627256=10, srv16848368=45, srv628985673=80, srv857111501=93, srv2142131442=61, srv1665795864=44, srv1260970219=15, srv974242954=98, srv2100076963=58, srv1316110101=21, srv881423995=95, srv1200260681=12, srv482588120=73, srv311666110=64, srv1924675997=53, srv193946221=54, srv1520890161=33, srv1173918122=9, srv735374154=87, srv1405847637=28, srv2077362636=55, srv1106765323=4, srv476598549=72, srv532718043=76, srv2141951174=60, srv1054239320=2, srv1310659430=18, srv1293936126=17, srv1312968161=19, srv1243178876=14, srv1505234773=32, srv1855970723=50, srv454986971=69, srv1490662399=31, srv563912759=78, srv1824904467=49, srv688835030=85, srv695084789=86, srv492722670=74, srv1089999841=3, srv2093159399=57, srv1167857721=8, srv761421142=89, srv165066687=41, srv1822502759=48, srv1386165651=26, srv652592287=82, srv2082862237=56, srv43270481=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-28 05:56:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-28 05:56:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-28 05:56:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-28 05:56:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-28 05:56:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-28 05:56:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-28 05:56:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-28 05:56:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-28 05:56:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-28 05:56:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-28 05:56:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-28 05:56:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-28 05:56:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-28 05:56:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-28 05:56:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-28 05:56:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-28 05:56:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-28 05:56:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-28 05:56:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-28 05:56:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-28 05:56:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-28 05:56:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-28 05:56:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-28 05:56:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-28 05:56:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-28 05:56:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-28 05:56:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-28 05:56:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-28 05:56:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-28 05:56:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-28 05:56:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-28 05:56:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-28 05:56:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-28 05:56:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-28 05:56:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-28 05:56:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-28 05:56:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-28 05:56:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-28 05:56:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-28 05:56:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-28 05:56:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-28 05:56:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-28 05:56:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-28 05:56:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-28 05:56:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-28 05:56:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-28 05:56:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-28 05:56:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-28 05:56:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-28 05:56:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-28 05:56:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-28 05:56:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-28 05:56:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-28 05:56:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-28 05:56:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-28 05:56:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-28 05:56:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-28 05:56:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-28 05:56:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-28 05:56:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-28 05:56:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-28 05:56:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-28 05:56:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-28 05:56:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-28 05:56:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-28 05:56:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-28 05:56:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-28 05:56:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-28 05:56:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-28 05:56:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-28 05:56:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-28 05:56:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-28 05:56:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-28 05:56:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-28 05:56:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-28 05:56:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-28 05:56:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-28 05:56:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-28 05:56:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-28 05:56:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-28 05:56:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-28 05:56:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-28 05:56:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-28 05:56:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-28 05:56:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-28 05:56:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-28 05:56:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-28 05:56:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-28 05:56:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-28 05:56:06,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-28 05:56:06,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-28 05:56:06,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-28 05:56:06,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-28 05:56:06,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-28 05:56:06,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-28 05:56:06,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-28 05:56:06,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-28 05:56:06,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-28 05:56:06,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-28 05:56:06,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-28 05:56:06,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-28 05:56:06,239 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 05:56:06,239 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-28 05:56:06,239 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.2417455803923784E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-28 05:56:06,457 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 318 ms to try 5000 different iterations.  Found a solution that moves 26 regions; Going from a computed imbalance of 3.2417455803923784E-5 to a new imbalance of 5.8245488846852596E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0012535023884839248); MoveCostFunction : (multiplier=7.0, imbalance=0.0028888888888888888); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 05:56:06,457 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1243178876:83 , srv1314380919:83 , srv1490662399:83 , srv1505234773:83 , srv1706802283:83 , srv296595934:83 , srv455623035:83 , srv492722670:83 , srv781399995:83 , srv834299625:83 , srv869145969:83 , srv1021361571:84 , srv1037219583:84 , srv1089999841:84 , srv1106765323:84 , srv1116753685:84 , srv1128958542:84 , srv1167492920:84 , srv1167857721:84 , srv118627256:84 , srv1189172488:84 , srv1200260681:84 , srv1214444747:84 , srv1260970219:84 , srv1310659430:84 , srv1312968161:84 , srv1316110101:84 , srv1321754755:84 , srv1335309357:84 , srv1381036818:84 , srv1382933807:84 , srv1386165651:84 , srv1396549304:84 , srv1405847637:84 , srv1447204430:84 , srv1457312742:84 , srv1520890161:84 , srv1540227482:84 , srv1540820400:84 , srv1574057810:84 , srv1609322071:84 , srv1626310890:84 , srv1629220348:84 , srv1653772430:84 , srv1658310314:84 , srv1665795864:84 , srv16848368:84 , srv1747165876:84 , srv1822502759:84 , srv1855970723:84 , srv1857409856:84 , srv1904458506:84 , srv1924675997:84 , srv193946221:84 , srv2077362636:84 , srv2082862237:84 , srv2100076963:84 , srv2107366898:84 , srv2141951174:84 , srv2142705611:84 , srv311666110:84 , srv34659052:84 , srv355915984:84 , srv431240307:84 , srv43270481:84 , srv454986971:84 , srv475480428:84 , srv476598549:84 , srv482588120:84 , srv528312206:84 , srv532718043:84 , srv549315097:84 , srv563912759:84 , srv577532550:84 , srv628985673:84 , srv643242439:84 , srv652592287:84 , srv663286005:84 , srv695084789:84 , srv735374154:84 , srv736057304:84 , srv761421142:84 , srv817163316:84 , srv857111501:84 , srv881423995:84 , srv922154542:84 , srv974242954:84 , srv99266360:84 , srv1054239320:85 , srv1173918122:85 , srv1273550020:85 , srv1293936126:85 , srv1608054614:85 , srv165066687:85 , srv1824904467:85 , srv2093159399:85 , srv2142131442:85 , srv673556121:85 , srv688835030:85 , srv942671864:684 }
2023-03-28 05:56:06,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-28 05:56:06,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1747165876=47, srv1857409856=51, srv643242439=81, srv1335309357=23, srv922154542=96, srv34659052=65, srv355915984=66, srv834299625=92, srv1574057810=36, srv475480428=71, srv2107366898=59, srv296595934=63, srv1540227482=34, srv431240307=67, srv1447204430=29, srv736057304=88, srv1609322071=38, srv1626310890=39, srv1037219583=1, srv1658310314=43, srv1116753685=5, srv1653772430=42, srv1629220348=40, srv1706802283=46, srv781399995=90, srv1314380919=20, srv942671864=97, srv1540820400=35, srv549315097=77, srv817163316=91, srv1381036818=24, srv1189172488=11, srv1904458506=52, srv2142705611=62, srv1167492920=7, srv455623035=70, srv99266360=99, srv528312206=75, srv1214444747=13, srv1396549304=27, srv1021361571=0, srv1128958542=6, srv1457312742=30, srv577532550=79, srv869145969=94, srv1321754755=22, srv1273550020=16, srv663286005=83, srv1382933807=25, srv1608054614=37, srv673556121=84, srv118627256=10, srv16848368=45, srv628985673=80, srv857111501=93, srv2142131442=61, srv1665795864=44, srv1260970219=15, srv974242954=98, srv2100076963=58, srv1316110101=21, srv881423995=95, srv1200260681=12, srv482588120=73, srv311666110=64, srv1924675997=53, srv193946221=54, srv1520890161=33, srv1173918122=9, srv735374154=87, srv1405847637=28, srv2077362636=55, srv1106765323=4, srv476598549=72, srv532718043=76, srv2141951174=60, srv1054239320=2, srv1310659430=18, srv1293936126=17, srv1312968161=19, srv1243178876=14, srv1505234773=32, srv1855970723=50, srv454986971=69, srv1490662399=31, srv563912759=78, srv1824904467=49, srv688835030=85, srv695084789=86, srv492722670=74, srv1089999841=3, srv2093159399=57, srv1167857721=8, srv761421142=89, srv165066687=41, srv1822502759=48, srv1386165651=26, srv652592287=82, srv2082862237=56, srv43270481=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-28 05:56:06,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-28 05:56:06,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-28 05:56:06,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-28 05:56:06,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-28 05:56:06,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-28 05:56:06,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-28 05:56:06,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-28 05:56:06,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-28 05:56:06,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-28 05:56:06,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-28 05:56:06,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-28 05:56:06,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-28 05:56:06,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-28 05:56:06,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-28 05:56:06,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-28 05:56:06,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-28 05:56:06,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-28 05:56:06,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-28 05:56:06,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-28 05:56:06,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-28 05:56:06,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-28 05:56:06,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-28 05:56:06,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-28 05:56:06,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-28 05:56:06,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-28 05:56:06,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-28 05:56:06,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-28 05:56:06,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-28 05:56:06,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-28 05:56:06,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-28 05:56:06,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-28 05:56:06,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-28 05:56:06,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-28 05:56:06,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-28 05:56:06,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-28 05:56:06,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-28 05:56:06,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-28 05:56:06,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-28 05:56:06,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-28 05:56:06,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-28 05:56:06,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-28 05:56:06,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-28 05:56:06,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-28 05:56:06,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-28 05:56:06,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-28 05:56:06,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-28 05:56:06,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-28 05:56:06,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-28 05:56:06,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-28 05:56:06,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-28 05:56:06,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-28 05:56:06,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-28 05:56:06,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-28 05:56:06,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-28 05:56:06,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-28 05:56:06,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-28 05:56:06,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-28 05:56:06,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-28 05:56:06,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-28 05:56:06,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-28 05:56:06,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-28 05:56:06,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-28 05:56:06,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-28 05:56:06,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-28 05:56:06,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-28 05:56:06,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-28 05:56:06,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-28 05:56:06,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-28 05:56:06,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-28 05:56:06,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-28 05:56:06,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-28 05:56:06,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-28 05:56:06,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-28 05:56:06,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-28 05:56:06,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-28 05:56:06,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-28 05:56:06,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-28 05:56:06,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-28 05:56:06,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-28 05:56:06,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-28 05:56:06,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-28 05:56:06,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-28 05:56:06,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-28 05:56:06,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-28 05:56:06,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-28 05:56:06,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-28 05:56:06,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-28 05:56:06,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-28 05:56:06,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-28 05:56:06,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-28 05:56:06,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-28 05:56:06,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-28 05:56:06,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-28 05:56:06,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-28 05:56:06,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-28 05:56:06,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-28 05:56:06,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-28 05:56:06,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-28 05:56:06,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-28 05:56:06,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-28 05:56:06,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-28 05:56:06,585 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=5.642492993526674E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0012535023884839248); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 05:56:06,585 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1243178876:83 , srv1314380919:83 , srv1490662399:83 , srv1505234773:83 , srv1706802283:83 , srv296595934:83 , srv455623035:83 , srv492722670:83 , srv781399995:83 , srv834299625:83 , srv869145969:83 , srv1021361571:84 , srv1037219583:84 , srv1089999841:84 , srv1106765323:84 , srv1116753685:84 , srv1128958542:84 , srv1167492920:84 , srv1167857721:84 , srv118627256:84 , srv1189172488:84 , srv1200260681:84 , srv1214444747:84 , srv1260970219:84 , srv1310659430:84 , srv1312968161:84 , srv1316110101:84 , srv1321754755:84 , srv1335309357:84 , srv1381036818:84 , srv1382933807:84 , srv1386165651:84 , srv1396549304:84 , srv1405847637:84 , srv1447204430:84 , srv1457312742:84 , srv1520890161:84 , srv1540227482:84 , srv1540820400:84 , srv1574057810:84 , srv1609322071:84 , srv1626310890:84 , srv1629220348:84 , srv1653772430:84 , srv1658310314:84 , srv1665795864:84 , srv16848368:84 , srv1747165876:84 , srv1822502759:84 , srv1855970723:84 , srv1857409856:84 , srv1904458506:84 , srv1924675997:84 , srv193946221:84 , srv2077362636:84 , srv2082862237:84 , srv2100076963:84 , srv2107366898:84 , srv2141951174:84 , srv2142705611:84 , srv311666110:84 , srv34659052:84 , srv355915984:84 , srv431240307:84 , srv43270481:84 , srv454986971:84 , srv475480428:84 , srv476598549:84 , srv482588120:84 , srv528312206:84 , srv532718043:84 , srv549315097:84 , srv563912759:84 , srv577532550:84 , srv628985673:84 , srv643242439:84 , srv652592287:84 , srv663286005:84 , srv695084789:84 , srv735374154:84 , srv736057304:84 , srv761421142:84 , srv817163316:84 , srv857111501:84 , srv881423995:84 , srv922154542:84 , srv974242954:84 , srv99266360:84 , srv1054239320:85 , srv1173918122:85 , srv1273550020:85 , srv1293936126:85 , srv1608054614:85 , srv165066687:85 , srv1824904467:85 , srv2093159399:85 , srv2142131442:85 , srv673556121:85 , srv688835030:85 , srv942671864:684 }
2023-03-28 05:56:06,609 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=403 (was 343) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=4617 (was 5109)
2023-03-28 05:56:06,617 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=403, ProcessCount=170, AvailableMemoryMB=4616
2023-03-28 05:56:06,618 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-28 05:56:06,618 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-03-28 05:56:06,626 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1024737269:84 , srv1278554476:84 , srv1316629580:84 , srv1396400213:84 , srv1560943776:84 , srv1605646541:84 , srv1632724159:84 , srv1635925112:84 , srv1706608726:84 , srv171560692:84 , srv18162431:84 , srv2069449279:84 , srv2094750157:84 , srv2142622109:84 , srv248012442:84 , srv258436703:84 , srv289693207:84 , srv304162398:84 , srv327577178:84 , srv392235968:84 , srv50371076:84 , srv556342150:84 , srv59344333:84 , srv605901000:84 , srv649208591:84 , srv709586099:84 , srv747690923:84 , srv827992381:84 , srv949456759:84 , srv1218734060:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-03-28 05:56:06,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-28 05:56:06,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2094750157=13, srv59344333=23, srv18162431=11, srv1605646541=6, srv289693207=17, srv649208591=25, srv248012442=15, srv605901000=24, srv1560943776=5, srv1632724159=7, srv1024737269=0, srv1218734060=1, srv1635925112=8, srv327577178=19, srv50371076=21, srv827992381=28, srv392235968=20, srv1396400213=4, srv709586099=26, srv304162398=18, srv556342150=22, srv1706608726=9, srv949456759=29, srv1316629580=3, srv747690923=27, srv2069449279=12, srv1278554476=2, srv2142622109=14, srv258436703=16, srv171560692=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-28 05:56:06,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-28 05:56:06,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-28 05:56:06,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-28 05:56:06,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-28 05:56:06,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-28 05:56:06,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-28 05:56:06,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-28 05:56:06,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-28 05:56:06,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-28 05:56:06,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-28 05:56:06,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-28 05:56:06,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-28 05:56:06,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-28 05:56:06,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-28 05:56:06,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-28 05:56:06,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-28 05:56:06,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-28 05:56:06,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-28 05:56:06,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-28 05:56:06,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-28 05:56:06,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-28 05:56:06,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-28 05:56:06,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-28 05:56:06,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-28 05:56:06,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-28 05:56:06,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-28 05:56:06,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-28 05:56:06,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-28 05:56:06,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-28 05:56:06,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-28 05:56:06,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-28 05:56:06,653 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.161567475056232, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 05:56:06,653 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912702137102517, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.161567475056232, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-03-28 05:58:06,639 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5970135 different iterations.  Found a solution that moves 1876 regions; Going from a computed imbalance of 0.9912702137102517 to a new imbalance of 4.37867758735265E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6948148148148148, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 05:58:06,647 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1024737269:90 , srv1218734060:90 , srv1278554476:90 , srv1316629580:90 , srv1396400213:90 , srv1560943776:90 , srv1605646541:90 , srv1632724159:90 , srv1635925112:90 , srv1706608726:90 , srv171560692:90 , srv18162431:90 , srv2069449279:90 , srv2094750157:90 , srv2142622109:90 , srv248012442:90 , srv258436703:90 , srv289693207:90 , srv304162398:90 , srv327577178:90 , srv392235968:90 , srv50371076:90 , srv556342150:90 , srv59344333:90 , srv605901000:90 , srv649208591:90 , srv709586099:90 , srv747690923:90 , srv827992381:90 , srv949456759:90 }
2023-03-28 05:58:06,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-28 05:58:06,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2094750157=13, srv59344333=23, srv18162431=11, srv1605646541=6, srv289693207=17, srv649208591=25, srv248012442=15, srv605901000=24, srv1560943776=5, srv1632724159=7, srv1024737269=0, srv1218734060=1, srv1635925112=8, srv327577178=19, srv50371076=21, srv827992381=28, srv392235968=20, srv1396400213=4, srv709586099=26, srv304162398=18, srv556342150=22, srv1706608726=9, srv949456759=29, srv1316629580=3, srv747690923=27, srv2069449279=12, srv1278554476=2, srv2142622109=14, srv258436703=16, srv171560692=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-28 05:58:06,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-28 05:58:06,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-28 05:58:06,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-28 05:58:06,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-28 05:58:06,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-28 05:58:06,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-28 05:58:06,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-28 05:58:06,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-28 05:58:06,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-28 05:58:06,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-28 05:58:06,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-28 05:58:06,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-28 05:58:06,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-28 05:58:06,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-28 05:58:06,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-28 05:58:06,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-28 05:58:06,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-28 05:58:06,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-28 05:58:06,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-28 05:58:06,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-28 05:58:06,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-28 05:58:06,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-28 05:58:06,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-28 05:58:06,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-28 05:58:06,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-28 05:58:06,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-28 05:58:06,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-28 05:58:06,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-28 05:58:06,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-28 05:58:06,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-28 05:58:06,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-28 05:58:06,670 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 05:58:06,670 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1024737269:90 , srv1218734060:90 , srv1278554476:90 , srv1316629580:90 , srv1396400213:90 , srv1560943776:90 , srv1605646541:90 , srv1632724159:90 , srv1635925112:90 , srv1706608726:90 , srv171560692:90 , srv18162431:90 , srv2069449279:90 , srv2094750157:90 , srv2142622109:90 , srv248012442:90 , srv258436703:90 , srv289693207:90 , srv304162398:90 , srv327577178:90 , srv392235968:90 , srv50371076:90 , srv556342150:90 , srv59344333:90 , srv605901000:90 , srv649208591:90 , srv709586099:90 , srv747690923:90 , srv827992381:90 , srv949456759:90 }
2023-03-28 05:58:06,686 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=256 (was 403), ProcessCount=171 (was 170) - ProcessCount LEAK? -, AvailableMemoryMB=5638 (was 4616) - AvailableMemoryMB LEAK? -