2023-03-28 22:56:07,507 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-03-28 22:56:07,921 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-03-28 22:56:08,055 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-03-28 22:56:08,122 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-28 22:56:08,131 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-28 22:56:08,147 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=375, ProcessCount=169, AvailableMemoryMB=7731
2023-03-28 22:56:08,150 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-28 22:56:08,150 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-28 22:56:08,296 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1016883791:84 , srv1084417228:84 , srv1100153732:84 , srv1124558547:84 , srv1136324979:84 , srv1144076507:84 , srv1225635612:84 , srv1242637818:84 , srv1246006536:84 , srv1248788017:84 , srv1257882259:84 , srv1293883430:84 , srv1298076497:84 , srv1298366630:84 , srv1305432105:84 , srv131963353:84 , srv1333848749:84 , srv1337785245:84 , srv1364599815:84 , srv1373941404:84 , srv1404593730:84 , srv1416451051:84 , srv1426822100:84 , srv1434279016:84 , srv146134621:84 , srv1485477795:84 , srv1525616221:84 , srv1532661215:84 , srv1564931578:84 , srv1587715539:84 , srv1728447493:84 , srv1729096831:84 , srv173423467:84 , srv176020563:84 , srv1771182410:84 , srv1784000568:84 , srv1795020122:84 , srv1809127233:84 , srv1822672305:84 , srv1870413341:84 , srv1874755382:84 , srv1898235659:84 , srv1907098184:84 , srv1915187457:84 , srv192277820:84 , srv1929180431:84 , srv1930829159:84 , srv1937599842:84 , srv1942887615:84 , srv195170720:84 , srv1971135070:84 , srv2031712906:84 , srv2048997899:84 , srv2067911860:84 , srv2075260001:84 , srv2099897182:84 , srv2105787785:84 , srv2127241614:84 , srv2135663415:84 , srv217931015:84 , srv220091052:84 , srv244832148:84 , srv294610619:84 , srv3211682:84 , srv325797459:84 , srv328678554:84 , srv342880967:84 , srv351785965:84 , srv358944971:84 , srv388977332:84 , srv403784332:84 , srv429964645:84 , srv452158090:84 , srv456331258:84 , srv476740148:84 , srv477377218:84 , srv498570630:84 , srv500645911:84 , srv504889239:84 , srv513794824:84 , srv556523821:84 , srv601188762:84 , srv611942608:84 , srv637548025:84 , srv667629708:84 , srv678600084:84 , srv692003784:84 , srv717703021:84 , srv718451227:84 , srv720807655:84 , srv729938898:84 , srv737356488:84 , srv765878035:84 , srv858847199:84 , srv860979120:84 , srv874495266:84 , srv938950966:84 , srv952161018:84 , srv953981088:84 , srv2093342793:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-03-28 22:56:08,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-28 22:56:08,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1587715539=29, srv858847199=94, srv1416451051=21, srv504889239=79, srv601188762=82, srv1729096831=31, srv146134621=24, srv1874755382=40, srv1898235659=41, srv717703021=88, srv1870413341=39, srv1333848749=16, srv1971135070=50, srv1784000568=35, srv192277820=44, srv720807655=90, srv342880967=67, srv1404593730=20, srv173423467=32, srv2067911860=53, srv2105787785=57, srv1124558547=3, srv860979120=95, srv131963353=15, srv718451227=89, srv1298076497=12, srv1298366630=13, srv388977332=70, srv1915187457=43, srv1136324979=4, srv938950966=97, srv1434279016=23, srv1426822100=22, srv1771182410=34, srv1084417228=1, srv456331258=74, srv667629708=85, srv358944971=69, srv294610619=63, srv1144076507=5, srv1728447493=30, srv176020563=33, srv637548025=84, srv765878035=93, srv692003784=87, srv611942608=83, srv1822672305=38, srv476740148=75, srv678600084=86, srv220091052=61, srv1246006536=8, srv1942887615=48, srv3211682=64, srv2093342793=55, srv500645911=78, srv1337785245=17, srv953981088=99, srv1225635612=6, srv2031712906=51, srv1016883791=0, srv1532661215=27, srv1809127233=37, srv325797459=65, srv556523821=81, srv351785965=68, srv1525616221=26, srv1248788017=9, srv2099897182=56, srv1930829159=46, srv452158090=73, srv1564931578=28, srv1100153732=2, srv737356488=92, srv1305432105=14, srv1485477795=25, srv1364599815=18, srv429964645=72, srv1373941404=19, srv328678554=66, srv1257882259=10, srv195170720=49, srv477377218=76, srv217931015=60, srv1242637818=7, srv244832148=62, srv1795020122=36, srv2135663415=59, srv2127241614=58, srv952161018=98, srv1929180431=45, srv403784332=71, srv513794824=80, srv2075260001=54, srv1907098184=42, srv1937599842=47, srv2048997899=52, srv874495266=96, srv498570630=77, srv1293883430=11, srv729938898=91} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-28 22:56:08,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-28 22:56:08,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-28 22:56:08,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-28 22:56:08,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-28 22:56:08,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-28 22:56:08,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-28 22:56:08,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-28 22:56:08,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-28 22:56:08,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-28 22:56:08,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-28 22:56:08,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-28 22:56:08,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-28 22:56:08,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-28 22:56:08,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-28 22:56:08,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-28 22:56:08,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-28 22:56:08,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-28 22:56:08,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-28 22:56:08,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-28 22:56:08,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-28 22:56:08,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-28 22:56:08,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-28 22:56:08,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-28 22:56:08,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-28 22:56:08,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-28 22:56:08,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-28 22:56:08,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-28 22:56:08,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-28 22:56:08,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-28 22:56:08,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-28 22:56:08,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-28 22:56:08,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-28 22:56:08,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-28 22:56:08,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-28 22:56:08,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-28 22:56:08,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-28 22:56:08,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-28 22:56:08,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-28 22:56:08,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-28 22:56:08,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-28 22:56:08,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-28 22:56:08,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-28 22:56:08,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-28 22:56:08,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-28 22:56:08,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-28 22:56:08,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-28 22:56:08,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-28 22:56:08,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-28 22:56:08,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-28 22:56:08,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-28 22:56:08,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-28 22:56:08,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-28 22:56:08,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-28 22:56:08,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-28 22:56:08,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-28 22:56:08,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-28 22:56:08,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-28 22:56:08,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-28 22:56:08,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-28 22:56:08,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-28 22:56:08,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-28 22:56:08,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-28 22:56:08,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-28 22:56:08,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-28 22:56:08,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-28 22:56:08,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-28 22:56:08,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-28 22:56:08,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-28 22:56:08,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-28 22:56:08,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-28 22:56:08,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-28 22:56:08,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-28 22:56:08,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-28 22:56:08,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-28 22:56:08,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-28 22:56:08,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-28 22:56:08,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-28 22:56:08,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-28 22:56:08,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-28 22:56:08,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-28 22:56:08,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-28 22:56:08,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-28 22:56:08,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-28 22:56:08,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-28 22:56:08,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-28 22:56:08,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-28 22:56:08,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-28 22:56:08,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-28 22:56:08,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-28 22:56:08,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-28 22:56:08,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-28 22:56:08,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-28 22:56:08,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-28 22:56:08,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-28 22:56:08,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-28 22:56:08,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-28 22:56:08,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-28 22:56:08,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-28 22:56:08,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-28 22:56:08,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-28 22:56:08,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-28 22:56:08,716 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 22:56:08,716 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-28 22:56:08,716 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-28 22:56:09,090 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 606 ms to try 5000 different iterations.  Found a solution that moves 4746 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1697785888122603. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04013818256925179, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05832876364977519, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5273333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04013818256925179, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16475, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23291666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 22:56:09,111 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1364599815:80 , srv1144076507:82 , srv1907098184:82 , srv1136324979:83 , srv1729096831:83 , srv1942887615:83 , srv513794824:83 , srv737356488:83 , srv938950966:83 , srv1416451051:84 , srv1426822100:84 , srv1434279016:84 , srv1485477795:84 , srv1795020122:84 , srv1874755382:84 , srv1971135070:84 , srv2105787785:84 , srv351785965:84 , srv476740148:84 , srv1248788017:85 , srv1728447493:85 , srv1809127233:85 , srv1822672305:85 , srv1870413341:85 , srv2031712906:85 , srv2048997899:85 , srv217931015:85 , srv328678554:85 , srv342880967:85 , srv718451227:85 , srv858847199:85 , srv1225635612:86 , srv1242637818:86 , srv1293883430:86 , srv1298076497:86 , srv1305432105:86 , srv131963353:86 , srv1898235659:86 , srv195170720:86 , srv2075260001:86 , srv2099897182:86 , srv220091052:86 , srv456331258:86 , srv477377218:86 , srv498570630:86 , srv556523821:86 , srv601188762:86 , srv611942608:86 , srv720807655:86 , srv874495266:86 , srv1016883791:87 , srv1100153732:87 , srv1124558547:87 , srv1246006536:87 , srv1298366630:87 , srv1333848749:87 , srv1373941404:87 , srv146134621:87 , srv1784000568:87 , srv1915187457:87 , srv1930829159:87 , srv2127241614:87 , srv2135663415:87 , srv325797459:87 , srv358944971:87 , srv504889239:87 , srv637548025:87 , srv667629708:87 , srv678600084:87 , srv717703021:87 , srv860979120:87 , srv952161018:87 , srv1257882259:88 , srv1532661215:88 , srv1564931578:88 , srv173423467:88 , srv1771182410:88 , srv1929180431:88 , srv1937599842:88 , srv388977332:88 , srv403784332:88 , srv452158090:88 , srv500645911:88 , srv1337785245:89 , srv176020563:89 , srv192277820:89 , srv2067911860:89 , srv244832148:89 , srv3211682:89 , srv429964645:89 , srv729938898:89 , srv765878035:89 , srv1404593730:90 , srv1587715539:90 , srv294610619:90 , srv953981088:90 , srv1084417228:91 , srv692003784:91 , srv1525616221:92 , srv2093342793:447 }
2023-03-28 22:56:09,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-28 22:56:09,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1587715539=29, srv858847199=94, srv1416451051=21, srv504889239=79, srv601188762=82, srv1729096831=31, srv146134621=24, srv1874755382=40, srv1898235659=41, srv717703021=88, srv1870413341=39, srv1333848749=16, srv1971135070=50, srv1784000568=35, srv192277820=44, srv720807655=90, srv342880967=67, srv1404593730=20, srv173423467=32, srv2067911860=53, srv2105787785=57, srv1124558547=3, srv860979120=95, srv131963353=15, srv718451227=89, srv1298076497=12, srv1298366630=13, srv388977332=70, srv1915187457=43, srv1136324979=4, srv938950966=97, srv1434279016=23, srv1426822100=22, srv1771182410=34, srv1084417228=1, srv456331258=74, srv667629708=85, srv358944971=69, srv294610619=63, srv1144076507=5, srv1728447493=30, srv176020563=33, srv637548025=84, srv765878035=93, srv692003784=87, srv611942608=83, srv1822672305=38, srv476740148=75, srv678600084=86, srv220091052=61, srv1246006536=8, srv1942887615=48, srv3211682=64, srv2093342793=55, srv500645911=78, srv1337785245=17, srv953981088=99, srv1225635612=6, srv2031712906=51, srv1016883791=0, srv1532661215=27, srv1809127233=37, srv325797459=65, srv556523821=81, srv351785965=68, srv1525616221=26, srv1248788017=9, srv2099897182=56, srv1930829159=46, srv452158090=73, srv1564931578=28, srv1100153732=2, srv737356488=92, srv1305432105=14, srv1485477795=25, srv1364599815=18, srv429964645=72, srv1373941404=19, srv328678554=66, srv1257882259=10, srv195170720=49, srv477377218=76, srv217931015=60, srv1242637818=7, srv244832148=62, srv1795020122=36, srv2135663415=59, srv2127241614=58, srv952161018=98, srv1929180431=45, srv403784332=71, srv513794824=80, srv2075260001=54, srv1907098184=42, srv1937599842=47, srv2048997899=52, srv874495266=96, srv498570630=77, srv1293883430=11, srv729938898=91} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-28 22:56:09,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-28 22:56:09,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-28 22:56:09,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-28 22:56:09,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-28 22:56:09,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-28 22:56:09,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-28 22:56:09,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-28 22:56:09,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-28 22:56:09,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-28 22:56:09,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-28 22:56:09,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-28 22:56:09,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-28 22:56:09,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-28 22:56:09,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-28 22:56:09,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-28 22:56:09,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-28 22:56:09,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-28 22:56:09,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-28 22:56:09,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-28 22:56:09,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-28 22:56:09,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-28 22:56:09,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-28 22:56:09,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-28 22:56:09,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-28 22:56:09,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-28 22:56:09,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-28 22:56:09,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-28 22:56:09,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-28 22:56:09,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-28 22:56:09,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-28 22:56:09,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-28 22:56:09,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-28 22:56:09,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-28 22:56:09,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-28 22:56:09,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-28 22:56:09,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-28 22:56:09,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-28 22:56:09,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-28 22:56:09,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-28 22:56:09,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-28 22:56:09,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-28 22:56:09,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-28 22:56:09,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-28 22:56:09,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-28 22:56:09,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-28 22:56:09,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-28 22:56:09,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-28 22:56:09,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-28 22:56:09,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-28 22:56:09,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-28 22:56:09,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-28 22:56:09,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-28 22:56:09,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-28 22:56:09,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-28 22:56:09,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-28 22:56:09,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-28 22:56:09,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-28 22:56:09,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-28 22:56:09,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-28 22:56:09,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-28 22:56:09,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-28 22:56:09,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-28 22:56:09,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-28 22:56:09,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-28 22:56:09,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-28 22:56:09,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-28 22:56:09,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-28 22:56:09,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-28 22:56:09,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-28 22:56:09,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-28 22:56:09,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-28 22:56:09,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-28 22:56:09,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-28 22:56:09,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-28 22:56:09,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-28 22:56:09,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-28 22:56:09,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-28 22:56:09,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-28 22:56:09,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-28 22:56:09,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-28 22:56:09,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-28 22:56:09,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-28 22:56:09,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-28 22:56:09,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-28 22:56:09,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-28 22:56:09,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-28 22:56:09,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-28 22:56:09,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-28 22:56:09,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-28 22:56:09,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-28 22:56:09,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-28 22:56:09,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-28 22:56:09,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-28 22:56:09,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-28 22:56:09,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-28 22:56:09,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-28 22:56:09,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-28 22:56:09,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-28 22:56:09,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-28 22:56:09,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-28 22:56:09,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-28 22:56:09,315 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04013818256925179, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05832876364977519, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04013818256925179, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16475, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23291666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 22:56:09,316 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-28 22:56:09,316 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.16974535660997422, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04013818256925179, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05832876364977519, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04013818256925179, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16475, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23291666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-28 22:56:09,655 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 472 ms to try 5000 different iterations.  Found a solution that moves 2658 regions; Going from a computed imbalance of 0.16974535660997422 to a new imbalance of 0.012800069684581215. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021375566127829166); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03158015828501177, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29533333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021375566127829166); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043416666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 22:56:09,667 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1084417228:79 , srv1525616221:80 , srv1587715539:81 , srv294610619:81 , srv429964645:81 , srv1257882259:82 , srv176020563:82 , srv2067911860:82 , srv244832148:82 , srv452158090:82 , srv692003784:82 , srv1404593730:83 , srv1771182410:83 , srv388977332:83 , srv403784332:83 , srv952161018:83 , srv1100153732:84 , srv1337785245:84 , srv1373941404:84 , srv146134621:84 , srv1564931578:84 , srv173423467:84 , srv1784000568:84 , srv192277820:84 , srv1929180431:84 , srv2127241614:84 , srv2135663415:84 , srv358944971:84 , srv477377218:84 , srv498570630:84 , srv504889239:84 , srv717703021:84 , srv729938898:84 , srv1016883791:85 , srv1298366630:85 , srv131963353:85 , srv1532661215:85 , srv1822672305:85 , srv1915187457:85 , srv2075260001:85 , srv325797459:85 , srv456331258:85 , srv500645911:85 , srv637548025:85 , srv667629708:85 , srv765878035:85 , srv1124558547:86 , srv1242637818:86 , srv1305432105:86 , srv1333848749:86 , srv1870413341:86 , srv1937599842:86 , srv1971135070:86 , srv2099897182:86 , srv220091052:86 , srv328678554:86 , srv556523821:86 , srv601188762:86 , srv678600084:86 , srv718451227:86 , srv860979120:86 , srv953981088:86 , srv1225635612:87 , srv1248788017:87 , srv1434279016:87 , srv1809127233:87 , srv1898235659:87 , srv1930829159:87 , srv2048997899:87 , srv217931015:87 , srv3211682:87 , srv342880967:87 , srv476740148:87 , srv720807655:87 , srv874495266:87 , srv1246006536:88 , srv1293883430:88 , srv1298076497:88 , srv1426822100:88 , srv1485477795:88 , srv1729096831:88 , srv1942887615:88 , srv195170720:88 , srv737356488:88 , srv938950966:88 , srv1136324979:89 , srv1416451051:89 , srv1728447493:89 , srv1795020122:89 , srv2031712906:89 , srv2105787785:89 , srv351785965:89 , srv611942608:89 , srv858847199:89 , srv1874755382:90 , srv1907098184:90 , srv513794824:90 , srv1144076507:91 , srv1364599815:92 , srv2093342793:517 }
2023-03-28 22:56:09,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-28 22:56:09,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1587715539=29, srv858847199=94, srv1416451051=21, srv504889239=79, srv601188762=82, srv1729096831=31, srv146134621=24, srv1874755382=40, srv1898235659=41, srv717703021=88, srv1870413341=39, srv1333848749=16, srv1971135070=50, srv1784000568=35, srv192277820=44, srv720807655=90, srv342880967=67, srv1404593730=20, srv173423467=32, srv2067911860=53, srv2105787785=57, srv1124558547=3, srv860979120=95, srv131963353=15, srv718451227=89, srv1298076497=12, srv1298366630=13, srv388977332=70, srv1915187457=43, srv1136324979=4, srv938950966=97, srv1434279016=23, srv1426822100=22, srv1771182410=34, srv1084417228=1, srv456331258=74, srv667629708=85, srv358944971=69, srv294610619=63, srv1144076507=5, srv1728447493=30, srv176020563=33, srv637548025=84, srv765878035=93, srv692003784=87, srv611942608=83, srv1822672305=38, srv476740148=75, srv678600084=86, srv220091052=61, srv1246006536=8, srv1942887615=48, srv3211682=64, srv2093342793=55, srv500645911=78, srv1337785245=17, srv953981088=99, srv1225635612=6, srv2031712906=51, srv1016883791=0, srv1532661215=27, srv1809127233=37, srv325797459=65, srv556523821=81, srv351785965=68, srv1525616221=26, srv1248788017=9, srv2099897182=56, srv1930829159=46, srv452158090=73, srv1564931578=28, srv1100153732=2, srv737356488=92, srv1305432105=14, srv1485477795=25, srv1364599815=18, srv429964645=72, srv1373941404=19, srv328678554=66, srv1257882259=10, srv195170720=49, srv477377218=76, srv217931015=60, srv1242637818=7, srv244832148=62, srv1795020122=36, srv2135663415=59, srv2127241614=58, srv952161018=98, srv1929180431=45, srv403784332=71, srv513794824=80, srv2075260001=54, srv1907098184=42, srv1937599842=47, srv2048997899=52, srv874495266=96, srv498570630=77, srv1293883430=11, srv729938898=91} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-28 22:56:09,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-28 22:56:09,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-28 22:56:09,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-28 22:56:09,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-28 22:56:09,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-28 22:56:09,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-28 22:56:09,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-28 22:56:09,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-28 22:56:09,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-28 22:56:09,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-28 22:56:09,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-28 22:56:09,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-28 22:56:09,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-28 22:56:09,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-28 22:56:09,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-28 22:56:09,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-28 22:56:09,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-28 22:56:09,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-28 22:56:09,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-28 22:56:09,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-28 22:56:09,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-28 22:56:09,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-28 22:56:09,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-28 22:56:09,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-28 22:56:09,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-28 22:56:09,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-28 22:56:09,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-28 22:56:09,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-28 22:56:09,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-28 22:56:09,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-28 22:56:09,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-28 22:56:09,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-28 22:56:09,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-28 22:56:09,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-28 22:56:09,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-28 22:56:09,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-28 22:56:09,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-28 22:56:09,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-28 22:56:09,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-28 22:56:09,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-28 22:56:09,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-28 22:56:09,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-28 22:56:09,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-28 22:56:09,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-28 22:56:09,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-28 22:56:09,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-28 22:56:09,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-28 22:56:09,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-28 22:56:09,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-28 22:56:09,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-28 22:56:09,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-28 22:56:09,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-28 22:56:09,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-28 22:56:09,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-28 22:56:09,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-28 22:56:09,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-28 22:56:09,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-28 22:56:09,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-28 22:56:09,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-28 22:56:09,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-28 22:56:09,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-28 22:56:09,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-28 22:56:09,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-28 22:56:09,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-28 22:56:09,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-28 22:56:09,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-28 22:56:09,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-28 22:56:09,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-28 22:56:09,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-28 22:56:09,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-28 22:56:09,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-28 22:56:09,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-28 22:56:09,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-28 22:56:09,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-28 22:56:09,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-28 22:56:09,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-28 22:56:09,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-28 22:56:09,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-28 22:56:09,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-28 22:56:09,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-28 22:56:09,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-28 22:56:09,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-28 22:56:09,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-28 22:56:09,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-28 22:56:09,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-28 22:56:09,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-28 22:56:09,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-28 22:56:09,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-28 22:56:09,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-28 22:56:09,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-28 22:56:09,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-28 22:56:09,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-28 22:56:09,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-28 22:56:09,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-28 22:56:09,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-28 22:56:09,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-28 22:56:09,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-28 22:56:09,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-28 22:56:09,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-28 22:56:09,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-28 22:56:09,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-28 22:56:09,877 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021375566127829166); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03158015828501177, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021375566127829166); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043416666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 22:56:09,878 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-28 22:56:09,878 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.012781457970785082, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021375566127829166); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03158015828501177, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021375566127829166); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043416666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-28 22:56:10,202 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 456 ms to try 5000 different iterations.  Found a solution that moves 955 regions; Going from a computed imbalance of 0.012781457970785082 to a new imbalance of 7.014728644282809E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006779810554212878); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008761716542303613); MoveCostFunction : (multiplier=7.0, imbalance=0.10611111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006779810554212878); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 22:56:10,204 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv3211682:81 , srv953981088:82 , srv1124558547:83 , srv1246006536:83 , srv1293883430:83 , srv1298076497:83 , srv1937599842:83 , srv195170720:83 , srv2031712906:83 , srv611942608:83 , srv765878035:83 , srv1144076507:84 , srv1333848749:84 , srv1404593730:84 , srv1416451051:84 , srv1485477795:84 , srv1532661215:84 , srv1728447493:84 , srv1874755382:84 , srv1898235659:84 , srv1915187457:84 , srv192277820:84 , srv1930829159:84 , srv2105787785:84 , srv351785965:84 , srv556523821:84 , srv678600084:84 , srv729938898:84 , srv858847199:84 , srv860979120:84 , srv874495266:84 , srv1100153732:85 , srv1136324979:85 , srv1225635612:85 , srv1242637818:85 , srv1248788017:85 , srv1305432105:85 , srv1337785245:85 , srv1364599815:85 , srv1426822100:85 , srv1564931578:85 , srv1587715539:85 , srv173423467:85 , srv1795020122:85 , srv1809127233:85 , srv2067911860:85 , srv2099897182:85 , srv2135663415:85 , srv217931015:85 , srv220091052:85 , srv325797459:85 , srv342880967:85 , srv500645911:85 , srv513794824:85 , srv637548025:85 , srv692003784:85 , srv720807655:85 , srv1016883791:86 , srv1298366630:86 , srv1434279016:86 , srv146134621:86 , srv1729096831:86 , srv176020563:86 , srv1907098184:86 , srv1942887615:86 , srv2048997899:86 , srv244832148:86 , srv294610619:86 , srv328678554:86 , srv388977332:86 , srv403784332:86 , srv476740148:86 , srv504889239:86 , srv667629708:86 , srv717703021:86 , srv938950966:86 , srv131963353:87 , srv1373941404:87 , srv1525616221:87 , srv1784000568:87 , srv1870413341:87 , srv1929180431:87 , srv2127241614:87 , srv358944971:87 , srv456331258:87 , srv601188762:87 , srv718451227:87 , srv737356488:87 , srv1084417228:88 , srv1257882259:88 , srv1771182410:88 , srv1822672305:88 , srv1971135070:88 , srv2075260001:88 , srv429964645:88 , srv452158090:88 , srv477377218:88 , srv498570630:88 , srv952161018:88 , srv2093342793:554 }
2023-03-28 22:56:10,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-28 22:56:10,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1587715539=29, srv858847199=94, srv1416451051=21, srv504889239=79, srv601188762=82, srv1729096831=31, srv146134621=24, srv1874755382=40, srv1898235659=41, srv717703021=88, srv1870413341=39, srv1333848749=16, srv1971135070=50, srv1784000568=35, srv192277820=44, srv720807655=90, srv342880967=67, srv1404593730=20, srv173423467=32, srv2067911860=53, srv2105787785=57, srv1124558547=3, srv860979120=95, srv131963353=15, srv718451227=89, srv1298076497=12, srv1298366630=13, srv388977332=70, srv1915187457=43, srv1136324979=4, srv938950966=97, srv1434279016=23, srv1426822100=22, srv1771182410=34, srv1084417228=1, srv456331258=74, srv667629708=85, srv358944971=69, srv294610619=63, srv1144076507=5, srv1728447493=30, srv176020563=33, srv637548025=84, srv765878035=93, srv692003784=87, srv611942608=83, srv1822672305=38, srv476740148=75, srv678600084=86, srv220091052=61, srv1246006536=8, srv1942887615=48, srv3211682=64, srv2093342793=55, srv500645911=78, srv1337785245=17, srv953981088=99, srv1225635612=6, srv2031712906=51, srv1016883791=0, srv1532661215=27, srv1809127233=37, srv325797459=65, srv556523821=81, srv351785965=68, srv1525616221=26, srv1248788017=9, srv2099897182=56, srv1930829159=46, srv452158090=73, srv1564931578=28, srv1100153732=2, srv737356488=92, srv1305432105=14, srv1485477795=25, srv1364599815=18, srv429964645=72, srv1373941404=19, srv328678554=66, srv1257882259=10, srv195170720=49, srv477377218=76, srv217931015=60, srv1242637818=7, srv244832148=62, srv1795020122=36, srv2135663415=59, srv2127241614=58, srv952161018=98, srv1929180431=45, srv403784332=71, srv513794824=80, srv2075260001=54, srv1907098184=42, srv1937599842=47, srv2048997899=52, srv874495266=96, srv498570630=77, srv1293883430=11, srv729938898=91} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-28 22:56:10,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-28 22:56:10,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-28 22:56:10,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-28 22:56:10,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-28 22:56:10,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-28 22:56:10,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-28 22:56:10,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-28 22:56:10,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-28 22:56:10,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-28 22:56:10,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-28 22:56:10,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-28 22:56:10,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-28 22:56:10,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-28 22:56:10,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-28 22:56:10,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-28 22:56:10,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-28 22:56:10,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-28 22:56:10,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-28 22:56:10,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-28 22:56:10,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-28 22:56:10,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-28 22:56:10,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-28 22:56:10,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-28 22:56:10,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-28 22:56:10,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-28 22:56:10,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-28 22:56:10,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-28 22:56:10,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-28 22:56:10,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-28 22:56:10,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-28 22:56:10,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-28 22:56:10,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-28 22:56:10,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-28 22:56:10,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-28 22:56:10,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-28 22:56:10,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-28 22:56:10,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-28 22:56:10,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-28 22:56:10,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-28 22:56:10,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-28 22:56:10,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-28 22:56:10,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-28 22:56:10,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-28 22:56:10,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-28 22:56:10,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-28 22:56:10,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-28 22:56:10,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-28 22:56:10,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-28 22:56:10,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-28 22:56:10,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-28 22:56:10,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-28 22:56:10,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-28 22:56:10,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-28 22:56:10,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-28 22:56:10,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-28 22:56:10,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-28 22:56:10,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-28 22:56:10,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-28 22:56:10,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-28 22:56:10,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-28 22:56:10,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-28 22:56:10,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-28 22:56:10,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-28 22:56:10,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-28 22:56:10,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-28 22:56:10,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-28 22:56:10,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-28 22:56:10,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-28 22:56:10,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-28 22:56:10,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-28 22:56:10,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-28 22:56:10,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-28 22:56:10,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-28 22:56:10,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-28 22:56:10,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-28 22:56:10,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-28 22:56:10,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-28 22:56:10,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-28 22:56:10,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-28 22:56:10,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-28 22:56:10,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-28 22:56:10,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-28 22:56:10,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-28 22:56:10,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-28 22:56:10,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-28 22:56:10,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-28 22:56:10,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-28 22:56:10,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-28 22:56:10,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-28 22:56:10,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-28 22:56:10,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-28 22:56:10,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-28 22:56:10,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-28 22:56:10,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-28 22:56:10,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-28 22:56:10,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-28 22:56:10,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-28 22:56:10,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-28 22:56:10,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-28 22:56:10,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-28 22:56:10,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-28 22:56:10,389 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006779810554212878); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008761716542303613); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006779810554212878); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 22:56:10,389 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-28 22:56:10,389 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.947858115030327E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006779810554212878); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008761716542303613); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006779810554212878); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-28 22:56:10,833 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 581 ms to try 5000 different iterations.  Found a solution that moves 275 regions; Going from a computed imbalance of 6.947858115030327E-4 to a new imbalance of 1.9034262635180772E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=2.2334173672426932E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.030555555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=2.2334173672426932E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 22:56:10,834 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1525616221:83 , srv1771182410:83 , srv1929180431:83 , srv2075260001:83 , srv601188762:83 , srv1016883791:84 , srv1084417228:84 , srv1225635612:84 , srv1246006536:84 , srv1257882259:84 , srv1298366630:84 , srv131963353:84 , srv1337785245:84 , srv1373941404:84 , srv1434279016:84 , srv1728447493:84 , srv1784000568:84 , srv1795020122:84 , srv1822672305:84 , srv1870413341:84 , srv1907098184:84 , srv1930829159:84 , srv1971135070:84 , srv2048997899:84 , srv2127241614:84 , srv342880967:84 , srv358944971:84 , srv429964645:84 , srv452158090:84 , srv456331258:84 , srv477377218:84 , srv498570630:84 , srv500645911:84 , srv513794824:84 , srv611942608:84 , srv667629708:84 , srv692003784:84 , srv718451227:84 , srv720807655:84 , srv737356488:84 , srv858847199:84 , srv952161018:84 , srv953981088:84 , srv1136324979:85 , srv1144076507:85 , srv1242637818:85 , srv1248788017:85 , srv1293883430:85 , srv1298076497:85 , srv1305432105:85 , srv1333848749:85 , srv1364599815:85 , srv1404593730:85 , srv1416451051:85 , srv1426822100:85 , srv146134621:85 , srv1532661215:85 , srv1729096831:85 , srv173423467:85 , srv176020563:85 , srv1809127233:85 , srv1874755382:85 , srv1898235659:85 , srv192277820:85 , srv1937599842:85 , srv1942887615:85 , srv195170720:85 , srv2031712906:85 , srv2099897182:85 , srv2105787785:85 , srv217931015:85 , srv220091052:85 , srv244832148:85 , srv294610619:85 , srv3211682:85 , srv325797459:85 , srv328678554:85 , srv351785965:85 , srv388977332:85 , srv403784332:85 , srv476740148:85 , srv504889239:85 , srv637548025:85 , srv678600084:85 , srv717703021:85 , srv729938898:85 , srv765878035:85 , srv860979120:85 , srv874495266:85 , srv938950966:85 , srv1100153732:86 , srv1124558547:86 , srv1485477795:86 , srv1564931578:86 , srv1587715539:86 , srv1915187457:86 , srv2067911860:86 , srv2135663415:86 , srv556523821:86 , srv2093342793:624 }
2023-03-28 22:56:10,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-28 22:56:10,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1587715539=29, srv858847199=94, srv1416451051=21, srv504889239=79, srv601188762=82, srv1729096831=31, srv146134621=24, srv1874755382=40, srv1898235659=41, srv717703021=88, srv1870413341=39, srv1333848749=16, srv1971135070=50, srv1784000568=35, srv192277820=44, srv720807655=90, srv342880967=67, srv1404593730=20, srv173423467=32, srv2067911860=53, srv2105787785=57, srv1124558547=3, srv860979120=95, srv131963353=15, srv718451227=89, srv1298076497=12, srv1298366630=13, srv388977332=70, srv1915187457=43, srv1136324979=4, srv938950966=97, srv1434279016=23, srv1426822100=22, srv1771182410=34, srv1084417228=1, srv456331258=74, srv667629708=85, srv358944971=69, srv294610619=63, srv1144076507=5, srv1728447493=30, srv176020563=33, srv637548025=84, srv765878035=93, srv692003784=87, srv611942608=83, srv1822672305=38, srv476740148=75, srv678600084=86, srv220091052=61, srv1246006536=8, srv1942887615=48, srv3211682=64, srv2093342793=55, srv500645911=78, srv1337785245=17, srv953981088=99, srv1225635612=6, srv2031712906=51, srv1016883791=0, srv1532661215=27, srv1809127233=37, srv325797459=65, srv556523821=81, srv351785965=68, srv1525616221=26, srv1248788017=9, srv2099897182=56, srv1930829159=46, srv452158090=73, srv1564931578=28, srv1100153732=2, srv737356488=92, srv1305432105=14, srv1485477795=25, srv1364599815=18, srv429964645=72, srv1373941404=19, srv328678554=66, srv1257882259=10, srv195170720=49, srv477377218=76, srv217931015=60, srv1242637818=7, srv244832148=62, srv1795020122=36, srv2135663415=59, srv2127241614=58, srv952161018=98, srv1929180431=45, srv403784332=71, srv513794824=80, srv2075260001=54, srv1907098184=42, srv1937599842=47, srv2048997899=52, srv874495266=96, srv498570630=77, srv1293883430=11, srv729938898=91} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-28 22:56:10,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-28 22:56:10,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-28 22:56:10,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-28 22:56:10,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-28 22:56:10,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-28 22:56:10,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-28 22:56:10,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-28 22:56:10,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-28 22:56:10,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-28 22:56:10,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-28 22:56:10,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-28 22:56:10,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-28 22:56:10,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-28 22:56:10,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-28 22:56:10,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-28 22:56:10,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-28 22:56:10,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-28 22:56:10,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-28 22:56:10,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-28 22:56:10,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-28 22:56:10,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-28 22:56:10,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-28 22:56:10,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-28 22:56:10,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-28 22:56:10,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-28 22:56:10,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-28 22:56:10,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-28 22:56:10,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-28 22:56:10,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-28 22:56:10,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-28 22:56:10,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-28 22:56:10,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-28 22:56:10,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-28 22:56:10,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-28 22:56:10,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-28 22:56:10,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-28 22:56:10,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-28 22:56:10,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-28 22:56:10,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-28 22:56:10,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-28 22:56:10,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-28 22:56:10,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-28 22:56:10,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-28 22:56:10,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-28 22:56:10,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-28 22:56:10,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-28 22:56:10,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-28 22:56:10,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-28 22:56:10,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-28 22:56:10,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-28 22:56:10,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-28 22:56:10,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-28 22:56:10,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-28 22:56:10,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-28 22:56:10,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-28 22:56:10,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-28 22:56:10,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-28 22:56:10,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-28 22:56:10,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-28 22:56:10,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-28 22:56:10,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-28 22:56:10,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-28 22:56:10,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-28 22:56:10,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-28 22:56:10,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-28 22:56:10,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-28 22:56:10,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-28 22:56:10,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-28 22:56:10,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-28 22:56:10,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-28 22:56:10,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-28 22:56:10,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-28 22:56:10,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-28 22:56:10,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-28 22:56:10,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-28 22:56:10,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-28 22:56:10,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-28 22:56:10,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-28 22:56:10,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-28 22:56:10,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-28 22:56:10,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-28 22:56:10,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-28 22:56:10,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-28 22:56:10,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-28 22:56:10,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-28 22:56:10,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-28 22:56:10,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-28 22:56:10,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-28 22:56:10,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-28 22:56:10,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-28 22:56:10,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-28 22:56:10,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-28 22:56:10,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-28 22:56:10,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-28 22:56:10,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-28 22:56:10,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-28 22:56:10,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-28 22:56:10,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-28 22:56:10,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-28 22:56:10,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-28 22:56:10,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-28 22:56:11,024 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.7108671478695732E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=2.2334173672426932E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=2.2334173672426932E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 22:56:11,024 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1525616221:83 , srv1771182410:83 , srv1929180431:83 , srv2075260001:83 , srv601188762:83 , srv1016883791:84 , srv1084417228:84 , srv1225635612:84 , srv1246006536:84 , srv1257882259:84 , srv1298366630:84 , srv131963353:84 , srv1337785245:84 , srv1373941404:84 , srv1434279016:84 , srv1728447493:84 , srv1784000568:84 , srv1795020122:84 , srv1822672305:84 , srv1870413341:84 , srv1907098184:84 , srv1930829159:84 , srv1971135070:84 , srv2048997899:84 , srv2127241614:84 , srv342880967:84 , srv358944971:84 , srv429964645:84 , srv452158090:84 , srv456331258:84 , srv477377218:84 , srv498570630:84 , srv500645911:84 , srv513794824:84 , srv611942608:84 , srv667629708:84 , srv692003784:84 , srv718451227:84 , srv720807655:84 , srv737356488:84 , srv858847199:84 , srv952161018:84 , srv953981088:84 , srv1136324979:85 , srv1144076507:85 , srv1242637818:85 , srv1248788017:85 , srv1293883430:85 , srv1298076497:85 , srv1305432105:85 , srv1333848749:85 , srv1364599815:85 , srv1404593730:85 , srv1416451051:85 , srv1426822100:85 , srv146134621:85 , srv1532661215:85 , srv1729096831:85 , srv173423467:85 , srv176020563:85 , srv1809127233:85 , srv1874755382:85 , srv1898235659:85 , srv192277820:85 , srv1937599842:85 , srv1942887615:85 , srv195170720:85 , srv2031712906:85 , srv2099897182:85 , srv2105787785:85 , srv217931015:85 , srv220091052:85 , srv244832148:85 , srv294610619:85 , srv3211682:85 , srv325797459:85 , srv328678554:85 , srv351785965:85 , srv388977332:85 , srv403784332:85 , srv476740148:85 , srv504889239:85 , srv637548025:85 , srv678600084:85 , srv717703021:85 , srv729938898:85 , srv765878035:85 , srv860979120:85 , srv874495266:85 , srv938950966:85 , srv1100153732:86 , srv1124558547:86 , srv1485477795:86 , srv1564931578:86 , srv1587715539:86 , srv1915187457:86 , srv2067911860:86 , srv2135663415:86 , srv556523821:86 , srv2093342793:624 }
2023-03-28 22:56:11,064 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=409 (was 375) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=7246 (was 7731)
2023-03-28 22:56:11,078 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=409, ProcessCount=169, AvailableMemoryMB=7246
2023-03-28 22:56:11,078 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-28 22:56:11,079 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-03-28 22:56:11,093 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1006016429:84 , srv1010539992:84 , srv1055262055:84 , srv1233691555:84 , srv1365168094:84 , srv1418632874:84 , srv145953532:84 , srv1614523810:84 , srv1675300138:84 , srv1680387640:84 , srv1706727826:84 , srv1730294978:84 , srv1886918194:84 , srv189816286:84 , srv1906323901:84 , srv2037573382:84 , srv2042155616:84 , srv2054092123:84 , srv2146347196:84 , srv388299971:84 , srv397705098:84 , srv551817098:84 , srv657574478:84 , srv814262981:84 , srv837994195:84 , srv895727865:84 , srv897957277:84 , srv900855469:84 , srv9310863:84 , srv395426325:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-03-28 22:56:11,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-28 22:56:11,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1706727826=10, srv895727865=26, srv1886918194=12, srv2054092123=17, srv388299971=19, srv2146347196=18, srv897957277=27, srv1675300138=8, srv9310863=29, srv1233691555=3, srv1680387640=9, srv657574478=23, srv1730294978=11, srv551817098=22, srv397705098=21, srv1006016429=0, srv2037573382=15, srv1614523810=7, srv1010539992=1, srv1055262055=2, srv395426325=20, srv900855469=28, srv189816286=13, srv837994195=25, srv1418632874=5, srv2042155616=16, srv1365168094=4, srv1906323901=14, srv145953532=6, srv814262981=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-28 22:56:11,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-28 22:56:11,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-28 22:56:11,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-28 22:56:11,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-28 22:56:11,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-28 22:56:11,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-28 22:56:11,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-28 22:56:11,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-28 22:56:11,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-28 22:56:11,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-28 22:56:11,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-28 22:56:11,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-28 22:56:11,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-28 22:56:11,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-28 22:56:11,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-28 22:56:11,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-28 22:56:11,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-28 22:56:11,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-28 22:56:11,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-28 22:56:11,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-28 22:56:11,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-28 22:56:11,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-28 22:56:11,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-28 22:56:11,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-28 22:56:11,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-28 22:56:11,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-28 22:56:11,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-28 22:56:11,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-28 22:56:11,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-28 22:56:11,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-28 22:56:11,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-28 22:56:11,139 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1229611706627087, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 22:56:11,140 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912580489897986, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1229611706627087, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-03-28 22:58:11,120 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6100911 different iterations.  Found a solution that moves 1851 regions; Going from a computed imbalance of 0.9912580489897986 to a new imbalance of 4.320326340186438E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6855555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 22:58:11,126 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1006016429:90 , srv1010539992:90 , srv1055262055:90 , srv1233691555:90 , srv1365168094:90 , srv1418632874:90 , srv145953532:90 , srv1614523810:90 , srv1675300138:90 , srv1680387640:90 , srv1706727826:90 , srv1730294978:90 , srv1886918194:90 , srv189816286:90 , srv1906323901:90 , srv2037573382:90 , srv2042155616:90 , srv2054092123:90 , srv2146347196:90 , srv388299971:90 , srv395426325:90 , srv397705098:90 , srv551817098:90 , srv657574478:90 , srv814262981:90 , srv837994195:90 , srv895727865:90 , srv897957277:90 , srv900855469:90 , srv9310863:90 }
2023-03-28 22:58:11,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-28 22:58:11,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1706727826=10, srv895727865=26, srv1886918194=12, srv2054092123=17, srv388299971=19, srv2146347196=18, srv897957277=27, srv1675300138=8, srv9310863=29, srv1233691555=3, srv1680387640=9, srv657574478=23, srv1730294978=11, srv551817098=22, srv397705098=21, srv1006016429=0, srv2037573382=15, srv1614523810=7, srv1010539992=1, srv1055262055=2, srv395426325=20, srv900855469=28, srv189816286=13, srv837994195=25, srv1418632874=5, srv2042155616=16, srv1365168094=4, srv1906323901=14, srv145953532=6, srv814262981=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-28 22:58:11,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-28 22:58:11,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-28 22:58:11,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-28 22:58:11,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-28 22:58:11,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-28 22:58:11,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-28 22:58:11,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-28 22:58:11,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-28 22:58:11,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-28 22:58:11,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-28 22:58:11,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-28 22:58:11,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-28 22:58:11,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-28 22:58:11,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-28 22:58:11,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-28 22:58:11,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-28 22:58:11,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-28 22:58:11,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-28 22:58:11,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-28 22:58:11,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-28 22:58:11,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-28 22:58:11,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-28 22:58:11,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-28 22:58:11,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-28 22:58:11,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-28 22:58:11,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-28 22:58:11,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-28 22:58:11,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-28 22:58:11,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-28 22:58:11,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-28 22:58:11,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-28 22:58:11,155 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-28 22:58:11,155 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1006016429:90 , srv1010539992:90 , srv1055262055:90 , srv1233691555:90 , srv1365168094:90 , srv1418632874:90 , srv145953532:90 , srv1614523810:90 , srv1675300138:90 , srv1680387640:90 , srv1706727826:90 , srv1730294978:90 , srv1886918194:90 , srv189816286:90 , srv1906323901:90 , srv2037573382:90 , srv2042155616:90 , srv2054092123:90 , srv2146347196:90 , srv388299971:90 , srv395426325:90 , srv397705098:90 , srv551817098:90 , srv657574478:90 , srv814262981:90 , srv837994195:90 , srv895727865:90 , srv897957277:90 , srv900855469:90 , srv9310863:90 }
2023-03-28 22:58:11,170 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=279 (was 409), ProcessCount=170 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=7221 (was 7246)