2023-10-23 23:56:01,465 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-10-23 23:56:01,718 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-10-23 23:56:01,812 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-10-23 23:56:01,867 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-10-23 23:56:01,872 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-10-23 23:56:01,884 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=238, ProcessCount=168, AvailableMemoryMB=3981
2023-10-23 23:56:01,886 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-10-23 23:56:01,886 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-10-23 23:56:02,000 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1026270914:84 , srv1035046063:84 , srv104617525:84 , srv1052595777:84 , srv1116178010:84 , srv117085024:84 , srv1175353315:84 , srv117707041:84 , srv1222660267:84 , srv1254269956:84 , srv1266508452:84 , srv1294940716:84 , srv1308088008:84 , srv1326242121:84 , srv1332529443:84 , srv1371292637:84 , srv1373037896:84 , srv1388116062:84 , srv1391941322:84 , srv1446244579:84 , srv1467029568:84 , srv1485997514:84 , srv1494672628:84 , srv1499325583:84 , srv1570447787:84 , srv1616724587:84 , srv1617655148:84 , srv1631612825:84 , srv1636574725:84 , srv1645696652:84 , srv1650352582:84 , srv1655859764:84 , srv1658126879:84 , srv1663704678:84 , srv1735704838:84 , srv1749881108:84 , srv1808366878:84 , srv1810609920:84 , srv1820799455:84 , srv1826444074:84 , srv1856266143:84 , srv1885158354:84 , srv188903726:84 , srv189309610:84 , srv1893104098:84 , srv1906474525:84 , srv1976010656:84 , srv1993128100:84 , srv2110171147:84 , srv2117440700:84 , srv2138167630:84 , srv227428860:84 , srv248301043:84 , srv252628898:84 , srv258278262:84 , srv2767234:84 , srv27743147:84 , srv303197018:84 , srv314217561:84 , srv340467461:84 , srv3518731:84 , srv385691204:84 , srv387896377:84 , srv393516551:84 , srv427274515:84 , srv428207384:84 , srv442543346:84 , srv463779282:84 , srv486573574:84 , srv489660969:84 , srv532349957:84 , srv586935963:84 , srv598304267:84 , srv608561348:84 , srv609299918:84 , srv618434094:84 , srv653845043:84 , srv667066770:84 , srv672139375:84 , srv679032083:84 , srv698182543:84 , srv7057762:84 , srv7353974:84 , srv735557786:84 , srv761429852:84 , srv797392709:84 , srv803637326:84 , srv822708484:84 , srv864246902:84 , srv865872821:84 , srv874219631:84 , srv874402522:84 , srv879755829:84 , srv899131209:84 , srv910197171:84 , srv932502380:84 , srv933995368:84 , srv941002063:84 , srv983027318:84 , srv867476024:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-10-23 23:56:02,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-23 23:56:02,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1856266143=40, srv1175353315=6, srv27743147=56, srv3518731=60, srv1570447787=24, srv1026270914=0, srv1116178010=4, srv1645696652=29, srv387896377=62, srv1650352582=30, srv1254269956=9, srv1467029568=20, srv1446244579=19, srv899131209=94, srv258278262=54, srv1371292637=15, srv428207384=65, srv486573574=68, srv797392709=85, srv1906474525=45, srv227428860=51, srv1663704678=33, srv761429852=84, srv1052595777=3, srv1373037896=16, srv1332529443=14, srv1820799455=38, srv667066770=77, srv1616724587=25, srv442543346=66, srv1655859764=31, srv1499325583=23, srv608561348=73, srv865872821=89, srv1222660267=8, srv1885158354=41, srv188903726=42, srv874402522=92, srv463779282=67, srv879755829=93, srv598304267=72, srv941002063=98, srv1388116062=17, srv1993128100=47, srv933995368=97, srv1326242121=13, srv1808366878=36, srv104617525=2, srv653845043=76, srv983027318=99, srv2138167630=50, srv1494672628=22, srv874219631=91, srv427274515=64, srv7353974=82, srv910197171=95, srv609299918=74, srv2767234=55, srv1391941322=18, srv618434094=75, srv1035046063=1, srv672139375=78, srv532349957=70, srv698182543=80, srv252628898=53, srv864246902=88, srv1308088008=12, srv1749881108=35, srv340467461=59, srv735557786=83, srv822708484=87, srv393516551=63, srv489660969=69, srv586935963=71, srv1266508452=10, srv1631612825=27, srv1735704838=34, srv1810609920=37, srv314217561=58, srv2110171147=48, srv1294940716=11, srv1658126879=32, srv303197018=57, srv189309610=43, srv117085024=5, srv803637326=86, srv679032083=79, srv1826444074=39, srv248301043=52, srv7057762=81, srv867476024=90, srv932502380=96, srv385691204=61, srv117707041=7, srv2117440700=49, srv1893104098=44, srv1976010656=46, srv1617655148=26, srv1636574725=28, srv1485997514=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-23 23:56:02,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-23 23:56:02,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-23 23:56:02,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-23 23:56:02,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-23 23:56:02,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-23 23:56:02,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-23 23:56:02,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-23 23:56:02,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-23 23:56:02,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-23 23:56:02,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-23 23:56:02,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-23 23:56:02,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-23 23:56:02,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-23 23:56:02,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-23 23:56:02,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-23 23:56:02,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-23 23:56:02,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-23 23:56:02,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-23 23:56:02,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-23 23:56:02,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-23 23:56:02,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-23 23:56:02,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-23 23:56:02,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-23 23:56:02,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-23 23:56:02,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-23 23:56:02,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-23 23:56:02,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-23 23:56:02,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-23 23:56:02,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-23 23:56:02,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-23 23:56:02,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-23 23:56:02,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-23 23:56:02,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-23 23:56:02,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-23 23:56:02,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-23 23:56:02,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-23 23:56:02,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-23 23:56:02,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-23 23:56:02,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-23 23:56:02,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-23 23:56:02,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-23 23:56:02,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-23 23:56:02,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-23 23:56:02,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-23 23:56:02,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-23 23:56:02,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-23 23:56:02,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-23 23:56:02,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-23 23:56:02,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-23 23:56:02,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-23 23:56:02,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-23 23:56:02,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-23 23:56:02,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-23 23:56:02,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-23 23:56:02,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-23 23:56:02,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-23 23:56:02,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-23 23:56:02,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-23 23:56:02,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-23 23:56:02,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-23 23:56:02,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-23 23:56:02,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-23 23:56:02,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-23 23:56:02,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-23 23:56:02,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-23 23:56:02,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-23 23:56:02,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-23 23:56:02,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-23 23:56:02,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-23 23:56:02,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-23 23:56:02,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-23 23:56:02,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-23 23:56:02,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-23 23:56:02,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-23 23:56:02,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-23 23:56:02,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-23 23:56:02,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-23 23:56:02,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-23 23:56:02,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-23 23:56:02,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-23 23:56:02,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-23 23:56:02,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-23 23:56:02,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-23 23:56:02,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-23 23:56:02,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-23 23:56:02,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-23 23:56:02,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-23 23:56:02,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-23 23:56:02,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-23 23:56:02,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-23 23:56:02,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-23 23:56:02,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-23 23:56:02,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-23 23:56:02,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-23 23:56:02,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-23 23:56:02,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-23 23:56:02,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-23 23:56:02,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-23 23:56:02,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-23 23:56:02,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-23 23:56:02,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-23 23:56:02,259 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-23 23:56:02,260 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-23 23:56:02,260 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-23 23:56:02,566 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 447 ms to try 5000 different iterations.  Found a solution that moves 4619 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1857166682177386. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03829903799393342, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05620229614554396, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5132222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03829903799393342, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18075, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.25016666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-23 23:56:02,590 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv735557786:81 , srv1749881108:82 , srv1885158354:82 , srv303197018:82 , srv428207384:82 , srv874219631:82 , srv932502380:82 , srv1254269956:83 , srv7353974:83 , srv933995368:83 , srv117707041:84 , srv1222660267:84 , srv1467029568:84 , srv1856266143:84 , srv2110171147:84 , srv27743147:84 , srv608561348:84 , srv667066770:84 , srv698182543:84 , srv864246902:84 , srv104617525:85 , srv1266508452:85 , srv1485997514:85 , srv1494672628:85 , srv1636574725:85 , srv1658126879:85 , srv1663704678:85 , srv1735704838:85 , srv1808366878:85 , srv188903726:85 , srv314217561:85 , srv385691204:85 , srv532349957:85 , srv653845043:85 , srv874402522:85 , srv1308088008:86 , srv1373037896:86 , srv1446244579:86 , srv1645696652:86 , srv189309610:86 , srv1906474525:86 , srv248301043:86 , srv2767234:86 , srv340467461:86 , srv393516551:86 , srv427274515:86 , srv463779282:86 , srv489660969:86 , srv679032083:86 , srv7057762:86 , srv803637326:86 , srv865872821:86 , srv1026270914:87 , srv1035046063:87 , srv1175353315:87 , srv1391941322:87 , srv1631612825:87 , srv1655859764:87 , srv1893104098:87 , srv2138167630:87 , srv3518731:87 , srv387896377:87 , srv586935963:87 , srv598304267:87 , srv609299918:87 , srv618434094:87 , srv672139375:87 , srv761429852:87 , srv797392709:87 , srv822708484:87 , srv879755829:87 , srv910197171:87 , srv1052595777:88 , srv1116178010:88 , srv1371292637:88 , srv1570447787:88 , srv1810609920:88 , srv1976010656:88 , srv2117440700:88 , srv941002063:88 , srv983027318:88 , srv117085024:89 , srv1388116062:89 , srv1499325583:89 , srv1993128100:89 , srv227428860:89 , srv258278262:89 , srv899131209:89 , srv1294940716:90 , srv1326242121:90 , srv1616724587:90 , srv1617655148:90 , srv486573574:90 , srv252628898:91 , srv442543346:91 , srv1332529443:93 , srv1820799455:95 , srv1826444074:95 , srv1650352582:101 , srv867476024:430 }
2023-10-23 23:56:02,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-23 23:56:02,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1856266143=40, srv1175353315=6, srv27743147=56, srv3518731=60, srv1570447787=24, srv1026270914=0, srv1116178010=4, srv1645696652=29, srv387896377=62, srv1650352582=30, srv1254269956=9, srv1467029568=20, srv1446244579=19, srv899131209=94, srv258278262=54, srv1371292637=15, srv428207384=65, srv486573574=68, srv797392709=85, srv1906474525=45, srv227428860=51, srv1663704678=33, srv761429852=84, srv1052595777=3, srv1373037896=16, srv1332529443=14, srv1820799455=38, srv667066770=77, srv1616724587=25, srv442543346=66, srv1655859764=31, srv1499325583=23, srv608561348=73, srv865872821=89, srv1222660267=8, srv1885158354=41, srv188903726=42, srv874402522=92, srv463779282=67, srv879755829=93, srv598304267=72, srv941002063=98, srv1388116062=17, srv1993128100=47, srv933995368=97, srv1326242121=13, srv1808366878=36, srv104617525=2, srv653845043=76, srv983027318=99, srv2138167630=50, srv1494672628=22, srv874219631=91, srv427274515=64, srv7353974=82, srv910197171=95, srv609299918=74, srv2767234=55, srv1391941322=18, srv618434094=75, srv1035046063=1, srv672139375=78, srv532349957=70, srv698182543=80, srv252628898=53, srv864246902=88, srv1308088008=12, srv1749881108=35, srv340467461=59, srv735557786=83, srv822708484=87, srv393516551=63, srv489660969=69, srv586935963=71, srv1266508452=10, srv1631612825=27, srv1735704838=34, srv1810609920=37, srv314217561=58, srv2110171147=48, srv1294940716=11, srv1658126879=32, srv303197018=57, srv189309610=43, srv117085024=5, srv803637326=86, srv679032083=79, srv1826444074=39, srv248301043=52, srv7057762=81, srv867476024=90, srv932502380=96, srv385691204=61, srv117707041=7, srv2117440700=49, srv1893104098=44, srv1976010656=46, srv1617655148=26, srv1636574725=28, srv1485997514=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-23 23:56:02,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-23 23:56:02,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-23 23:56:02,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-23 23:56:02,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-23 23:56:02,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-23 23:56:02,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-23 23:56:02,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-23 23:56:02,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-23 23:56:02,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-23 23:56:02,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-23 23:56:02,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-23 23:56:02,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-23 23:56:02,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-23 23:56:02,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-23 23:56:02,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-23 23:56:02,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-23 23:56:02,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-23 23:56:02,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-23 23:56:02,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-23 23:56:02,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-23 23:56:02,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-23 23:56:02,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-23 23:56:02,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-23 23:56:02,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-23 23:56:02,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-23 23:56:02,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-23 23:56:02,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-23 23:56:02,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-23 23:56:02,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-23 23:56:02,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-23 23:56:02,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-23 23:56:02,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-23 23:56:02,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-23 23:56:02,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-23 23:56:02,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-23 23:56:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-23 23:56:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-23 23:56:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-23 23:56:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-23 23:56:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-23 23:56:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-23 23:56:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-23 23:56:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-23 23:56:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-23 23:56:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-23 23:56:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-23 23:56:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-23 23:56:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-23 23:56:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-23 23:56:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-23 23:56:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-23 23:56:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-23 23:56:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-23 23:56:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-23 23:56:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-23 23:56:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-23 23:56:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-23 23:56:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-23 23:56:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-23 23:56:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-23 23:56:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-23 23:56:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-23 23:56:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-23 23:56:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-23 23:56:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-23 23:56:02,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-23 23:56:02,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-23 23:56:02,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-23 23:56:02,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-23 23:56:02,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-23 23:56:02,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-23 23:56:02,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-23 23:56:02,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-23 23:56:02,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-23 23:56:02,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-23 23:56:02,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-23 23:56:02,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-23 23:56:02,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-23 23:56:02,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-23 23:56:02,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-23 23:56:02,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-23 23:56:02,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-23 23:56:02,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-23 23:56:02,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-23 23:56:02,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-23 23:56:02,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-23 23:56:02,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-23 23:56:02,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-23 23:56:02,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-23 23:56:02,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-23 23:56:02,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-23 23:56:02,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-23 23:56:02,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-23 23:56:02,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-23 23:56:02,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-23 23:56:02,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-23 23:56:02,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-23 23:56:02,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-23 23:56:02,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-23 23:56:02,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-23 23:56:02,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-23 23:56:02,751 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03829903799393342, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05620229614554396, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03829903799393342, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18075, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.25016666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-23 23:56:02,751 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-23 23:56:02,751 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1856843252884593, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03829903799393342, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05620229614554396, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03829903799393342, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18075, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.25016666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-23 23:56:02,940 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 304 ms to try 5000 different iterations.  Found a solution that moves 2695 regions; Going from a computed imbalance of 0.1856843252884593 to a new imbalance of 0.01678423463578654. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02160766376908434); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03303910549004757, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29944444444444446, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02160766376908434); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01375); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04591666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-23 23:56:02,948 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1650352582:71 , srv1820799455:75 , srv1826444074:77 , srv1332529443:78 , srv252628898:79 , srv1616724587:81 , srv1617655148:81 , srv1326242121:82 , srv3518731:82 , srv442543346:82 , srv1294940716:83 , srv1371292637:83 , srv1388116062:83 , srv1976010656:83 , srv258278262:83 , srv387896377:83 , srv618434094:83 , srv983027318:83 , srv1116178010:84 , srv117085024:84 , srv1373037896:84 , srv1570447787:84 , srv1810609920:84 , srv1993128100:84 , srv2117440700:84 , srv227428860:84 , srv340467461:84 , srv486573574:84 , srv598304267:84 , srv879755829:84 , srv1026270914:85 , srv1035046063:85 , srv104617525:85 , srv1175353315:85 , srv1266508452:85 , srv1391941322:85 , srv1485997514:85 , srv1494672628:85 , srv1499325583:85 , srv1631612825:85 , srv189309610:85 , srv1893104098:85 , srv2138167630:85 , srv586935963:85 , srv672139375:85 , srv761429852:85 , srv941002063:85 , srv1052595777:86 , srv1655859764:86 , srv1735704838:86 , srv248301043:86 , srv314217561:86 , srv427274515:86 , srv653845043:86 , srv679032083:86 , srv7057762:86 , srv797392709:86 , srv803637326:86 , srv865872821:86 , srv874402522:86 , srv910197171:86 , srv933995368:86 , srv1308088008:87 , srv1645696652:87 , srv1658126879:87 , srv1906474525:87 , srv2110171147:87 , srv2767234:87 , srv27743147:87 , srv489660969:87 , srv609299918:87 , srv822708484:87 , srv899131209:87 , srv1222660267:88 , srv1446244579:88 , srv1636574725:88 , srv1749881108:88 , srv1808366878:88 , srv1885158354:88 , srv188903726:88 , srv303197018:88 , srv393516551:88 , srv428207384:88 , srv698182543:88 , srv117707041:89 , srv1467029568:89 , srv385691204:89 , srv463779282:89 , srv532349957:89 , srv608561348:89 , srv667066770:89 , srv7353974:89 , srv735557786:89 , srv1254269956:90 , srv1856266143:90 , srv864246902:90 , srv932502380:90 , srv1663704678:91 , srv874219631:92 , srv867476024:536 }
2023-10-23 23:56:02,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-23 23:56:02,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1856266143=40, srv1175353315=6, srv27743147=56, srv3518731=60, srv1570447787=24, srv1026270914=0, srv1116178010=4, srv1645696652=29, srv387896377=62, srv1650352582=30, srv1254269956=9, srv1467029568=20, srv1446244579=19, srv899131209=94, srv258278262=54, srv1371292637=15, srv428207384=65, srv486573574=68, srv797392709=85, srv1906474525=45, srv227428860=51, srv1663704678=33, srv761429852=84, srv1052595777=3, srv1373037896=16, srv1332529443=14, srv1820799455=38, srv667066770=77, srv1616724587=25, srv442543346=66, srv1655859764=31, srv1499325583=23, srv608561348=73, srv865872821=89, srv1222660267=8, srv1885158354=41, srv188903726=42, srv874402522=92, srv463779282=67, srv879755829=93, srv598304267=72, srv941002063=98, srv1388116062=17, srv1993128100=47, srv933995368=97, srv1326242121=13, srv1808366878=36, srv104617525=2, srv653845043=76, srv983027318=99, srv2138167630=50, srv1494672628=22, srv874219631=91, srv427274515=64, srv7353974=82, srv910197171=95, srv609299918=74, srv2767234=55, srv1391941322=18, srv618434094=75, srv1035046063=1, srv672139375=78, srv532349957=70, srv698182543=80, srv252628898=53, srv864246902=88, srv1308088008=12, srv1749881108=35, srv340467461=59, srv735557786=83, srv822708484=87, srv393516551=63, srv489660969=69, srv586935963=71, srv1266508452=10, srv1631612825=27, srv1735704838=34, srv1810609920=37, srv314217561=58, srv2110171147=48, srv1294940716=11, srv1658126879=32, srv303197018=57, srv189309610=43, srv117085024=5, srv803637326=86, srv679032083=79, srv1826444074=39, srv248301043=52, srv7057762=81, srv867476024=90, srv932502380=96, srv385691204=61, srv117707041=7, srv2117440700=49, srv1893104098=44, srv1976010656=46, srv1617655148=26, srv1636574725=28, srv1485997514=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-23 23:56:02,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-23 23:56:02,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-23 23:56:02,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-23 23:56:02,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-23 23:56:02,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-23 23:56:02,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-23 23:56:02,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-23 23:56:02,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-23 23:56:02,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-23 23:56:02,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-23 23:56:02,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-23 23:56:02,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-23 23:56:02,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-23 23:56:02,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-23 23:56:02,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-23 23:56:02,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-23 23:56:02,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-23 23:56:02,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-23 23:56:02,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-23 23:56:02,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-23 23:56:02,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-23 23:56:02,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-23 23:56:02,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-23 23:56:02,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-23 23:56:02,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-23 23:56:02,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-23 23:56:02,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-23 23:56:02,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-23 23:56:02,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-23 23:56:02,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-23 23:56:02,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-23 23:56:02,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-23 23:56:02,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-23 23:56:02,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-23 23:56:02,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-23 23:56:02,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-23 23:56:02,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-23 23:56:02,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-23 23:56:02,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-23 23:56:02,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-23 23:56:02,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-23 23:56:02,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-23 23:56:02,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-23 23:56:02,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-23 23:56:02,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-23 23:56:02,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-23 23:56:02,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-23 23:56:02,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-23 23:56:02,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-23 23:56:02,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-23 23:56:02,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-23 23:56:02,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-23 23:56:02,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-23 23:56:02,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-23 23:56:02,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-23 23:56:02,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-23 23:56:02,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-23 23:56:02,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-23 23:56:02,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-23 23:56:02,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-23 23:56:02,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-23 23:56:02,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-23 23:56:02,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-23 23:56:02,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-23 23:56:02,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-23 23:56:02,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-23 23:56:02,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-23 23:56:02,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-23 23:56:02,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-23 23:56:02,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-23 23:56:02,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-23 23:56:02,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-23 23:56:02,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-23 23:56:02,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-23 23:56:02,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-23 23:56:02,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-23 23:56:02,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-23 23:56:02,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-23 23:56:02,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-23 23:56:02,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-23 23:56:02,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-23 23:56:02,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-23 23:56:02,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-23 23:56:02,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-23 23:56:02,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-23 23:56:02,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-23 23:56:02,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-23 23:56:02,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-23 23:56:02,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-23 23:56:02,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-23 23:56:02,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-23 23:56:02,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-23 23:56:02,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-23 23:56:02,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-23 23:56:02,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-23 23:56:02,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-23 23:56:02,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-23 23:56:02,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-23 23:56:02,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-23 23:56:02,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-23 23:56:02,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-23 23:56:03,075 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02160766376908434); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03303910549004757, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02160766376908434); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01375); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04591666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-23 23:56:03,075 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-23 23:56:03,076 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016765363842452988, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02160766376908434); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03303910549004757, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02160766376908434); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01375); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04591666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-23 23:56:03,230 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 231 ms to try 5000 different iterations.  Found a solution that moves 1043 regions; Going from a computed imbalance of 0.016765363842452988 to a new imbalance of 8.378960007907937E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0045635007772605735); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006303224550231391); MoveCostFunction : (multiplier=7.0, imbalance=0.11588888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0045635007772605735); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-23 23:56:03,231 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv899131209:81 , srv117085024:83 , srv1663704678:83 , srv385691204:83 , srv797392709:83 , srv864246902:83 , srv1052595777:84 , srv117707041:84 , srv1254269956:84 , srv1446244579:84 , srv1467029568:84 , srv1499325583:84 , srv1645696652:84 , srv1655859764:84 , srv1826444074:84 , srv1856266143:84 , srv2767234:84 , srv393516551:84 , srv463779282:84 , srv486573574:84 , srv532349957:84 , srv608561348:84 , srv609299918:84 , srv7057762:84 , srv822708484:84 , srv874219631:84 , srv941002063:84 , srv1026270914:85 , srv1175353315:85 , srv1308088008:85 , srv1326242121:85 , srv1391941322:85 , srv1636574725:85 , srv1658126879:85 , srv1808366878:85 , srv1810609920:85 , srv1906474525:85 , srv227428860:85 , srv258278262:85 , srv427274515:85 , srv442543346:85 , srv489660969:85 , srv667066770:85 , srv672139375:85 , srv698182543:85 , srv803637326:85 , srv910197171:85 , srv1035046063:86 , srv1222660267:86 , srv1294940716:86 , srv1332529443:86 , srv1371292637:86 , srv1388116062:86 , srv1485997514:86 , srv1616724587:86 , srv1631612825:86 , srv1650352582:86 , srv1735704838:86 , srv188903726:86 , srv1893104098:86 , srv1993128100:86 , srv2117440700:86 , srv2138167630:86 , srv27743147:86 , srv314217561:86 , srv586935963:86 , srv653845043:86 , srv7353974:86 , srv761429852:86 , srv865872821:86 , srv932502380:86 , srv104617525:87 , srv1116178010:87 , srv1266508452:87 , srv1570447787:87 , srv1617655148:87 , srv1885158354:87 , srv189309610:87 , srv1976010656:87 , srv2110171147:87 , srv248301043:87 , srv252628898:87 , srv303197018:87 , srv340467461:87 , srv387896377:87 , srv428207384:87 , srv598304267:87 , srv618434094:87 , srv679032083:87 , srv735557786:87 , srv879755829:87 , srv983027318:87 , srv1373037896:88 , srv1494672628:88 , srv1749881108:88 , srv1820799455:88 , srv3518731:88 , srv874402522:88 , srv933995368:89 , srv867476024:532 }
2023-10-23 23:56:03,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-23 23:56:03,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1856266143=40, srv1175353315=6, srv27743147=56, srv3518731=60, srv1570447787=24, srv1026270914=0, srv1116178010=4, srv1645696652=29, srv387896377=62, srv1650352582=30, srv1254269956=9, srv1467029568=20, srv1446244579=19, srv899131209=94, srv258278262=54, srv1371292637=15, srv428207384=65, srv486573574=68, srv797392709=85, srv1906474525=45, srv227428860=51, srv1663704678=33, srv761429852=84, srv1052595777=3, srv1373037896=16, srv1332529443=14, srv1820799455=38, srv667066770=77, srv1616724587=25, srv442543346=66, srv1655859764=31, srv1499325583=23, srv608561348=73, srv865872821=89, srv1222660267=8, srv1885158354=41, srv188903726=42, srv874402522=92, srv463779282=67, srv879755829=93, srv598304267=72, srv941002063=98, srv1388116062=17, srv1993128100=47, srv933995368=97, srv1326242121=13, srv1808366878=36, srv104617525=2, srv653845043=76, srv983027318=99, srv2138167630=50, srv1494672628=22, srv874219631=91, srv427274515=64, srv7353974=82, srv910197171=95, srv609299918=74, srv2767234=55, srv1391941322=18, srv618434094=75, srv1035046063=1, srv672139375=78, srv532349957=70, srv698182543=80, srv252628898=53, srv864246902=88, srv1308088008=12, srv1749881108=35, srv340467461=59, srv735557786=83, srv822708484=87, srv393516551=63, srv489660969=69, srv586935963=71, srv1266508452=10, srv1631612825=27, srv1735704838=34, srv1810609920=37, srv314217561=58, srv2110171147=48, srv1294940716=11, srv1658126879=32, srv303197018=57, srv189309610=43, srv117085024=5, srv803637326=86, srv679032083=79, srv1826444074=39, srv248301043=52, srv7057762=81, srv867476024=90, srv932502380=96, srv385691204=61, srv117707041=7, srv2117440700=49, srv1893104098=44, srv1976010656=46, srv1617655148=26, srv1636574725=28, srv1485997514=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-23 23:56:03,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-23 23:56:03,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-23 23:56:03,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-23 23:56:03,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-23 23:56:03,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-23 23:56:03,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-23 23:56:03,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-23 23:56:03,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-23 23:56:03,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-23 23:56:03,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-23 23:56:03,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-23 23:56:03,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-23 23:56:03,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-23 23:56:03,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-23 23:56:03,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-23 23:56:03,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-23 23:56:03,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-23 23:56:03,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-23 23:56:03,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-23 23:56:03,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-23 23:56:03,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-23 23:56:03,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-23 23:56:03,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-23 23:56:03,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-23 23:56:03,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-23 23:56:03,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-23 23:56:03,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-23 23:56:03,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-23 23:56:03,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-23 23:56:03,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-23 23:56:03,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-23 23:56:03,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-23 23:56:03,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-23 23:56:03,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-23 23:56:03,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-23 23:56:03,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-23 23:56:03,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-23 23:56:03,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-23 23:56:03,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-23 23:56:03,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-23 23:56:03,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-23 23:56:03,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-23 23:56:03,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-23 23:56:03,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-23 23:56:03,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-23 23:56:03,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-23 23:56:03,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-23 23:56:03,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-23 23:56:03,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-23 23:56:03,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-23 23:56:03,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-23 23:56:03,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-23 23:56:03,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-23 23:56:03,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-23 23:56:03,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-23 23:56:03,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-23 23:56:03,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-23 23:56:03,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-23 23:56:03,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-23 23:56:03,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-23 23:56:03,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-23 23:56:03,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-23 23:56:03,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-23 23:56:03,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-23 23:56:03,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-23 23:56:03,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-23 23:56:03,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-23 23:56:03,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-23 23:56:03,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-23 23:56:03,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-23 23:56:03,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-23 23:56:03,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-23 23:56:03,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-23 23:56:03,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-23 23:56:03,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-23 23:56:03,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-23 23:56:03,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-23 23:56:03,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-23 23:56:03,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-23 23:56:03,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-23 23:56:03,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-23 23:56:03,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-23 23:56:03,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-23 23:56:03,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-23 23:56:03,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-23 23:56:03,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-23 23:56:03,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-23 23:56:03,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-23 23:56:03,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-23 23:56:03,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-23 23:56:03,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-23 23:56:03,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-23 23:56:03,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-23 23:56:03,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-23 23:56:03,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-23 23:56:03,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-23 23:56:03,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-23 23:56:03,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-23 23:56:03,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-23 23:56:03,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-23 23:56:03,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-23 23:56:03,345 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0045635007772605735); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006303224550231391); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0045635007772605735); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-23 23:56:03,345 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-23 23:56:03,345 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.305927586954705E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0045635007772605735); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006303224550231391); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0045635007772605735); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-23 23:56:03,560 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 288 ms to try 5000 different iterations.  Found a solution that moves 208 regions; Going from a computed imbalance of 8.305927586954705E-4 to a new imbalance of 1.31454424679483E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.02311111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-23 23:56:03,561 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv188903726:82 , srv1116178010:83 , srv1294940716:83 , srv1570447787:83 , srv1663704678:83 , srv1993128100:83 , srv2138167630:83 , srv248301043:83 , srv679032083:83 , srv874402522:83 , srv1035046063:84 , srv1052595777:84 , srv1222660267:84 , srv1266508452:84 , srv1308088008:84 , srv1332529443:84 , srv1373037896:84 , srv1388116062:84 , srv1446244579:84 , srv1494672628:84 , srv1499325583:84 , srv1617655148:84 , srv1631612825:84 , srv1636574725:84 , srv1650352582:84 , srv1749881108:84 , srv1808366878:84 , srv1820799455:84 , srv1856266143:84 , srv189309610:84 , srv1893104098:84 , srv1976010656:84 , srv2117440700:84 , srv227428860:84 , srv258278262:84 , srv314217561:84 , srv393516551:84 , srv442543346:84 , srv463779282:84 , srv486573574:84 , srv489660969:84 , srv532349957:84 , srv586935963:84 , srv598304267:84 , srv609299918:84 , srv667066770:84 , srv7353974:84 , srv761429852:84 , srv822708484:84 , srv865872821:84 , srv874219631:84 , srv879755829:84 , srv899131209:84 , srv932502380:84 , srv1026270914:85 , srv104617525:85 , srv1175353315:85 , srv1254269956:85 , srv1326242121:85 , srv1371292637:85 , srv1391941322:85 , srv1467029568:85 , srv1616724587:85 , srv1645696652:85 , srv1655859764:85 , srv1658126879:85 , srv1735704838:85 , srv1810609920:85 , srv1885158354:85 , srv1906474525:85 , srv2110171147:85 , srv252628898:85 , srv2767234:85 , srv27743147:85 , srv303197018:85 , srv340467461:85 , srv3518731:85 , srv385691204:85 , srv387896377:85 , srv427274515:85 , srv428207384:85 , srv608561348:85 , srv618434094:85 , srv653845043:85 , srv672139375:85 , srv698182543:85 , srv735557786:85 , srv803637326:85 , srv864246902:85 , srv910197171:85 , srv933995368:85 , srv941002063:85 , srv983027318:85 , srv117085024:86 , srv117707041:86 , srv1826444074:86 , srv7057762:86 , srv797392709:86 , srv1485997514:87 , srv867476024:643 }
2023-10-23 23:56:03,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-23 23:56:03,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1856266143=40, srv1175353315=6, srv27743147=56, srv3518731=60, srv1570447787=24, srv1026270914=0, srv1116178010=4, srv1645696652=29, srv387896377=62, srv1650352582=30, srv1254269956=9, srv1467029568=20, srv1446244579=19, srv899131209=94, srv258278262=54, srv1371292637=15, srv428207384=65, srv486573574=68, srv797392709=85, srv1906474525=45, srv227428860=51, srv1663704678=33, srv761429852=84, srv1052595777=3, srv1373037896=16, srv1332529443=14, srv1820799455=38, srv667066770=77, srv1616724587=25, srv442543346=66, srv1655859764=31, srv1499325583=23, srv608561348=73, srv865872821=89, srv1222660267=8, srv1885158354=41, srv188903726=42, srv874402522=92, srv463779282=67, srv879755829=93, srv598304267=72, srv941002063=98, srv1388116062=17, srv1993128100=47, srv933995368=97, srv1326242121=13, srv1808366878=36, srv104617525=2, srv653845043=76, srv983027318=99, srv2138167630=50, srv1494672628=22, srv874219631=91, srv427274515=64, srv7353974=82, srv910197171=95, srv609299918=74, srv2767234=55, srv1391941322=18, srv618434094=75, srv1035046063=1, srv672139375=78, srv532349957=70, srv698182543=80, srv252628898=53, srv864246902=88, srv1308088008=12, srv1749881108=35, srv340467461=59, srv735557786=83, srv822708484=87, srv393516551=63, srv489660969=69, srv586935963=71, srv1266508452=10, srv1631612825=27, srv1735704838=34, srv1810609920=37, srv314217561=58, srv2110171147=48, srv1294940716=11, srv1658126879=32, srv303197018=57, srv189309610=43, srv117085024=5, srv803637326=86, srv679032083=79, srv1826444074=39, srv248301043=52, srv7057762=81, srv867476024=90, srv932502380=96, srv385691204=61, srv117707041=7, srv2117440700=49, srv1893104098=44, srv1976010656=46, srv1617655148=26, srv1636574725=28, srv1485997514=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-23 23:56:03,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-23 23:56:03,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-23 23:56:03,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-23 23:56:03,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-23 23:56:03,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-23 23:56:03,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-23 23:56:03,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-23 23:56:03,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-23 23:56:03,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-23 23:56:03,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-23 23:56:03,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-23 23:56:03,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-23 23:56:03,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-23 23:56:03,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-23 23:56:03,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-23 23:56:03,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-23 23:56:03,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-23 23:56:03,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-23 23:56:03,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-23 23:56:03,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-23 23:56:03,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-23 23:56:03,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-23 23:56:03,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-23 23:56:03,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-23 23:56:03,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-23 23:56:03,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-23 23:56:03,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-23 23:56:03,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-23 23:56:03,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-23 23:56:03,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-23 23:56:03,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-23 23:56:03,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-23 23:56:03,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-23 23:56:03,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-23 23:56:03,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-23 23:56:03,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-23 23:56:03,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-23 23:56:03,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-23 23:56:03,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-23 23:56:03,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-23 23:56:03,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-23 23:56:03,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-23 23:56:03,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-23 23:56:03,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-23 23:56:03,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-23 23:56:03,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-23 23:56:03,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-23 23:56:03,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-23 23:56:03,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-23 23:56:03,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-23 23:56:03,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-23 23:56:03,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-23 23:56:03,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-23 23:56:03,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-23 23:56:03,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-23 23:56:03,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-23 23:56:03,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-23 23:56:03,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-23 23:56:03,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-23 23:56:03,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-23 23:56:03,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-23 23:56:03,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-23 23:56:03,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-23 23:56:03,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-23 23:56:03,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-23 23:56:03,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-23 23:56:03,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-23 23:56:03,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-23 23:56:03,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-23 23:56:03,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-23 23:56:03,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-23 23:56:03,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-23 23:56:03,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-23 23:56:03,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-23 23:56:03,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-23 23:56:03,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-23 23:56:03,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-23 23:56:03,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-23 23:56:03,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-23 23:56:03,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-23 23:56:03,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-23 23:56:03,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-23 23:56:03,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-23 23:56:03,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-23 23:56:03,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-23 23:56:03,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-23 23:56:03,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-23 23:56:03,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-23 23:56:03,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-23 23:56:03,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-23 23:56:03,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-23 23:56:03,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-23 23:56:03,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-23 23:56:03,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-23 23:56:03,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-23 23:56:03,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-23 23:56:03,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-23 23:56:03,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-23 23:56:03,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-23 23:56:03,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-23 23:56:03,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-23 23:56:03,660 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.1688995338679613E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-23 23:56:03,660 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv188903726:82 , srv1116178010:83 , srv1294940716:83 , srv1570447787:83 , srv1663704678:83 , srv1993128100:83 , srv2138167630:83 , srv248301043:83 , srv679032083:83 , srv874402522:83 , srv1035046063:84 , srv1052595777:84 , srv1222660267:84 , srv1266508452:84 , srv1308088008:84 , srv1332529443:84 , srv1373037896:84 , srv1388116062:84 , srv1446244579:84 , srv1494672628:84 , srv1499325583:84 , srv1617655148:84 , srv1631612825:84 , srv1636574725:84 , srv1650352582:84 , srv1749881108:84 , srv1808366878:84 , srv1820799455:84 , srv1856266143:84 , srv189309610:84 , srv1893104098:84 , srv1976010656:84 , srv2117440700:84 , srv227428860:84 , srv258278262:84 , srv314217561:84 , srv393516551:84 , srv442543346:84 , srv463779282:84 , srv486573574:84 , srv489660969:84 , srv532349957:84 , srv586935963:84 , srv598304267:84 , srv609299918:84 , srv667066770:84 , srv7353974:84 , srv761429852:84 , srv822708484:84 , srv865872821:84 , srv874219631:84 , srv879755829:84 , srv899131209:84 , srv932502380:84 , srv1026270914:85 , srv104617525:85 , srv1175353315:85 , srv1254269956:85 , srv1326242121:85 , srv1371292637:85 , srv1391941322:85 , srv1467029568:85 , srv1616724587:85 , srv1645696652:85 , srv1655859764:85 , srv1658126879:85 , srv1735704838:85 , srv1810609920:85 , srv1885158354:85 , srv1906474525:85 , srv2110171147:85 , srv252628898:85 , srv2767234:85 , srv27743147:85 , srv303197018:85 , srv340467461:85 , srv3518731:85 , srv385691204:85 , srv387896377:85 , srv427274515:85 , srv428207384:85 , srv608561348:85 , srv618434094:85 , srv653845043:85 , srv672139375:85 , srv698182543:85 , srv735557786:85 , srv803637326:85 , srv864246902:85 , srv910197171:85 , srv933995368:85 , srv941002063:85 , srv983027318:85 , srv117085024:86 , srv117707041:86 , srv1826444074:86 , srv7057762:86 , srv797392709:86 , srv1485997514:87 , srv867476024:643 }
2023-10-23 23:56:03,681 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=235 (was 238), ProcessCount=168 (was 168), AvailableMemoryMB=3494 (was 3981)
2023-10-23 23:56:03,688 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=235, ProcessCount=168, AvailableMemoryMB=3494
2023-10-23 23:56:03,689 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-10-23 23:56:03,689 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-10-23 23:56:03,698 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1013291127:84 , srv1020420181:84 , srv10601773:84 , srv1090302928:84 , srv1137528935:84 , srv1211789842:84 , srv126951278:84 , srv1356069921:84 , srv1475130992:84 , srv1511798137:84 , srv1563380285:84 , srv1604502771:84 , srv1644895240:84 , srv1792930391:84 , srv1868297297:84 , srv1900004731:84 , srv1941305330:84 , srv1945156904:84 , srv2016690985:84 , srv2146866071:84 , srv355670806:84 , srv363870576:84 , srv429843177:84 , srv584251980:84 , srv685689860:84 , srv743870382:84 , srv802747750:84 , srv89966160:84 , srv974930402:84 , srv1531063217:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-10-23 23:56:03,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-23 23:56:03,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1090302928=3, srv1792930391=14, srv802747750=27, srv1531063217=10, srv685689860=25, srv2146866071=20, srv1137528935=4, srv1604502771=12, srv126951278=6, srv974930402=29, srv1563380285=11, srv363870576=22, srv1475130992=8, srv1644895240=13, srv89966160=28, srv743870382=26, srv1356069921=7, srv1511798137=9, srv355670806=21, srv1013291127=0, srv1020420181=1, srv10601773=2, srv1868297297=15, srv1941305330=17, srv1211789842=5, srv1900004731=16, srv429843177=23, srv584251980=24, srv2016690985=19, srv1945156904=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-23 23:56:03,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-23 23:56:03,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-23 23:56:03,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-23 23:56:03,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-23 23:56:03,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-23 23:56:03,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-23 23:56:03,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-23 23:56:03,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-23 23:56:03,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-23 23:56:03,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-23 23:56:03,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-23 23:56:03,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-23 23:56:03,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-23 23:56:03,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-23 23:56:03,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-23 23:56:03,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-23 23:56:03,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-23 23:56:03,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-23 23:56:03,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-23 23:56:03,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-23 23:56:03,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-23 23:56:03,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-23 23:56:03,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-23 23:56:03,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-23 23:56:03,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-23 23:56:03,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-23 23:56:03,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-23 23:56:03,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-23 23:56:03,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-23 23:56:03,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-23 23:56:03,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-10-23 23:56:03,722 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0677591750151139, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-23 23:56:03,722 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912406550212213, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0677591750151139, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-10-23 23:58:03,712 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9166671 different iterations.  Found a solution that moves 1840 regions; Going from a computed imbalance of 0.9912406550212213 to a new imbalance of 4.2946517914333035E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6814814814814815, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-23 23:58:03,718 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1013291127:90 , srv1020420181:90 , srv10601773:90 , srv1090302928:90 , srv1137528935:90 , srv1211789842:90 , srv126951278:90 , srv1356069921:90 , srv1475130992:90 , srv1511798137:90 , srv1531063217:90 , srv1563380285:90 , srv1604502771:90 , srv1644895240:90 , srv1792930391:90 , srv1868297297:90 , srv1900004731:90 , srv1941305330:90 , srv1945156904:90 , srv2016690985:90 , srv2146866071:90 , srv355670806:90 , srv363870576:90 , srv429843177:90 , srv584251980:90 , srv685689860:90 , srv743870382:90 , srv802747750:90 , srv89966160:90 , srv974930402:90 }
2023-10-23 23:58:03,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-23 23:58:03,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1090302928=3, srv1792930391=14, srv802747750=27, srv1531063217=10, srv685689860=25, srv2146866071=20, srv1137528935=4, srv1604502771=12, srv126951278=6, srv974930402=29, srv1563380285=11, srv363870576=22, srv1475130992=8, srv1644895240=13, srv89966160=28, srv743870382=26, srv1356069921=7, srv1511798137=9, srv355670806=21, srv1013291127=0, srv1020420181=1, srv10601773=2, srv1868297297=15, srv1941305330=17, srv1211789842=5, srv1900004731=16, srv429843177=23, srv584251980=24, srv2016690985=19, srv1945156904=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-23 23:58:03,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-23 23:58:03,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-23 23:58:03,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-23 23:58:03,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-23 23:58:03,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-23 23:58:03,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-23 23:58:03,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-23 23:58:03,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-23 23:58:03,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-23 23:58:03,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-23 23:58:03,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-23 23:58:03,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-23 23:58:03,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-23 23:58:03,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-23 23:58:03,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-23 23:58:03,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-23 23:58:03,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-23 23:58:03,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-23 23:58:03,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-23 23:58:03,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-23 23:58:03,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-23 23:58:03,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-23 23:58:03,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-23 23:58:03,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-23 23:58:03,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-23 23:58:03,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-23 23:58:03,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-23 23:58:03,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-23 23:58:03,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-23 23:58:03,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-23 23:58:03,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-10-23 23:58:03,736 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-23 23:58:03,736 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1013291127:90 , srv1020420181:90 , srv10601773:90 , srv1090302928:90 , srv1137528935:90 , srv1211789842:90 , srv126951278:90 , srv1356069921:90 , srv1475130992:90 , srv1511798137:90 , srv1531063217:90 , srv1563380285:90 , srv1604502771:90 , srv1644895240:90 , srv1792930391:90 , srv1868297297:90 , srv1900004731:90 , srv1941305330:90 , srv1945156904:90 , srv2016690985:90 , srv2146866071:90 , srv355670806:90 , srv363870576:90 , srv429843177:90 , srv584251980:90 , srv685689860:90 , srv743870382:90 , srv802747750:90 , srv89966160:90 , srv974930402:90 }
2023-10-23 23:58:03,748 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=168 (was 235), ProcessCount=168 (was 168), AvailableMemoryMB=3507 (was 3494) - AvailableMemoryMB LEAK? -