2023-07-30 16:53:12,457 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-07-30 16:53:12,780 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-07-30 16:53:12,900 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-07-30 16:53:12,968 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-07-30 16:53:12,978 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-07-30 16:53:12,992 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=275, ProcessCount=173, AvailableMemoryMB=2890
2023-07-30 16:53:12,994 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-07-30 16:53:12,994 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-07-30 16:53:13,163 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1008895204:84 , srv1019053694:84 , srv102192185:84 , srv102913390:84 , srv1080133864:84 , srv1100903501:84 , srv1112350213:84 , srv1113062668:84 , srv112738356:84 , srv1130886181:84 , srv1136507330:84 , srv1137336076:84 , srv1140616541:84 , srv1155693490:84 , srv1166197341:84 , srv117914491:84 , srv1183764919:84 , srv1200579615:84 , srv1254060451:84 , srv1289029668:84 , srv1320685531:84 , srv1328307575:84 , srv13284827:84 , srv133331943:84 , srv1345185583:84 , srv134585485:84 , srv1353240125:84 , srv1373742572:84 , srv1418422847:84 , srv1442059470:84 , srv1487945896:84 , srv1541852654:84 , srv1556374317:84 , srv156896033:84 , srv1601491661:84 , srv1602663466:84 , srv1647590235:84 , srv1649335164:84 , srv1667716857:84 , srv1698370669:84 , srv1719571749:84 , srv1727965906:84 , srv1734018463:84 , srv1737248573:84 , srv1763262018:84 , srv1780125042:84 , srv1782086507:84 , srv1810343988:84 , srv1865181363:84 , srv1867723716:84 , srv1926898339:84 , srv1927915078:84 , srv1942682335:84 , srv1987767468:84 , srv2002686151:84 , srv201008269:84 , srv201347765:84 , srv2038791120:84 , srv204688969:84 , srv2079230457:84 , srv2081489920:84 , srv208574880:84 , srv2090438834:84 , srv2098918922:84 , srv25126411:84 , srv251703916:84 , srv253377758:84 , srv282791248:84 , srv284516860:84 , srv299358088:84 , srv302602143:84 , srv30372976:84 , srv338498173:84 , srv348333667:84 , srv367101846:84 , srv421444469:84 , srv435498071:84 , srv451471651:84 , srv490883946:84 , srv502784084:84 , srv511859002:84 , srv560191124:84 , srv575353408:84 , srv597404246:84 , srv623969841:84 , srv659904085:84 , srv697165304:84 , srv697216476:84 , srv712943198:84 , srv730746388:84 , srv758931283:84 , srv777682413:84 , srv785150240:84 , srv830230026:84 , srv864561168:84 , srv871220192:84 , srv936513760:84 , srv943704762:84 , srv968231363:84 , srv1233212638:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-07-30 16:53:13,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-30 16:53:13,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv659904085=86, srv597404246=84, srv623969841=85, srv338498173=73, srv1649335164=38, srv1328307575=22, srv1345185583=25, srv1112350213=6, srv758931283=91, srv490883946=79, srv1289029668=20, srv2090438834=63, srv435498071=77, srv1942682335=53, srv251703916=66, srv1602663466=36, srv1100903501=5, srv1200579615=17, srv697216476=88, srv1155693490=13, srv1734018463=43, srv1719571749=41, srv2002686151=55, srv208574880=62, srv117914491=15, srv1780125042=46, srv1183764919=16, srv204688969=59, srv1019053694=1, srv348333667=74, srv1867723716=50, srv201008269=56, srv2079230457=60, srv968231363=99, srv864561168=95, srv1320685531=21, srv1737248573=44, srv2081489920=61, srv112738356=8, srv1601491661=35, srv302602143=71, srv1080133864=4, srv712943198=89, srv1667716857=39, srv1763262018=45, srv1353240125=27, srv1418422847=29, srv1698370669=40, srv134585485=26, srv2098918922=64, srv1254060451=19, srv1556374317=33, srv30372976=72, srv830230026=94, srv511859002=81, srv1113062668=7, srv1541852654=32, srv1233212638=18, srv156896033=34, srv2038791120=58, srv871220192=96, srv1987767468=54, srv1782086507=47, srv1927915078=52, srv1647590235=37, srv502784084=80, srv936513760=97, srv730746388=90, srv253377758=67, srv1130886181=9, srv1810343988=48, srv284516860=69, srv697165304=87, srv575353408=83, srv133331943=24, srv1865181363=49, srv1926898339=51, srv785150240=93, srv1727965906=42, srv560191124=82, srv1373742572=28, srv451471651=78, srv367101846=75, srv943704762=98, srv299358088=70, srv13284827=23, srv102192185=2, srv25126411=65, srv1008895204=0, srv1140616541=12, srv1136507330=10, srv1442059470=30, srv1166197341=14, srv282791248=68, srv201347765=57, srv777682413=92, srv102913390=3, srv1487945896=31, srv421444469=76, srv1137336076=11} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-30 16:53:13,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-30 16:53:13,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-30 16:53:13,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-30 16:53:13,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-30 16:53:13,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-30 16:53:13,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-30 16:53:13,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-30 16:53:13,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-30 16:53:13,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-30 16:53:13,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-30 16:53:13,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-30 16:53:13,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-30 16:53:13,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-30 16:53:13,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-30 16:53:13,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-30 16:53:13,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-30 16:53:13,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-30 16:53:13,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-30 16:53:13,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-30 16:53:13,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-30 16:53:13,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-30 16:53:13,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-30 16:53:13,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-30 16:53:13,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-30 16:53:13,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-30 16:53:13,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-30 16:53:13,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-30 16:53:13,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-30 16:53:13,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-30 16:53:13,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-30 16:53:13,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-30 16:53:13,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-30 16:53:13,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-30 16:53:13,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-30 16:53:13,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-30 16:53:13,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-30 16:53:13,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-30 16:53:13,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-30 16:53:13,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-30 16:53:13,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-30 16:53:13,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-30 16:53:13,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-30 16:53:13,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-30 16:53:13,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-30 16:53:13,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-30 16:53:13,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-30 16:53:13,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-30 16:53:13,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-30 16:53:13,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-30 16:53:13,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-30 16:53:13,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-30 16:53:13,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-30 16:53:13,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-30 16:53:13,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-30 16:53:13,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-30 16:53:13,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-30 16:53:13,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-30 16:53:13,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-30 16:53:13,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-30 16:53:13,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-30 16:53:13,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-30 16:53:13,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-30 16:53:13,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-30 16:53:13,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-30 16:53:13,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-30 16:53:13,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-30 16:53:13,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-30 16:53:13,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-30 16:53:13,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-30 16:53:13,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-30 16:53:13,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-30 16:53:13,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-30 16:53:13,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-30 16:53:13,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-30 16:53:13,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-30 16:53:13,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-30 16:53:13,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-30 16:53:13,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-30 16:53:13,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-30 16:53:13,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-30 16:53:13,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-30 16:53:13,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-30 16:53:13,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-30 16:53:13,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-30 16:53:13,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-30 16:53:13,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-30 16:53:13,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-30 16:53:13,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-30 16:53:13,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-30 16:53:13,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-30 16:53:13,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-30 16:53:13,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-30 16:53:13,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-30 16:53:13,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-30 16:53:13,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-30 16:53:13,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-30 16:53:13,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-30 16:53:13,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-30 16:53:13,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-30 16:53:13,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-30 16:53:13,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-30 16:53:13,556 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-30 16:53:13,557 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-07-30 16:53:13,557 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-07-30 16:53:13,926 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 625 ms to try 5000 different iterations.  Found a solution that moves 4700 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17884782163628252. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.044077610700422355, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05561814655947787, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5222222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.044077610700422355, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17425, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23858333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-30 16:53:13,938 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1140616541:82 , srv13284827:82 , srv134585485:82 , srv1782086507:82 , srv2098918922:82 , srv490883946:82 , srv864561168:82 , srv871220192:82 , srv102913390:83 , srv1442059470:83 , srv1763262018:83 , srv25126411:83 , srv284516860:83 , srv712943198:83 , srv1137336076:84 , srv1155693490:84 , srv1320685531:84 , srv1647590235:84 , srv1727965906:84 , srv1734018463:84 , srv1942682335:84 , srv2038791120:84 , srv204688969:84 , srv348333667:84 , srv367101846:84 , srv1080133864:85 , srv1113062668:85 , srv1130886181:85 , srv1166197341:85 , srv1254060451:85 , srv1418422847:85 , srv1487945896:85 , srv1602663466:85 , srv1649335164:85 , srv1810343988:85 , srv253377758:85 , srv282791248:85 , srv302602143:85 , srv575353408:85 , srv597404246:85 , srv777682413:85 , srv830230026:85 , srv1183764919:86 , srv1289029668:86 , srv1373742572:86 , srv1541852654:86 , srv1719571749:86 , srv1737248573:86 , srv1867723716:86 , srv1987767468:86 , srv201347765:86 , srv2081489920:86 , srv299358088:86 , srv338498173:86 , srv435498071:86 , srv451471651:86 , srv511859002:86 , srv659904085:86 , srv697165304:86 , srv697216476:86 , srv758931283:86 , srv1008895204:87 , srv1019053694:87 , srv117914491:87 , srv1353240125:87 , srv1601491661:87 , srv1667716857:87 , srv1865181363:87 , srv1927915078:87 , srv201008269:87 , srv2079230457:87 , srv30372976:87 , srv943704762:87 , srv1100903501:88 , srv1136507330:88 , srv1200579615:88 , srv1328307575:88 , srv1556374317:88 , srv156896033:88 , srv1698370669:88 , srv1780125042:88 , srv208574880:88 , srv560191124:88 , srv623969841:88 , srv785150240:88 , srv133331943:89 , srv1345185583:89 , srv1926898339:89 , srv2002686151:89 , srv2090438834:89 , srv730746388:89 , srv968231363:89 , srv102192185:90 , srv112738356:90 , srv502784084:90 , srv936513760:90 , srv1112350213:91 , srv421444469:91 , srv251703916:96 , srv1233212638:482 }
2023-07-30 16:53:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-30 16:53:13,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv659904085=86, srv597404246=84, srv623969841=85, srv338498173=73, srv1649335164=38, srv1328307575=22, srv1345185583=25, srv1112350213=6, srv758931283=91, srv490883946=79, srv1289029668=20, srv2090438834=63, srv435498071=77, srv1942682335=53, srv251703916=66, srv1602663466=36, srv1100903501=5, srv1200579615=17, srv697216476=88, srv1155693490=13, srv1734018463=43, srv1719571749=41, srv2002686151=55, srv208574880=62, srv117914491=15, srv1780125042=46, srv1183764919=16, srv204688969=59, srv1019053694=1, srv348333667=74, srv1867723716=50, srv201008269=56, srv2079230457=60, srv968231363=99, srv864561168=95, srv1320685531=21, srv1737248573=44, srv2081489920=61, srv112738356=8, srv1601491661=35, srv302602143=71, srv1080133864=4, srv712943198=89, srv1667716857=39, srv1763262018=45, srv1353240125=27, srv1418422847=29, srv1698370669=40, srv134585485=26, srv2098918922=64, srv1254060451=19, srv1556374317=33, srv30372976=72, srv830230026=94, srv511859002=81, srv1113062668=7, srv1541852654=32, srv1233212638=18, srv156896033=34, srv2038791120=58, srv871220192=96, srv1987767468=54, srv1782086507=47, srv1927915078=52, srv1647590235=37, srv502784084=80, srv936513760=97, srv730746388=90, srv253377758=67, srv1130886181=9, srv1810343988=48, srv284516860=69, srv697165304=87, srv575353408=83, srv133331943=24, srv1865181363=49, srv1926898339=51, srv785150240=93, srv1727965906=42, srv560191124=82, srv1373742572=28, srv451471651=78, srv367101846=75, srv943704762=98, srv299358088=70, srv13284827=23, srv102192185=2, srv25126411=65, srv1008895204=0, srv1140616541=12, srv1136507330=10, srv1442059470=30, srv1166197341=14, srv282791248=68, srv201347765=57, srv777682413=92, srv102913390=3, srv1487945896=31, srv421444469=76, srv1137336076=11} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-30 16:53:13,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-30 16:53:13,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-30 16:53:13,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-30 16:53:13,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-30 16:53:13,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-30 16:53:13,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-30 16:53:13,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-30 16:53:13,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-30 16:53:13,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-30 16:53:13,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-30 16:53:13,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-30 16:53:13,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-30 16:53:13,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-30 16:53:13,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-30 16:53:13,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-30 16:53:13,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-30 16:53:13,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-30 16:53:13,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-30 16:53:13,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-30 16:53:13,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-30 16:53:13,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-30 16:53:13,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-30 16:53:13,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-30 16:53:13,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-30 16:53:13,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-30 16:53:13,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-30 16:53:13,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-30 16:53:13,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-30 16:53:13,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-30 16:53:13,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-30 16:53:13,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-30 16:53:13,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-30 16:53:13,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-30 16:53:13,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-30 16:53:13,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-30 16:53:13,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-30 16:53:13,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-30 16:53:13,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-30 16:53:13,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-30 16:53:13,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-30 16:53:13,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-30 16:53:13,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-30 16:53:13,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-30 16:53:13,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-30 16:53:13,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-30 16:53:13,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-30 16:53:13,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-30 16:53:13,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-30 16:53:13,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-30 16:53:13,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-30 16:53:13,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-30 16:53:13,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-30 16:53:13,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-30 16:53:13,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-30 16:53:13,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-30 16:53:13,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-30 16:53:13,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-30 16:53:13,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-30 16:53:13,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-30 16:53:13,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-30 16:53:13,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-30 16:53:13,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-30 16:53:13,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-30 16:53:13,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-30 16:53:13,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-30 16:53:13,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-30 16:53:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-30 16:53:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-30 16:53:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-30 16:53:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-30 16:53:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-30 16:53:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-30 16:53:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-30 16:53:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-30 16:53:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-30 16:53:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-30 16:53:13,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-30 16:53:13,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-30 16:53:13,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-30 16:53:13,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-30 16:53:13,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-30 16:53:13,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-30 16:53:13,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-30 16:53:13,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-30 16:53:13,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-30 16:53:13,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-30 16:53:13,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-30 16:53:13,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-30 16:53:13,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-30 16:53:13,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-30 16:53:13,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-30 16:53:13,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-30 16:53:13,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-30 16:53:13,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-30 16:53:13,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-30 16:53:13,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-30 16:53:13,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-30 16:53:13,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-30 16:53:13,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-30 16:53:13,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-30 16:53:13,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-30 16:53:14,131 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.044077610700422355, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05561814655947787, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.044077610700422355, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17425, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23858333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-30 16:53:14,131 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-07-30 16:53:14,132 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17881491153288076, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.044077610700422355, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05561814655947787, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.044077610700422355, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17425, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23858333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-07-30 16:53:14,496 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 503 ms to try 5000 different iterations.  Found a solution that moves 2600 regions; Going from a computed imbalance of 0.17881491153288076 to a new imbalance of 0.01381112603277424. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021375566127829166); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032947259150373734, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28888888888888886, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021375566127829166); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042083333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-30 16:53:14,512 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv251703916:77 , srv2002686151:81 , srv421444469:81 , srv502784084:81 , srv1100903501:82 , srv1112350213:82 , srv112738356:82 , srv1780125042:82 , srv785150240:82 , srv102192185:83 , srv117914491:83 , srv1556374317:83 , srv1926898339:83 , srv968231363:83 , srv1019053694:84 , srv1136507330:84 , srv133331943:84 , srv1345185583:84 , srv1353240125:84 , srv156896033:84 , srv1927915078:84 , srv201008269:84 , srv299358088:84 , srv30372976:84 , srv730746388:84 , srv936513760:84 , srv1008895204:85 , srv1166197341:85 , srv1183764919:85 , srv1200579615:85 , srv1328307575:85 , srv1601491661:85 , srv1734018463:85 , srv1867723716:85 , srv1987767468:85 , srv2079230457:85 , srv208574880:85 , srv2090438834:85 , srv560191124:85 , srv623969841:85 , srv943704762:85 , srv1080133864:86 , srv1155693490:86 , srv1289029668:86 , srv1320685531:86 , srv1373742572:86 , srv1487945896:86 , srv1649335164:86 , srv1667716857:86 , srv1727965906:86 , srv2038791120:86 , srv435498071:86 , srv511859002:86 , srv697165304:86 , srv758931283:86 , srv830230026:86 , srv1418422847:87 , srv1541852654:87 , srv1698370669:87 , srv1719571749:87 , srv1737248573:87 , srv1865181363:87 , srv201347765:87 , srv2081489920:87 , srv253377758:87 , srv284516860:87 , srv302602143:87 , srv348333667:87 , srv597404246:87 , srv697216476:87 , srv1113062668:88 , srv1130886181:88 , srv1137336076:88 , srv1254060451:88 , srv1647590235:88 , srv1810343988:88 , srv1942682335:88 , srv338498173:88 , srv451471651:88 , srv490883946:88 , srv659904085:88 , srv102913390:89 , srv13284827:89 , srv1442059470:89 , srv1763262018:89 , srv204688969:89 , srv2098918922:89 , srv282791248:89 , srv712943198:89 , srv777682413:89 , srv871220192:89 , srv1602663466:90 , srv1782086507:90 , srv25126411:90 , srv575353408:90 , srv864561168:90 , srv1140616541:91 , srv367101846:91 , srv134585485:92 , srv1233212638:482 }
2023-07-30 16:53:14,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-30 16:53:14,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv659904085=86, srv597404246=84, srv623969841=85, srv338498173=73, srv1649335164=38, srv1328307575=22, srv1345185583=25, srv1112350213=6, srv758931283=91, srv490883946=79, srv1289029668=20, srv2090438834=63, srv435498071=77, srv1942682335=53, srv251703916=66, srv1602663466=36, srv1100903501=5, srv1200579615=17, srv697216476=88, srv1155693490=13, srv1734018463=43, srv1719571749=41, srv2002686151=55, srv208574880=62, srv117914491=15, srv1780125042=46, srv1183764919=16, srv204688969=59, srv1019053694=1, srv348333667=74, srv1867723716=50, srv201008269=56, srv2079230457=60, srv968231363=99, srv864561168=95, srv1320685531=21, srv1737248573=44, srv2081489920=61, srv112738356=8, srv1601491661=35, srv302602143=71, srv1080133864=4, srv712943198=89, srv1667716857=39, srv1763262018=45, srv1353240125=27, srv1418422847=29, srv1698370669=40, srv134585485=26, srv2098918922=64, srv1254060451=19, srv1556374317=33, srv30372976=72, srv830230026=94, srv511859002=81, srv1113062668=7, srv1541852654=32, srv1233212638=18, srv156896033=34, srv2038791120=58, srv871220192=96, srv1987767468=54, srv1782086507=47, srv1927915078=52, srv1647590235=37, srv502784084=80, srv936513760=97, srv730746388=90, srv253377758=67, srv1130886181=9, srv1810343988=48, srv284516860=69, srv697165304=87, srv575353408=83, srv133331943=24, srv1865181363=49, srv1926898339=51, srv785150240=93, srv1727965906=42, srv560191124=82, srv1373742572=28, srv451471651=78, srv367101846=75, srv943704762=98, srv299358088=70, srv13284827=23, srv102192185=2, srv25126411=65, srv1008895204=0, srv1140616541=12, srv1136507330=10, srv1442059470=30, srv1166197341=14, srv282791248=68, srv201347765=57, srv777682413=92, srv102913390=3, srv1487945896=31, srv421444469=76, srv1137336076=11} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-30 16:53:14,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-30 16:53:14,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-30 16:53:14,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-30 16:53:14,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-30 16:53:14,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-30 16:53:14,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-30 16:53:14,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-30 16:53:14,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-30 16:53:14,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-30 16:53:14,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-30 16:53:14,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-30 16:53:14,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-30 16:53:14,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-30 16:53:14,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-30 16:53:14,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-30 16:53:14,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-30 16:53:14,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-30 16:53:14,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-30 16:53:14,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-30 16:53:14,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-30 16:53:14,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-30 16:53:14,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-30 16:53:14,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-30 16:53:14,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-30 16:53:14,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-30 16:53:14,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-30 16:53:14,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-30 16:53:14,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-30 16:53:14,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-30 16:53:14,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-30 16:53:14,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-30 16:53:14,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-30 16:53:14,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-30 16:53:14,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-30 16:53:14,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-30 16:53:14,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-30 16:53:14,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-30 16:53:14,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-30 16:53:14,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-30 16:53:14,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-30 16:53:14,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-30 16:53:14,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-30 16:53:14,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-30 16:53:14,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-30 16:53:14,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-30 16:53:14,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-30 16:53:14,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-30 16:53:14,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-30 16:53:14,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-30 16:53:14,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-30 16:53:14,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-30 16:53:14,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-30 16:53:14,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-30 16:53:14,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-30 16:53:14,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-30 16:53:14,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-30 16:53:14,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-30 16:53:14,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-30 16:53:14,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-30 16:53:14,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-30 16:53:14,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-30 16:53:14,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-30 16:53:14,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-30 16:53:14,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-30 16:53:14,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-30 16:53:14,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-30 16:53:14,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-30 16:53:14,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-30 16:53:14,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-30 16:53:14,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-30 16:53:14,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-30 16:53:14,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-30 16:53:14,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-30 16:53:14,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-30 16:53:14,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-30 16:53:14,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-30 16:53:14,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-30 16:53:14,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-30 16:53:14,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-30 16:53:14,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-30 16:53:14,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-30 16:53:14,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-30 16:53:14,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-30 16:53:14,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-30 16:53:14,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-30 16:53:14,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-30 16:53:14,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-30 16:53:14,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-30 16:53:14,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-30 16:53:14,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-30 16:53:14,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-30 16:53:14,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-30 16:53:14,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-30 16:53:14,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-30 16:53:14,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-30 16:53:14,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-30 16:53:14,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-30 16:53:14,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-30 16:53:14,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-30 16:53:14,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-30 16:53:14,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-30 16:53:14,702 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021375566127829166); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032947259150373734, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021375566127829166); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042083333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-30 16:53:14,702 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-07-30 16:53:14,702 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013792920443658381, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021375566127829166); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032947259150373734, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021375566127829166); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042083333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-07-30 16:53:15,003 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 433 ms to try 5000 different iterations.  Found a solution that moves 937 regions; Going from a computed imbalance of 0.013792920443658381 to a new imbalance of 4.850020702987882E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006653559880595792); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010834951964030606); MoveCostFunction : (multiplier=7.0, imbalance=0.10411111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006653559880595792); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-30 16:53:15,005 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv134585485:83 , srv1602663466:83 , srv1698370669:83 , srv1865181363:83 , srv2090438834:83 , srv367101846:83 , srv451471651:83 , srv575353408:83 , srv659904085:83 , srv102192185:84 , srv1112350213:84 , srv1130886181:84 , srv1328307575:84 , srv133331943:84 , srv1345185583:84 , srv1719571749:84 , srv1810343988:84 , srv201347765:84 , srv204688969:84 , srv2081489920:84 , srv208574880:84 , srv25126411:84 , srv251703916:84 , srv282791248:84 , srv338498173:84 , srv560191124:84 , srv712943198:84 , srv730746388:84 , srv777682413:84 , srv1008895204:85 , srv102913390:85 , srv1113062668:85 , srv112738356:85 , srv1136507330:85 , srv1137336076:85 , srv1140616541:85 , srv1200579615:85 , srv1254060451:85 , srv1289029668:85 , srv1442059470:85 , srv1541852654:85 , srv156896033:85 , srv1647590235:85 , srv1737248573:85 , srv1763262018:85 , srv1782086507:85 , srv1942682335:85 , srv253377758:85 , srv435498071:85 , srv597404246:85 , srv697216476:85 , srv864561168:85 , srv936513760:85 , srv943704762:85 , srv1019053694:86 , srv1080133864:86 , srv1183764919:86 , srv13284827:86 , srv1353240125:86 , srv1373742572:86 , srv1418422847:86 , srv1487945896:86 , srv1601491661:86 , srv1649335164:86 , srv1667716857:86 , srv1867723716:86 , srv1926898339:86 , srv1927915078:86 , srv1987767468:86 , srv201008269:86 , srv2079230457:86 , srv2098918922:86 , srv302602143:86 , srv30372976:86 , srv421444469:86 , srv502784084:86 , srv511859002:86 , srv623969841:86 , srv697165304:86 , srv758931283:86 , srv968231363:86 , srv1155693490:87 , srv117914491:87 , srv1320685531:87 , srv1556374317:87 , srv1727965906:87 , srv1780125042:87 , srv2002686151:87 , srv2038791120:87 , srv299358088:87 , srv830230026:87 , srv871220192:87 , srv1100903501:88 , srv1166197341:88 , srv1734018463:88 , srv284516860:88 , srv348333667:88 , srv490883946:88 , srv785150240:88 , srv1233212638:553 }
2023-07-30 16:53:15,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-30 16:53:15,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv659904085=86, srv597404246=84, srv623969841=85, srv338498173=73, srv1649335164=38, srv1328307575=22, srv1345185583=25, srv1112350213=6, srv758931283=91, srv490883946=79, srv1289029668=20, srv2090438834=63, srv435498071=77, srv1942682335=53, srv251703916=66, srv1602663466=36, srv1100903501=5, srv1200579615=17, srv697216476=88, srv1155693490=13, srv1734018463=43, srv1719571749=41, srv2002686151=55, srv208574880=62, srv117914491=15, srv1780125042=46, srv1183764919=16, srv204688969=59, srv1019053694=1, srv348333667=74, srv1867723716=50, srv201008269=56, srv2079230457=60, srv968231363=99, srv864561168=95, srv1320685531=21, srv1737248573=44, srv2081489920=61, srv112738356=8, srv1601491661=35, srv302602143=71, srv1080133864=4, srv712943198=89, srv1667716857=39, srv1763262018=45, srv1353240125=27, srv1418422847=29, srv1698370669=40, srv134585485=26, srv2098918922=64, srv1254060451=19, srv1556374317=33, srv30372976=72, srv830230026=94, srv511859002=81, srv1113062668=7, srv1541852654=32, srv1233212638=18, srv156896033=34, srv2038791120=58, srv871220192=96, srv1987767468=54, srv1782086507=47, srv1927915078=52, srv1647590235=37, srv502784084=80, srv936513760=97, srv730746388=90, srv253377758=67, srv1130886181=9, srv1810343988=48, srv284516860=69, srv697165304=87, srv575353408=83, srv133331943=24, srv1865181363=49, srv1926898339=51, srv785150240=93, srv1727965906=42, srv560191124=82, srv1373742572=28, srv451471651=78, srv367101846=75, srv943704762=98, srv299358088=70, srv13284827=23, srv102192185=2, srv25126411=65, srv1008895204=0, srv1140616541=12, srv1136507330=10, srv1442059470=30, srv1166197341=14, srv282791248=68, srv201347765=57, srv777682413=92, srv102913390=3, srv1487945896=31, srv421444469=76, srv1137336076=11} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-30 16:53:15,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-30 16:53:15,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-30 16:53:15,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-30 16:53:15,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-30 16:53:15,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-30 16:53:15,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-30 16:53:15,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-30 16:53:15,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-30 16:53:15,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-30 16:53:15,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-30 16:53:15,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-30 16:53:15,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-30 16:53:15,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-30 16:53:15,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-30 16:53:15,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-30 16:53:15,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-30 16:53:15,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-30 16:53:15,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-30 16:53:15,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-30 16:53:15,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-30 16:53:15,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-30 16:53:15,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-30 16:53:15,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-30 16:53:15,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-30 16:53:15,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-30 16:53:15,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-30 16:53:15,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-30 16:53:15,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-30 16:53:15,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-30 16:53:15,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-30 16:53:15,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-30 16:53:15,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-30 16:53:15,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-30 16:53:15,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-30 16:53:15,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-30 16:53:15,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-30 16:53:15,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-30 16:53:15,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-30 16:53:15,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-30 16:53:15,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-30 16:53:15,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-30 16:53:15,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-30 16:53:15,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-30 16:53:15,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-30 16:53:15,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-30 16:53:15,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-30 16:53:15,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-30 16:53:15,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-30 16:53:15,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-30 16:53:15,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-30 16:53:15,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-30 16:53:15,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-30 16:53:15,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-30 16:53:15,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-30 16:53:15,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-30 16:53:15,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-30 16:53:15,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-30 16:53:15,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-30 16:53:15,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-30 16:53:15,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-30 16:53:15,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-30 16:53:15,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-30 16:53:15,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-30 16:53:15,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-30 16:53:15,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-30 16:53:15,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-30 16:53:15,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-30 16:53:15,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-30 16:53:15,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-30 16:53:15,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-30 16:53:15,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-30 16:53:15,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-30 16:53:15,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-30 16:53:15,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-30 16:53:15,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-30 16:53:15,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-30 16:53:15,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-30 16:53:15,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-30 16:53:15,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-30 16:53:15,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-30 16:53:15,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-30 16:53:15,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-30 16:53:15,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-30 16:53:15,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-30 16:53:15,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-30 16:53:15,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-30 16:53:15,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-30 16:53:15,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-30 16:53:15,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-30 16:53:15,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-30 16:53:15,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-30 16:53:15,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-30 16:53:15,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-30 16:53:15,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-30 16:53:15,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-30 16:53:15,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-30 16:53:15,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-30 16:53:15,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-30 16:53:15,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-30 16:53:15,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-30 16:53:15,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-30 16:53:15,185 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006653559880595792); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010834951964030606); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006653559880595792); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-30 16:53:15,185 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-07-30 16:53:15,185 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.784410560674192E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006653559880595792); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010834951964030606); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006653559880595792); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-07-30 16:53:15,549 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 498 ms to try 5000 different iterations.  Found a solution that moves 243 regions; Going from a computed imbalance of 4.784410560674192E-4 to a new imbalance of 2.0016421360587148E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.027, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-30 16:53:15,550 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1166197341:83 , srv1418422847:83 , srv1667716857:83 , srv348333667:83 , srv623969841:83 , srv936513760:83 , srv102192185:84 , srv1100903501:84 , srv1113062668:84 , srv1140616541:84 , srv117914491:84 , srv1200579615:84 , srv1373742572:84 , srv1487945896:84 , srv1556374317:84 , srv156896033:84 , srv1602663466:84 , srv1647590235:84 , srv1698370669:84 , srv1737248573:84 , srv1780125042:84 , srv1926898339:84 , srv2079230457:84 , srv282791248:84 , srv302602143:84 , srv338498173:84 , srv367101846:84 , srv421444469:84 , srv435498071:84 , srv490883946:84 , srv575353408:84 , srv697165304:84 , srv697216476:84 , srv758931283:84 , srv777682413:84 , srv785150240:84 , srv830230026:84 , srv871220192:84 , srv968231363:84 , srv1008895204:85 , srv1019053694:85 , srv102913390:85 , srv1080133864:85 , srv1112350213:85 , srv112738356:85 , srv1136507330:85 , srv1137336076:85 , srv1155693490:85 , srv1183764919:85 , srv1254060451:85 , srv1289029668:85 , srv1320685531:85 , srv1328307575:85 , srv13284827:85 , srv133331943:85 , srv1345185583:85 , srv134585485:85 , srv1353240125:85 , srv1442059470:85 , srv1541852654:85 , srv1601491661:85 , srv1649335164:85 , srv1719571749:85 , srv1727965906:85 , srv1734018463:85 , srv1763262018:85 , srv1782086507:85 , srv1810343988:85 , srv1865181363:85 , srv1867723716:85 , srv1927915078:85 , srv1942682335:85 , srv1987767468:85 , srv2002686151:85 , srv201347765:85 , srv2038791120:85 , srv204688969:85 , srv2081489920:85 , srv208574880:85 , srv2090438834:85 , srv2098918922:85 , srv25126411:85 , srv251703916:85 , srv253377758:85 , srv284516860:85 , srv299358088:85 , srv30372976:85 , srv451471651:85 , srv511859002:85 , srv560191124:85 , srv597404246:85 , srv659904085:85 , srv730746388:85 , srv864561168:85 , srv943704762:85 , srv1130886181:86 , srv201008269:86 , srv502784084:86 , srv712943198:86 , srv1233212638:626 }
2023-07-30 16:53:15,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-30 16:53:15,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv659904085=86, srv597404246=84, srv623969841=85, srv338498173=73, srv1649335164=38, srv1328307575=22, srv1345185583=25, srv1112350213=6, srv758931283=91, srv490883946=79, srv1289029668=20, srv2090438834=63, srv435498071=77, srv1942682335=53, srv251703916=66, srv1602663466=36, srv1100903501=5, srv1200579615=17, srv697216476=88, srv1155693490=13, srv1734018463=43, srv1719571749=41, srv2002686151=55, srv208574880=62, srv117914491=15, srv1780125042=46, srv1183764919=16, srv204688969=59, srv1019053694=1, srv348333667=74, srv1867723716=50, srv201008269=56, srv2079230457=60, srv968231363=99, srv864561168=95, srv1320685531=21, srv1737248573=44, srv2081489920=61, srv112738356=8, srv1601491661=35, srv302602143=71, srv1080133864=4, srv712943198=89, srv1667716857=39, srv1763262018=45, srv1353240125=27, srv1418422847=29, srv1698370669=40, srv134585485=26, srv2098918922=64, srv1254060451=19, srv1556374317=33, srv30372976=72, srv830230026=94, srv511859002=81, srv1113062668=7, srv1541852654=32, srv1233212638=18, srv156896033=34, srv2038791120=58, srv871220192=96, srv1987767468=54, srv1782086507=47, srv1927915078=52, srv1647590235=37, srv502784084=80, srv936513760=97, srv730746388=90, srv253377758=67, srv1130886181=9, srv1810343988=48, srv284516860=69, srv697165304=87, srv575353408=83, srv133331943=24, srv1865181363=49, srv1926898339=51, srv785150240=93, srv1727965906=42, srv560191124=82, srv1373742572=28, srv451471651=78, srv367101846=75, srv943704762=98, srv299358088=70, srv13284827=23, srv102192185=2, srv25126411=65, srv1008895204=0, srv1140616541=12, srv1136507330=10, srv1442059470=30, srv1166197341=14, srv282791248=68, srv201347765=57, srv777682413=92, srv102913390=3, srv1487945896=31, srv421444469=76, srv1137336076=11} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-30 16:53:15,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-30 16:53:15,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-30 16:53:15,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-30 16:53:15,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-30 16:53:15,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-30 16:53:15,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-30 16:53:15,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-30 16:53:15,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-30 16:53:15,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-30 16:53:15,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-30 16:53:15,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-30 16:53:15,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-30 16:53:15,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-30 16:53:15,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-30 16:53:15,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-30 16:53:15,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-30 16:53:15,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-30 16:53:15,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-30 16:53:15,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-30 16:53:15,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-30 16:53:15,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-30 16:53:15,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-30 16:53:15,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-30 16:53:15,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-30 16:53:15,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-30 16:53:15,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-30 16:53:15,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-30 16:53:15,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-30 16:53:15,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-30 16:53:15,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-30 16:53:15,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-30 16:53:15,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-30 16:53:15,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-30 16:53:15,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-30 16:53:15,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-30 16:53:15,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-30 16:53:15,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-30 16:53:15,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-30 16:53:15,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-30 16:53:15,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-30 16:53:15,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-30 16:53:15,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-30 16:53:15,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-30 16:53:15,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-30 16:53:15,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-30 16:53:15,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-30 16:53:15,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-30 16:53:15,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-30 16:53:15,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-30 16:53:15,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-30 16:53:15,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-30 16:53:15,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-30 16:53:15,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-30 16:53:15,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-30 16:53:15,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-30 16:53:15,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-30 16:53:15,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-30 16:53:15,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-30 16:53:15,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-30 16:53:15,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-30 16:53:15,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-30 16:53:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-30 16:53:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-30 16:53:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-30 16:53:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-30 16:53:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-30 16:53:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-30 16:53:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-30 16:53:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-30 16:53:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-30 16:53:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-30 16:53:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-30 16:53:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-30 16:53:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-30 16:53:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-30 16:53:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-30 16:53:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-30 16:53:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-30 16:53:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-30 16:53:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-30 16:53:15,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-30 16:53:15,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-30 16:53:15,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-30 16:53:15,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-30 16:53:15,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-30 16:53:15,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-30 16:53:15,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-30 16:53:15,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-30 16:53:15,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-30 16:53:15,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-30 16:53:15,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-30 16:53:15,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-30 16:53:15,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-30 16:53:15,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-30 16:53:15,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-30 16:53:15,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-30 16:53:15,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-30 16:53:15,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-30 16:53:15,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-30 16:53:15,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-30 16:53:15,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-30 16:53:15,729 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.8314898993220366E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-30 16:53:15,730 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1166197341:83 , srv1418422847:83 , srv1667716857:83 , srv348333667:83 , srv623969841:83 , srv936513760:83 , srv102192185:84 , srv1100903501:84 , srv1113062668:84 , srv1140616541:84 , srv117914491:84 , srv1200579615:84 , srv1373742572:84 , srv1487945896:84 , srv1556374317:84 , srv156896033:84 , srv1602663466:84 , srv1647590235:84 , srv1698370669:84 , srv1737248573:84 , srv1780125042:84 , srv1926898339:84 , srv2079230457:84 , srv282791248:84 , srv302602143:84 , srv338498173:84 , srv367101846:84 , srv421444469:84 , srv435498071:84 , srv490883946:84 , srv575353408:84 , srv697165304:84 , srv697216476:84 , srv758931283:84 , srv777682413:84 , srv785150240:84 , srv830230026:84 , srv871220192:84 , srv968231363:84 , srv1008895204:85 , srv1019053694:85 , srv102913390:85 , srv1080133864:85 , srv1112350213:85 , srv112738356:85 , srv1136507330:85 , srv1137336076:85 , srv1155693490:85 , srv1183764919:85 , srv1254060451:85 , srv1289029668:85 , srv1320685531:85 , srv1328307575:85 , srv13284827:85 , srv133331943:85 , srv1345185583:85 , srv134585485:85 , srv1353240125:85 , srv1442059470:85 , srv1541852654:85 , srv1601491661:85 , srv1649335164:85 , srv1719571749:85 , srv1727965906:85 , srv1734018463:85 , srv1763262018:85 , srv1782086507:85 , srv1810343988:85 , srv1865181363:85 , srv1867723716:85 , srv1927915078:85 , srv1942682335:85 , srv1987767468:85 , srv2002686151:85 , srv201347765:85 , srv2038791120:85 , srv204688969:85 , srv2081489920:85 , srv208574880:85 , srv2090438834:85 , srv2098918922:85 , srv25126411:85 , srv251703916:85 , srv253377758:85 , srv284516860:85 , srv299358088:85 , srv30372976:85 , srv451471651:85 , srv511859002:85 , srv560191124:85 , srv597404246:85 , srv659904085:85 , srv730746388:85 , srv864561168:85 , srv943704762:85 , srv1130886181:86 , srv201008269:86 , srv502784084:86 , srv712943198:86 , srv1233212638:626 }
2023-07-30 16:53:15,776 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=293 (was 275) - SystemLoadAverage LEAK? -, ProcessCount=173 (was 173), AvailableMemoryMB=2175 (was 2890)
2023-07-30 16:53:15,787 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=293, ProcessCount=173, AvailableMemoryMB=2169
2023-07-30 16:53:15,788 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-07-30 16:53:15,789 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-07-30 16:53:15,800 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1014888591:84 , srv1079272183:84 , srv1109738706:84 , srv1134424348:84 , srv1308164819:84 , srv138680549:84 , srv1453679739:84 , srv1487951513:84 , srv1586877359:84 , srv163560581:84 , srv173915585:84 , srv1882722459:84 , srv1977330633:84 , srv2011650652:84 , srv218497889:84 , srv226183165:84 , srv245279598:84 , srv36135912:84 , srv510501372:84 , srv543886373:84 , srv59383368:84 , srv663506260:84 , srv696267567:84 , srv719837726:84 , srv73315553:84 , srv849431588:84 , srv857629810:84 , srv97926680:84 , srv995097948:84 , srv1331703721:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-07-30 16:53:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-30 16:53:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv163560581=10, srv995097948=29, srv1586877359=9, srv1134424348=3, srv1109738706=2, srv173915585=11, srv1882722459=12, srv510501372=19, srv849431588=26, srv226183165=16, srv97926680=28, srv2011650652=14, srv663506260=22, srv696267567=23, srv719837726=24, srv73315553=25, srv245279598=17, srv1308164819=4, srv1331703721=5, srv218497889=15, srv1453679739=7, srv138680549=6, srv36135912=18, srv1079272183=1, srv1487951513=8, srv1014888591=0, srv857629810=27, srv59383368=21, srv1977330633=13, srv543886373=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-30 16:53:15,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-30 16:53:15,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-30 16:53:15,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-30 16:53:15,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-30 16:53:15,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-30 16:53:15,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-30 16:53:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-30 16:53:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-30 16:53:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-30 16:53:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-30 16:53:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-30 16:53:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-30 16:53:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-30 16:53:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-30 16:53:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-30 16:53:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-30 16:53:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-30 16:53:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-30 16:53:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-30 16:53:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-30 16:53:15,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-30 16:53:15,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-30 16:53:15,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-30 16:53:15,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-30 16:53:15,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-30 16:53:15,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-30 16:53:15,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-30 16:53:15,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-30 16:53:15,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-30 16:53:15,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-30 16:53:15,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-07-30 16:53:15,829 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.2081775203537977, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-30 16:53:15,830 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912849003833292, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.2081775203537977, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-07-30 16:55:15,819 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6046836 different iterations.  Found a solution that moves 1863 regions; Going from a computed imbalance of 0.9912849003833292 to a new imbalance of 4.3483349388262195E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.69, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-30 16:55:15,827 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1014888591:90 , srv1079272183:90 , srv1109738706:90 , srv1134424348:90 , srv1308164819:90 , srv1331703721:90 , srv138680549:90 , srv1453679739:90 , srv1487951513:90 , srv1586877359:90 , srv163560581:90 , srv173915585:90 , srv1882722459:90 , srv1977330633:90 , srv2011650652:90 , srv218497889:90 , srv226183165:90 , srv245279598:90 , srv36135912:90 , srv510501372:90 , srv543886373:90 , srv59383368:90 , srv663506260:90 , srv696267567:90 , srv719837726:90 , srv73315553:90 , srv849431588:90 , srv857629810:90 , srv97926680:90 , srv995097948:90 }
2023-07-30 16:55:15,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-30 16:55:15,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv163560581=10, srv995097948=29, srv1586877359=9, srv1134424348=3, srv1109738706=2, srv173915585=11, srv1882722459=12, srv510501372=19, srv849431588=26, srv226183165=16, srv97926680=28, srv2011650652=14, srv663506260=22, srv696267567=23, srv719837726=24, srv73315553=25, srv245279598=17, srv1308164819=4, srv1331703721=5, srv218497889=15, srv1453679739=7, srv138680549=6, srv36135912=18, srv1079272183=1, srv1487951513=8, srv1014888591=0, srv857629810=27, srv59383368=21, srv1977330633=13, srv543886373=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-30 16:55:15,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-30 16:55:15,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-30 16:55:15,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-30 16:55:15,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-30 16:55:15,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-30 16:55:15,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-30 16:55:15,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-30 16:55:15,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-30 16:55:15,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-30 16:55:15,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-30 16:55:15,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-30 16:55:15,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-30 16:55:15,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-30 16:55:15,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-30 16:55:15,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-30 16:55:15,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-30 16:55:15,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-30 16:55:15,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-30 16:55:15,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-30 16:55:15,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-30 16:55:15,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-30 16:55:15,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-30 16:55:15,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-30 16:55:15,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-30 16:55:15,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-30 16:55:15,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-30 16:55:15,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-30 16:55:15,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-30 16:55:15,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-30 16:55:15,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-30 16:55:15,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-07-30 16:55:15,845 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-30 16:55:15,845 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1014888591:90 , srv1079272183:90 , srv1109738706:90 , srv1134424348:90 , srv1308164819:90 , srv1331703721:90 , srv138680549:90 , srv1453679739:90 , srv1487951513:90 , srv1586877359:90 , srv163560581:90 , srv173915585:90 , srv1882722459:90 , srv1977330633:90 , srv2011650652:90 , srv218497889:90 , srv226183165:90 , srv245279598:90 , srv36135912:90 , srv510501372:90 , srv543886373:90 , srv59383368:90 , srv663506260:90 , srv696267567:90 , srv719837726:90 , srv73315553:90 , srv849431588:90 , srv857629810:90 , srv97926680:90 , srv995097948:90 }
2023-07-30 16:55:15,858 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=224 (was 293), ProcessCount=171 (was 173), AvailableMemoryMB=1882 (was 2169)