2023-07-26 17:53:07,232 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-07-26 17:53:07,539 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-07-26 17:53:07,653 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-07-26 17:53:07,724 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-07-26 17:53:07,732 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-07-26 17:53:07,745 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=287, ProcessCount=174, AvailableMemoryMB=9115
2023-07-26 17:53:07,748 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-07-26 17:53:07,748 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-07-26 17:53:07,898 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1050500324:84 , srv1078964916:84 , srv1087233750:84 , srv1152806126:84 , srv1153630798:84 , srv1156744216:84 , srv1171801983:84 , srv1175338581:84 , srv1192655475:84 , srv1245739639:84 , srv1270831774:84 , srv1289909315:84 , srv1290519899:84 , srv1320919761:84 , srv1324184468:84 , srv1376378060:84 , srv1381600752:84 , srv1387778176:84 , srv140622061:84 , srv1416382082:84 , srv1427652360:84 , srv1432958925:84 , srv1451914798:84 , srv1464077733:84 , srv1465562084:84 , srv1507686682:84 , srv15139813:84 , srv1517222287:84 , srv1582508080:84 , srv1590648970:84 , srv1596649719:84 , srv1632320229:84 , srv1653231722:84 , srv1669136952:84 , srv1671532398:84 , srv1738848608:84 , srv1792111868:84 , srv1796568834:84 , srv1808743605:84 , srv1817953501:84 , srv181881977:84 , srv182867565:84 , srv1865303598:84 , srv1873603123:84 , srv1885529068:84 , srv189254736:84 , srv1993720694:84 , srv2000012152:84 , srv2046693680:84 , srv207199061:84 , srv2076685067:84 , srv238641257:84 , srv239587824:84 , srv252892775:84 , srv263011284:84 , srv270826484:84 , srv276583167:84 , srv28847248:84 , srv303519105:84 , srv307731876:84 , srv32475497:84 , srv35686499:84 , srv362065098:84 , srv405349531:84 , srv420584756:84 , srv443222849:84 , srv453113494:84 , srv481670812:84 , srv481726822:84 , srv52132240:84 , srv562910625:84 , srv567056378:84 , srv592869523:84 , srv601894363:84 , srv617650387:84 , srv620673100:84 , srv672120792:84 , srv70773430:84 , srv714735606:84 , srv72094419:84 , srv747785749:84 , srv759057871:84 , srv780696660:84 , srv789341263:84 , srv791746229:84 , srv802505274:84 , srv817651055:84 , srv83918065:84 , srv859198576:84 , srv894597746:84 , srv898625234:84 , srv92167501:84 , srv938726840:84 , srv95271551:84 , srv955193962:84 , srv96474595:84 , srv973505341:84 , srv984266959:84 , srv996926258:84 , srv364034014:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-07-26 17:53:07,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-26 17:53:07,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1507686682=25, srv672120792=77, srv747785749=81, srv1152806126=3, srv1153630798=4, srv894597746=90, srv1290519899=12, srv276583167=56, srv1376378060=15, srv1078964916=1, srv1427652360=20, srv817651055=87, srv802505274=86, srv238641257=51, srv35686499=61, srv984266959=98, srv1156744216=5, srv189254736=45, srv481670812=68, srv96474595=96, srv1817953501=39, srv364034014=63, srv938726840=93, srv362065098=62, srv420584756=65, srv1175338581=7, srv1596649719=30, srv955193962=95, srv789341263=84, srv1387778176=17, srv1792111868=36, srv481726822=69, srv601894363=74, srv1796568834=37, srv252892775=53, srv1192655475=8, srv1738848608=35, srv2046693680=48, srv1416382082=19, srv1050500324=0, srv1885529068=44, srv1381600752=16, srv207199061=49, srv28847248=57, srv1669136952=33, srv1171801983=6, srv1590648970=29, srv898625234=91, srv567056378=72, srv92167501=92, srv714735606=79, srv307731876=59, srv562910625=71, srv1324184468=14, srv1873603123=43, srv263011284=54, srv239587824=52, srv973505341=97, srv1270831774=10, srv2076685067=50, srv95271551=94, srv620673100=76, srv1808743605=38, srv52132240=70, srv1993720694=46, srv83918065=88, srv791746229=85, srv1632320229=31, srv140622061=18, srv1653231722=32, srv592869523=73, srv1465562084=24, srv2000012152=47, srv1289909315=11, srv270826484=55, srv859198576=89, srv72094419=80, srv32475497=60, srv1245739639=9, srv1320919761=13, srv759057871=82, srv443222849=66, srv1517222287=27, srv780696660=83, srv182867565=41, srv1671532398=34, srv453113494=67, srv1087233750=2, srv617650387=75, srv1582508080=28, srv1451914798=22, srv1432958925=21, srv1865303598=42, srv303519105=58, srv70773430=78, srv15139813=26, srv996926258=99, srv1464077733=23, srv405349531=64, srv181881977=40} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-26 17:53:07,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-26 17:53:07,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-26 17:53:07,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-26 17:53:07,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-26 17:53:07,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-26 17:53:07,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-26 17:53:07,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-26 17:53:07,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-26 17:53:07,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-26 17:53:07,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-26 17:53:07,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-26 17:53:07,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-26 17:53:07,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-26 17:53:07,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-26 17:53:07,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-26 17:53:07,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-26 17:53:07,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-26 17:53:07,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-26 17:53:07,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-26 17:53:07,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-26 17:53:07,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-26 17:53:07,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-26 17:53:07,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-26 17:53:07,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-26 17:53:07,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-26 17:53:07,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-26 17:53:07,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-26 17:53:07,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-26 17:53:07,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-26 17:53:07,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-26 17:53:07,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-26 17:53:07,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-26 17:53:07,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-26 17:53:07,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-26 17:53:07,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-26 17:53:07,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-26 17:53:07,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-26 17:53:07,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-26 17:53:07,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-26 17:53:07,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-26 17:53:07,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-26 17:53:07,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-26 17:53:07,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-26 17:53:07,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-26 17:53:07,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-26 17:53:07,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-26 17:53:07,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-26 17:53:07,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-26 17:53:07,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-26 17:53:07,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-26 17:53:07,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-26 17:53:07,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-26 17:53:07,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-26 17:53:07,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-26 17:53:07,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-26 17:53:07,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-26 17:53:07,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-26 17:53:07,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-26 17:53:07,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-26 17:53:07,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-26 17:53:07,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-26 17:53:07,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-26 17:53:07,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-26 17:53:07,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-26 17:53:07,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-26 17:53:07,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-26 17:53:07,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-26 17:53:07,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-26 17:53:07,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-26 17:53:07,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-26 17:53:07,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-26 17:53:07,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-26 17:53:07,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-26 17:53:07,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-26 17:53:07,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-26 17:53:07,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-26 17:53:07,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-26 17:53:07,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-26 17:53:07,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-26 17:53:07,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-26 17:53:07,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-26 17:53:07,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-26 17:53:07,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-26 17:53:07,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-26 17:53:07,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-26 17:53:07,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-26 17:53:07,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-26 17:53:07,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-26 17:53:07,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-26 17:53:07,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-26 17:53:07,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-26 17:53:07,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-26 17:53:07,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-26 17:53:07,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-26 17:53:07,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-26 17:53:07,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-26 17:53:07,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-26 17:53:07,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-26 17:53:07,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-26 17:53:07,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-26 17:53:07,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-26 17:53:08,478 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-26 17:53:08,479 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-07-26 17:53:08,479 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-07-26 17:53:09,217 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1208 ms to try 5000 different iterations.  Found a solution that moves 4687 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17996789395795884. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040075997140761044, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05874864184021265, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5207777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040075997140761044, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17533333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24025, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-26 17:53:09,253 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv15139813:81 , srv1050500324:82 , srv1517222287:82 , srv1808743605:82 , srv1865303598:82 , srv405349531:82 , srv817651055:82 , srv1153630798:83 , srv1270831774:83 , srv1432958925:83 , srv1465562084:83 , srv1507686682:83 , srv443222849:83 , srv83918065:83 , srv92167501:83 , srv1320919761:84 , srv1376378060:84 , srv140622061:84 , srv1427652360:84 , srv1653231722:84 , srv1738848608:84 , srv2046693680:84 , srv252892775:84 , srv28847248:84 , srv32475497:84 , srv481670812:84 , srv72094419:84 , srv780696660:84 , srv938726840:84 , srv96474595:84 , srv1078964916:85 , srv1152806126:85 , srv1171801983:85 , srv1671532398:85 , srv182867565:85 , srv1885529068:85 , srv189254736:85 , srv307731876:85 , srv35686499:85 , srv789341263:85 , srv802505274:85 , srv1245739639:86 , srv1596649719:86 , srv270826484:86 , srv453113494:86 , srv747785749:86 , srv791746229:86 , srv859198576:86 , srv95271551:86 , srv996926258:86 , srv1087233750:87 , srv1381600752:87 , srv1464077733:87 , srv1792111868:87 , srv1796568834:87 , srv181881977:87 , srv276583167:87 , srv481726822:87 , srv592869523:87 , srv601894363:87 , srv672120792:87 , srv70773430:87 , srv894597746:87 , srv973505341:87 , srv1289909315:88 , srv1290519899:88 , srv1324184468:88 , srv1387778176:88 , srv1416382082:88 , srv1582508080:88 , srv1632320229:88 , srv239587824:88 , srv263011284:88 , srv420584756:88 , srv52132240:88 , srv562910625:88 , srv567056378:88 , srv1175338581:89 , srv1192655475:89 , srv1451914798:89 , srv1669136952:89 , srv1993720694:89 , srv207199061:89 , srv303519105:89 , srv759057871:89 , srv898625234:89 , srv1156744216:90 , srv1590648970:90 , srv1873603123:90 , srv2000012152:90 , srv2076685067:90 , srv362065098:90 , srv617650387:90 , srv620673100:91 , srv238641257:92 , srv714735606:92 , srv955193962:92 , srv984266959:92 , srv1817953501:96 , srv364034014:446 }
2023-07-26 17:53:09,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-26 17:53:09,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1507686682=25, srv672120792=77, srv747785749=81, srv1152806126=3, srv1153630798=4, srv894597746=90, srv1290519899=12, srv276583167=56, srv1376378060=15, srv1078964916=1, srv1427652360=20, srv817651055=87, srv802505274=86, srv238641257=51, srv35686499=61, srv984266959=98, srv1156744216=5, srv189254736=45, srv481670812=68, srv96474595=96, srv1817953501=39, srv364034014=63, srv938726840=93, srv362065098=62, srv420584756=65, srv1175338581=7, srv1596649719=30, srv955193962=95, srv789341263=84, srv1387778176=17, srv1792111868=36, srv481726822=69, srv601894363=74, srv1796568834=37, srv252892775=53, srv1192655475=8, srv1738848608=35, srv2046693680=48, srv1416382082=19, srv1050500324=0, srv1885529068=44, srv1381600752=16, srv207199061=49, srv28847248=57, srv1669136952=33, srv1171801983=6, srv1590648970=29, srv898625234=91, srv567056378=72, srv92167501=92, srv714735606=79, srv307731876=59, srv562910625=71, srv1324184468=14, srv1873603123=43, srv263011284=54, srv239587824=52, srv973505341=97, srv1270831774=10, srv2076685067=50, srv95271551=94, srv620673100=76, srv1808743605=38, srv52132240=70, srv1993720694=46, srv83918065=88, srv791746229=85, srv1632320229=31, srv140622061=18, srv1653231722=32, srv592869523=73, srv1465562084=24, srv2000012152=47, srv1289909315=11, srv270826484=55, srv859198576=89, srv72094419=80, srv32475497=60, srv1245739639=9, srv1320919761=13, srv759057871=82, srv443222849=66, srv1517222287=27, srv780696660=83, srv182867565=41, srv1671532398=34, srv453113494=67, srv1087233750=2, srv617650387=75, srv1582508080=28, srv1451914798=22, srv1432958925=21, srv1865303598=42, srv303519105=58, srv70773430=78, srv15139813=26, srv996926258=99, srv1464077733=23, srv405349531=64, srv181881977=40} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-26 17:53:09,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-26 17:53:09,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-26 17:53:09,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-26 17:53:09,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-26 17:53:09,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-26 17:53:09,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-26 17:53:09,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-26 17:53:09,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-26 17:53:09,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-26 17:53:09,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-26 17:53:09,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-26 17:53:09,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-26 17:53:09,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-26 17:53:09,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-26 17:53:09,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-26 17:53:09,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-26 17:53:09,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-26 17:53:09,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-26 17:53:09,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-26 17:53:09,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-26 17:53:09,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-26 17:53:09,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-26 17:53:09,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-26 17:53:09,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-26 17:53:09,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-26 17:53:09,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-26 17:53:09,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-26 17:53:09,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-26 17:53:09,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-26 17:53:09,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-26 17:53:09,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-26 17:53:09,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-26 17:53:09,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-26 17:53:09,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-26 17:53:09,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-26 17:53:09,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-26 17:53:09,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-26 17:53:09,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-26 17:53:09,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-26 17:53:09,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-26 17:53:09,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-26 17:53:09,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-26 17:53:09,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-26 17:53:09,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-26 17:53:09,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-26 17:53:09,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-26 17:53:09,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-26 17:53:09,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-26 17:53:09,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-26 17:53:09,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-26 17:53:09,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-26 17:53:09,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-26 17:53:09,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-26 17:53:09,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-26 17:53:09,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-26 17:53:09,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-26 17:53:09,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-26 17:53:09,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-26 17:53:09,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-26 17:53:09,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-26 17:53:09,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-26 17:53:09,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-26 17:53:09,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-26 17:53:09,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-26 17:53:09,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-26 17:53:09,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-26 17:53:09,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-26 17:53:09,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-26 17:53:09,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-26 17:53:09,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-26 17:53:09,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-26 17:53:09,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-26 17:53:09,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-26 17:53:09,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-26 17:53:09,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-26 17:53:09,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-26 17:53:09,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-26 17:53:09,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-26 17:53:09,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-26 17:53:09,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-26 17:53:09,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-26 17:53:09,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-26 17:53:09,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-26 17:53:09,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-26 17:53:09,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-26 17:53:09,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-26 17:53:09,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-26 17:53:09,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-26 17:53:09,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-26 17:53:09,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-26 17:53:09,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-26 17:53:09,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-26 17:53:09,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-26 17:53:09,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-26 17:53:09,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-26 17:53:09,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-26 17:53:09,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-26 17:53:09,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-26 17:53:09,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-26 17:53:09,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-26 17:53:09,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-26 17:53:09,464 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040075997140761044, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05874864184021265, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040075997140761044, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17533333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24025, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-26 17:53:09,464 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-07-26 17:53:09,464 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17993507488250265, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040075997140761044, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05874864184021265, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040075997140761044, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17533333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24025, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-07-26 17:53:09,792 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 481 ms to try 5000 different iterations.  Found a solution that moves 2651 regions; Going from a computed imbalance of 0.17993507488250265 to a new imbalance of 0.015610357274807182. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019824016869532182); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030901167532670266, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29455555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019824016869532182); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-26 17:53:09,805 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1817953501:78 , srv714735606:78 , srv955193962:80 , srv2000012152:81 , srv620673100:81 , srv984266959:81 , srv1590648970:82 , srv238641257:82 , srv362065098:82 , srv617650387:82 , srv1192655475:83 , srv1289909315:83 , srv1324184468:83 , srv1416382082:83 , srv1632320229:83 , srv2076685067:83 , srv263011284:83 , srv973505341:83 , srv1087233750:84 , srv1156744216:84 , srv1290519899:84 , srv1381600752:84 , srv1387778176:84 , srv1451914798:84 , srv1993720694:84 , srv276583167:84 , srv420584756:84 , srv52132240:84 , srv562910625:84 , srv70773430:84 , srv759057871:84 , srv894597746:84 , srv898625234:84 , srv1175338581:85 , srv1245739639:85 , srv1464077733:85 , srv1792111868:85 , srv1873603123:85 , srv1885529068:85 , srv303519105:85 , srv453113494:85 , srv481726822:85 , srv567056378:85 , srv1171801983:86 , srv1669136952:86 , srv1796568834:86 , srv239587824:86 , srv270826484:86 , srv35686499:86 , srv592869523:86 , srv672120792:86 , srv747785749:86 , srv791746229:86 , srv802505274:86 , srv95271551:86 , srv1078964916:87 , srv1152806126:87 , srv1376378060:87 , srv140622061:87 , srv1427652360:87 , srv1432958925:87 , srv1653231722:87 , srv1738848608:87 , srv181881977:87 , srv182867565:87 , srv1865303598:87 , srv189254736:87 , srv481670812:87 , srv601894363:87 , srv72094419:87 , srv780696660:87 , srv789341263:87 , srv859198576:87 , srv938726840:87 , srv996926258:87 , srv1270831774:88 , srv1596649719:88 , srv1671532398:88 , srv207199061:88 , srv28847248:88 , srv307731876:88 , srv817651055:88 , srv92167501:88 , srv96474595:88 , srv1153630798:89 , srv1320919761:89 , srv1507686682:89 , srv15139813:89 , srv1582508080:89 , srv252892775:89 , srv443222849:89 , srv83918065:89 , srv1465562084:90 , srv32475497:90 , srv405349531:90 , srv1517222287:91 , srv1050500324:92 , srv1808743605:92 , srv2046693680:94 , srv364034014:504 }
2023-07-26 17:53:09,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-26 17:53:09,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1507686682=25, srv672120792=77, srv747785749=81, srv1152806126=3, srv1153630798=4, srv894597746=90, srv1290519899=12, srv276583167=56, srv1376378060=15, srv1078964916=1, srv1427652360=20, srv817651055=87, srv802505274=86, srv238641257=51, srv35686499=61, srv984266959=98, srv1156744216=5, srv189254736=45, srv481670812=68, srv96474595=96, srv1817953501=39, srv364034014=63, srv938726840=93, srv362065098=62, srv420584756=65, srv1175338581=7, srv1596649719=30, srv955193962=95, srv789341263=84, srv1387778176=17, srv1792111868=36, srv481726822=69, srv601894363=74, srv1796568834=37, srv252892775=53, srv1192655475=8, srv1738848608=35, srv2046693680=48, srv1416382082=19, srv1050500324=0, srv1885529068=44, srv1381600752=16, srv207199061=49, srv28847248=57, srv1669136952=33, srv1171801983=6, srv1590648970=29, srv898625234=91, srv567056378=72, srv92167501=92, srv714735606=79, srv307731876=59, srv562910625=71, srv1324184468=14, srv1873603123=43, srv263011284=54, srv239587824=52, srv973505341=97, srv1270831774=10, srv2076685067=50, srv95271551=94, srv620673100=76, srv1808743605=38, srv52132240=70, srv1993720694=46, srv83918065=88, srv791746229=85, srv1632320229=31, srv140622061=18, srv1653231722=32, srv592869523=73, srv1465562084=24, srv2000012152=47, srv1289909315=11, srv270826484=55, srv859198576=89, srv72094419=80, srv32475497=60, srv1245739639=9, srv1320919761=13, srv759057871=82, srv443222849=66, srv1517222287=27, srv780696660=83, srv182867565=41, srv1671532398=34, srv453113494=67, srv1087233750=2, srv617650387=75, srv1582508080=28, srv1451914798=22, srv1432958925=21, srv1865303598=42, srv303519105=58, srv70773430=78, srv15139813=26, srv996926258=99, srv1464077733=23, srv405349531=64, srv181881977=40} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-26 17:53:09,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-26 17:53:09,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-26 17:53:09,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-26 17:53:09,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-26 17:53:09,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-26 17:53:09,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-26 17:53:09,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-26 17:53:09,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-26 17:53:09,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-26 17:53:09,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-26 17:53:09,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-26 17:53:09,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-26 17:53:09,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-26 17:53:09,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-26 17:53:09,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-26 17:53:09,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-26 17:53:09,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-26 17:53:09,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-26 17:53:09,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-26 17:53:09,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-26 17:53:09,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-26 17:53:09,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-26 17:53:09,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-26 17:53:09,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-26 17:53:09,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-26 17:53:09,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-26 17:53:09,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-26 17:53:09,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-26 17:53:09,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-26 17:53:09,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-26 17:53:09,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-26 17:53:09,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-26 17:53:09,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-26 17:53:09,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-26 17:53:09,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-26 17:53:09,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-26 17:53:09,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-26 17:53:09,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-26 17:53:09,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-26 17:53:09,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-26 17:53:09,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-26 17:53:09,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-26 17:53:09,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-26 17:53:09,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-26 17:53:09,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-26 17:53:09,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-26 17:53:09,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-26 17:53:09,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-26 17:53:09,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-26 17:53:09,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-26 17:53:09,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-26 17:53:09,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-26 17:53:09,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-26 17:53:09,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-26 17:53:09,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-26 17:53:09,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-26 17:53:09,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-26 17:53:09,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-26 17:53:09,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-26 17:53:09,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-26 17:53:09,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-26 17:53:09,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-26 17:53:09,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-26 17:53:09,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-26 17:53:09,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-26 17:53:09,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-26 17:53:09,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-26 17:53:09,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-26 17:53:09,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-26 17:53:09,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-26 17:53:09,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-26 17:53:09,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-26 17:53:09,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-26 17:53:09,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-26 17:53:09,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-26 17:53:09,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-26 17:53:09,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-26 17:53:09,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-26 17:53:09,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-26 17:53:09,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-26 17:53:09,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-26 17:53:09,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-26 17:53:09,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-26 17:53:09,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-26 17:53:09,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-26 17:53:09,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-26 17:53:09,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-26 17:53:09,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-26 17:53:09,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-26 17:53:09,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-26 17:53:09,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-26 17:53:09,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-26 17:53:09,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-26 17:53:09,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-26 17:53:09,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-26 17:53:09,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-26 17:53:09,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-26 17:53:09,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-26 17:53:09,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-26 17:53:09,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-26 17:53:09,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-26 17:53:09,990 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019824016869532182); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030901167532670266, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019824016869532182); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-26 17:53:09,990 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-07-26 17:53:09,991 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015591794576058665, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019824016869532182); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030901167532670266, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019824016869532182); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-07-26 17:53:10,311 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 442 ms to try 5000 different iterations.  Found a solution that moves 1056 regions; Going from a computed imbalance of 0.015591794576058665 to a new imbalance of 8.650192337447798E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004118218141796809); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007784989441615229); MoveCostFunction : (multiplier=7.0, imbalance=0.11733333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004118218141796809); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0030833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-26 17:53:10,312 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv207199061:80 , srv2046693680:81 , srv1175338581:82 , srv1582508080:82 , srv1669136952:82 , srv1873603123:82 , srv1050500324:83 , srv1156744216:83 , srv239587824:83 , srv601894363:83 , srv1320919761:84 , srv1465562084:84 , srv1507686682:84 , srv1517222287:84 , srv1596649719:84 , srv1817953501:84 , srv238641257:84 , srv252892775:84 , srv303519105:84 , srv32475497:84 , srv567056378:84 , srv592869523:84 , srv984266959:84 , srv996926258:84 , srv1192655475:85 , srv1387778176:85 , srv1451914798:85 , srv1464077733:85 , srv1590648970:85 , srv1671532398:85 , srv1808743605:85 , srv181881977:85 , srv189254736:85 , srv1993720694:85 , srv2076685067:85 , srv270826484:85 , srv28847248:85 , srv307731876:85 , srv420584756:85 , srv443222849:85 , srv481726822:85 , srv52132240:85 , srv562910625:85 , srv672120792:85 , srv747785749:85 , srv759057871:85 , srv791746229:85 , srv859198576:85 , srv898625234:85 , srv1078964916:86 , srv1152806126:86 , srv1153630798:86 , srv1171801983:86 , srv1270831774:86 , srv1381600752:86 , srv1416382082:86 , srv1632320229:86 , srv1792111868:86 , srv1796568834:86 , srv182867565:86 , srv1885529068:86 , srv35686499:86 , srv362065098:86 , srv405349531:86 , srv453113494:86 , srv617650387:86 , srv620673100:86 , srv780696660:86 , srv789341263:86 , srv83918065:86 , srv894597746:86 , srv938726840:86 , srv95271551:86 , srv1087233750:87 , srv1245739639:87 , srv1289909315:87 , srv1290519899:87 , srv1324184468:87 , srv1376378060:87 , srv140622061:87 , srv1427652360:87 , srv1432958925:87 , srv1653231722:87 , srv1738848608:87 , srv2000012152:87 , srv263011284:87 , srv276583167:87 , srv481670812:87 , srv72094419:87 , srv802505274:87 , srv817651055:87 , srv92167501:87 , srv955193962:87 , srv96474595:87 , srv70773430:88 , srv714735606:88 , srv15139813:89 , srv1865303598:89 , srv973505341:89 , srv364034014:544 }
2023-07-26 17:53:10,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-26 17:53:10,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1507686682=25, srv672120792=77, srv747785749=81, srv1152806126=3, srv1153630798=4, srv894597746=90, srv1290519899=12, srv276583167=56, srv1376378060=15, srv1078964916=1, srv1427652360=20, srv817651055=87, srv802505274=86, srv238641257=51, srv35686499=61, srv984266959=98, srv1156744216=5, srv189254736=45, srv481670812=68, srv96474595=96, srv1817953501=39, srv364034014=63, srv938726840=93, srv362065098=62, srv420584756=65, srv1175338581=7, srv1596649719=30, srv955193962=95, srv789341263=84, srv1387778176=17, srv1792111868=36, srv481726822=69, srv601894363=74, srv1796568834=37, srv252892775=53, srv1192655475=8, srv1738848608=35, srv2046693680=48, srv1416382082=19, srv1050500324=0, srv1885529068=44, srv1381600752=16, srv207199061=49, srv28847248=57, srv1669136952=33, srv1171801983=6, srv1590648970=29, srv898625234=91, srv567056378=72, srv92167501=92, srv714735606=79, srv307731876=59, srv562910625=71, srv1324184468=14, srv1873603123=43, srv263011284=54, srv239587824=52, srv973505341=97, srv1270831774=10, srv2076685067=50, srv95271551=94, srv620673100=76, srv1808743605=38, srv52132240=70, srv1993720694=46, srv83918065=88, srv791746229=85, srv1632320229=31, srv140622061=18, srv1653231722=32, srv592869523=73, srv1465562084=24, srv2000012152=47, srv1289909315=11, srv270826484=55, srv859198576=89, srv72094419=80, srv32475497=60, srv1245739639=9, srv1320919761=13, srv759057871=82, srv443222849=66, srv1517222287=27, srv780696660=83, srv182867565=41, srv1671532398=34, srv453113494=67, srv1087233750=2, srv617650387=75, srv1582508080=28, srv1451914798=22, srv1432958925=21, srv1865303598=42, srv303519105=58, srv70773430=78, srv15139813=26, srv996926258=99, srv1464077733=23, srv405349531=64, srv181881977=40} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-26 17:53:10,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-26 17:53:10,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-26 17:53:10,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-26 17:53:10,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-26 17:53:10,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-26 17:53:10,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-26 17:53:10,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-26 17:53:10,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-26 17:53:10,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-26 17:53:10,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-26 17:53:10,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-26 17:53:10,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-26 17:53:10,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-26 17:53:10,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-26 17:53:10,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-26 17:53:10,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-26 17:53:10,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-26 17:53:10,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-26 17:53:10,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-26 17:53:10,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-26 17:53:10,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-26 17:53:10,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-26 17:53:10,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-26 17:53:10,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-26 17:53:10,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-26 17:53:10,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-26 17:53:10,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-26 17:53:10,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-26 17:53:10,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-26 17:53:10,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-26 17:53:10,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-26 17:53:10,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-26 17:53:10,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-26 17:53:10,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-26 17:53:10,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-26 17:53:10,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-26 17:53:10,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-26 17:53:10,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-26 17:53:10,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-26 17:53:10,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-26 17:53:10,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-26 17:53:10,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-26 17:53:10,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-26 17:53:10,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-26 17:53:10,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-26 17:53:10,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-26 17:53:10,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-26 17:53:10,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-26 17:53:10,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-26 17:53:10,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-26 17:53:10,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-26 17:53:10,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-26 17:53:10,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-26 17:53:10,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-26 17:53:10,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-26 17:53:10,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-26 17:53:10,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-26 17:53:10,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-26 17:53:10,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-26 17:53:10,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-26 17:53:10,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-26 17:53:10,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-26 17:53:10,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-26 17:53:10,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-26 17:53:10,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-26 17:53:10,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-26 17:53:10,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-26 17:53:10,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-26 17:53:10,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-26 17:53:10,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-26 17:53:10,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-26 17:53:10,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-26 17:53:10,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-26 17:53:10,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-26 17:53:10,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-26 17:53:10,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-26 17:53:10,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-26 17:53:10,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-26 17:53:10,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-26 17:53:10,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-26 17:53:10,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-26 17:53:10,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-26 17:53:10,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-26 17:53:10,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-26 17:53:10,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-26 17:53:10,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-26 17:53:10,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-26 17:53:10,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-26 17:53:10,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-26 17:53:10,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-26 17:53:10,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-26 17:53:10,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-26 17:53:10,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-26 17:53:10,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-26 17:53:10,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-26 17:53:10,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-26 17:53:10,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-26 17:53:10,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-26 17:53:10,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-26 17:53:10,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-26 17:53:10,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-26 17:53:10,457 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004118218141796809); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007784989441615229); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004118218141796809); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0030833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-26 17:53:10,457 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-07-26 17:53:10,457 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.576249637038772E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004118218141796809); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007784989441615229); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004118218141796809); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0030833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-07-26 17:53:10,799 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 442 ms to try 5000 different iterations.  Found a solution that moves 247 regions; Going from a computed imbalance of 8.576249637038772E-4 to a new imbalance of 1.412531437827633E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024158102511315956); MoveCostFunction : (multiplier=7.0, imbalance=0.027444444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-26 17:53:10,799 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1152806126:83 , srv1290519899:83 , srv15139813:83 , srv1582508080:83 , srv1796568834:83 , srv181881977:83 , srv2046693680:83 , srv70773430:83 , srv955193962:83 , srv96474595:83 , srv973505341:83 , srv1078964916:84 , srv1087233750:84 , srv1153630798:84 , srv1245739639:84 , srv1289909315:84 , srv1324184468:84 , srv1376378060:84 , srv140622061:84 , srv1427652360:84 , srv1451914798:84 , srv1517222287:84 , srv1596649719:84 , srv1653231722:84 , srv1671532398:84 , srv1738848608:84 , srv1792111868:84 , srv1808743605:84 , srv1817953501:84 , srv182867565:84 , srv1865303598:84 , srv1993720694:84 , srv2000012152:84 , srv2076685067:84 , srv238641257:84 , srv263011284:84 , srv276583167:84 , srv303519105:84 , srv307731876:84 , srv32475497:84 , srv362065098:84 , srv405349531:84 , srv481670812:84 , srv617650387:84 , srv620673100:84 , srv672120792:84 , srv714735606:84 , srv72094419:84 , srv759057871:84 , srv780696660:84 , srv789341263:84 , srv802505274:84 , srv83918065:84 , srv859198576:84 , srv898625234:84 , srv92167501:84 , srv1050500324:85 , srv1156744216:85 , srv1171801983:85 , srv1192655475:85 , srv1270831774:85 , srv1320919761:85 , srv1381600752:85 , srv1387778176:85 , srv1416382082:85 , srv1432958925:85 , srv1464077733:85 , srv1465562084:85 , srv1507686682:85 , srv1590648970:85 , srv1632320229:85 , srv1669136952:85 , srv1873603123:85 , srv189254736:85 , srv207199061:85 , srv239587824:85 , srv252892775:85 , srv270826484:85 , srv35686499:85 , srv420584756:85 , srv443222849:85 , srv453113494:85 , srv481726822:85 , srv52132240:85 , srv562910625:85 , srv567056378:85 , srv592869523:85 , srv747785749:85 , srv791746229:85 , srv817651055:85 , srv894597746:85 , srv938726840:85 , srv95271551:85 , srv984266959:85 , srv996926258:85 , srv1175338581:86 , srv1885529068:86 , srv28847248:86 , srv601894363:86 , srv364034014:648 }
2023-07-26 17:53:10,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-26 17:53:10,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1507686682=25, srv672120792=77, srv747785749=81, srv1152806126=3, srv1153630798=4, srv894597746=90, srv1290519899=12, srv276583167=56, srv1376378060=15, srv1078964916=1, srv1427652360=20, srv817651055=87, srv802505274=86, srv238641257=51, srv35686499=61, srv984266959=98, srv1156744216=5, srv189254736=45, srv481670812=68, srv96474595=96, srv1817953501=39, srv364034014=63, srv938726840=93, srv362065098=62, srv420584756=65, srv1175338581=7, srv1596649719=30, srv955193962=95, srv789341263=84, srv1387778176=17, srv1792111868=36, srv481726822=69, srv601894363=74, srv1796568834=37, srv252892775=53, srv1192655475=8, srv1738848608=35, srv2046693680=48, srv1416382082=19, srv1050500324=0, srv1885529068=44, srv1381600752=16, srv207199061=49, srv28847248=57, srv1669136952=33, srv1171801983=6, srv1590648970=29, srv898625234=91, srv567056378=72, srv92167501=92, srv714735606=79, srv307731876=59, srv562910625=71, srv1324184468=14, srv1873603123=43, srv263011284=54, srv239587824=52, srv973505341=97, srv1270831774=10, srv2076685067=50, srv95271551=94, srv620673100=76, srv1808743605=38, srv52132240=70, srv1993720694=46, srv83918065=88, srv791746229=85, srv1632320229=31, srv140622061=18, srv1653231722=32, srv592869523=73, srv1465562084=24, srv2000012152=47, srv1289909315=11, srv270826484=55, srv859198576=89, srv72094419=80, srv32475497=60, srv1245739639=9, srv1320919761=13, srv759057871=82, srv443222849=66, srv1517222287=27, srv780696660=83, srv182867565=41, srv1671532398=34, srv453113494=67, srv1087233750=2, srv617650387=75, srv1582508080=28, srv1451914798=22, srv1432958925=21, srv1865303598=42, srv303519105=58, srv70773430=78, srv15139813=26, srv996926258=99, srv1464077733=23, srv405349531=64, srv181881977=40} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-26 17:53:10,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-26 17:53:10,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-26 17:53:10,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-26 17:53:10,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-26 17:53:10,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-26 17:53:10,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-26 17:53:10,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-26 17:53:10,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-26 17:53:10,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-26 17:53:10,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-26 17:53:10,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-26 17:53:10,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-26 17:53:10,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-26 17:53:10,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-26 17:53:10,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-26 17:53:10,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-26 17:53:10,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-26 17:53:10,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-26 17:53:10,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-26 17:53:10,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-26 17:53:10,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-26 17:53:10,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-26 17:53:10,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-26 17:53:10,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-26 17:53:10,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-26 17:53:10,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-26 17:53:10,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-26 17:53:10,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-26 17:53:10,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-26 17:53:10,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-26 17:53:10,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-26 17:53:10,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-26 17:53:10,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-26 17:53:10,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-26 17:53:10,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-26 17:53:10,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-26 17:53:10,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-26 17:53:10,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-26 17:53:10,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-26 17:53:10,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-26 17:53:10,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-26 17:53:10,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-26 17:53:10,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-26 17:53:10,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-26 17:53:10,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-26 17:53:10,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-26 17:53:10,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-26 17:53:10,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-26 17:53:10,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-26 17:53:10,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-26 17:53:10,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-26 17:53:10,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-26 17:53:10,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-26 17:53:10,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-26 17:53:10,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-26 17:53:10,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-26 17:53:10,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-26 17:53:10,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-26 17:53:10,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-26 17:53:10,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-26 17:53:10,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-26 17:53:10,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-26 17:53:10,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-26 17:53:10,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-26 17:53:10,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-26 17:53:10,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-26 17:53:10,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-26 17:53:10,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-26 17:53:10,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-26 17:53:10,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-26 17:53:10,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-26 17:53:10,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-26 17:53:10,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-26 17:53:10,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-26 17:53:10,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-26 17:53:10,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-26 17:53:10,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-26 17:53:10,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-26 17:53:10,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-26 17:53:10,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-26 17:53:10,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-26 17:53:10,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-26 17:53:10,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-26 17:53:10,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-26 17:53:10,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-26 17:53:10,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-26 17:53:10,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-26 17:53:10,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-26 17:53:10,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-26 17:53:10,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-26 17:53:10,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-26 17:53:10,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-26 17:53:10,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-26 17:53:10,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-26 17:53:10,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-26 17:53:10,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-26 17:53:10,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-26 17:53:10,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-26 17:53:10,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-26 17:53:10,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-26 17:53:10,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-26 17:53:10,961 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.2395783412269768E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024158102511315956); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-26 17:53:10,961 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1152806126:83 , srv1290519899:83 , srv15139813:83 , srv1582508080:83 , srv1796568834:83 , srv181881977:83 , srv2046693680:83 , srv70773430:83 , srv955193962:83 , srv96474595:83 , srv973505341:83 , srv1078964916:84 , srv1087233750:84 , srv1153630798:84 , srv1245739639:84 , srv1289909315:84 , srv1324184468:84 , srv1376378060:84 , srv140622061:84 , srv1427652360:84 , srv1451914798:84 , srv1517222287:84 , srv1596649719:84 , srv1653231722:84 , srv1671532398:84 , srv1738848608:84 , srv1792111868:84 , srv1808743605:84 , srv1817953501:84 , srv182867565:84 , srv1865303598:84 , srv1993720694:84 , srv2000012152:84 , srv2076685067:84 , srv238641257:84 , srv263011284:84 , srv276583167:84 , srv303519105:84 , srv307731876:84 , srv32475497:84 , srv362065098:84 , srv405349531:84 , srv481670812:84 , srv617650387:84 , srv620673100:84 , srv672120792:84 , srv714735606:84 , srv72094419:84 , srv759057871:84 , srv780696660:84 , srv789341263:84 , srv802505274:84 , srv83918065:84 , srv859198576:84 , srv898625234:84 , srv92167501:84 , srv1050500324:85 , srv1156744216:85 , srv1171801983:85 , srv1192655475:85 , srv1270831774:85 , srv1320919761:85 , srv1381600752:85 , srv1387778176:85 , srv1416382082:85 , srv1432958925:85 , srv1464077733:85 , srv1465562084:85 , srv1507686682:85 , srv1590648970:85 , srv1632320229:85 , srv1669136952:85 , srv1873603123:85 , srv189254736:85 , srv207199061:85 , srv239587824:85 , srv252892775:85 , srv270826484:85 , srv35686499:85 , srv420584756:85 , srv443222849:85 , srv453113494:85 , srv481726822:85 , srv52132240:85 , srv562910625:85 , srv567056378:85 , srv592869523:85 , srv747785749:85 , srv791746229:85 , srv817651055:85 , srv894597746:85 , srv938726840:85 , srv95271551:85 , srv984266959:85 , srv996926258:85 , srv1175338581:86 , srv1885529068:86 , srv28847248:86 , srv601894363:86 , srv364034014:648 }
2023-07-26 17:53:11,000 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=312 (was 287) - SystemLoadAverage LEAK? -, ProcessCount=174 (was 174), AvailableMemoryMB=8252 (was 9115)
2023-07-26 17:53:11,010 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=312, ProcessCount=174, AvailableMemoryMB=8252
2023-07-26 17:53:11,011 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-07-26 17:53:11,012 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-07-26 17:53:11,027 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1044103912:84 , srv1050616662:84 , srv1060060671:84 , srv1077838132:84 , srv1091939767:84 , srv1145139215:84 , srv1205713548:84 , srv1290044191:84 , srv1312379250:84 , srv1371796405:84 , srv1503871296:84 , srv1573014693:84 , srv1764296428:84 , srv1802969309:84 , srv186523944:84 , srv1901724585:84 , srv1921519137:84 , srv2003735569:84 , srv2125109576:84 , srv256359477:84 , srv272354248:84 , srv280020253:84 , srv325549701:84 , srv745223936:84 , srv795134928:84 , srv850639223:84 , srv927789766:84 , srv932172374:84 , srv942517244:84 , srv1458358942:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-07-26 17:53:11,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-26 17:53:11,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1091939767=4, srv1458358942=10, srv1921519137=17, srv256359477=20, srv1371796405=9, srv1503871296=11, srv2125109576=19, srv1205713548=6, srv932172374=28, srv1145139215=5, srv1764296428=13, srv1802969309=14, srv2003735569=18, srv927789766=27, srv1901724585=16, srv1573014693=12, srv1044103912=0, srv1077838132=3, srv850639223=26, srv1060060671=2, srv1312379250=8, srv272354248=21, srv1050616662=1, srv795134928=25, srv186523944=15, srv745223936=24, srv1290044191=7, srv325549701=23, srv942517244=29, srv280020253=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-26 17:53:11,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-26 17:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-26 17:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-26 17:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-26 17:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-26 17:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-26 17:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-26 17:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-26 17:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-26 17:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-26 17:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-26 17:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-26 17:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-26 17:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-26 17:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-26 17:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-26 17:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-26 17:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-26 17:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-26 17:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-26 17:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-26 17:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-26 17:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-26 17:53:11,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-26 17:53:11,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-26 17:53:11,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-26 17:53:11,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-26 17:53:11,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-26 17:53:11,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-26 17:53:11,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-26 17:53:11,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-07-26 17:53:11,055 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0908748812916105, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-26 17:53:11,056 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912479387047892, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0908748812916105, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-07-26 17:55:11,045 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6206719 different iterations.  Found a solution that moves 1852 regions; Going from a computed imbalance of 0.9912479387047892 to a new imbalance of 4.322660390073086E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6859259259259259, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-26 17:55:11,053 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1044103912:90 , srv1050616662:90 , srv1060060671:90 , srv1077838132:90 , srv1091939767:90 , srv1145139215:90 , srv1205713548:90 , srv1290044191:90 , srv1312379250:90 , srv1371796405:90 , srv1458358942:90 , srv1503871296:90 , srv1573014693:90 , srv1764296428:90 , srv1802969309:90 , srv186523944:90 , srv1901724585:90 , srv1921519137:90 , srv2003735569:90 , srv2125109576:90 , srv256359477:90 , srv272354248:90 , srv280020253:90 , srv325549701:90 , srv745223936:90 , srv795134928:90 , srv850639223:90 , srv927789766:90 , srv932172374:90 , srv942517244:90 }
2023-07-26 17:55:11,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-26 17:55:11,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1091939767=4, srv1458358942=10, srv1921519137=17, srv256359477=20, srv1371796405=9, srv1503871296=11, srv2125109576=19, srv1205713548=6, srv932172374=28, srv1145139215=5, srv1764296428=13, srv1802969309=14, srv2003735569=18, srv927789766=27, srv1901724585=16, srv1573014693=12, srv1044103912=0, srv1077838132=3, srv850639223=26, srv1060060671=2, srv1312379250=8, srv272354248=21, srv1050616662=1, srv795134928=25, srv186523944=15, srv745223936=24, srv1290044191=7, srv325549701=23, srv942517244=29, srv280020253=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-26 17:55:11,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-26 17:55:11,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-26 17:55:11,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-26 17:55:11,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-26 17:55:11,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-26 17:55:11,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-26 17:55:11,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-26 17:55:11,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-26 17:55:11,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-26 17:55:11,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-26 17:55:11,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-26 17:55:11,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-26 17:55:11,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-26 17:55:11,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-26 17:55:11,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-26 17:55:11,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-26 17:55:11,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-26 17:55:11,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-26 17:55:11,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-26 17:55:11,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-26 17:55:11,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-26 17:55:11,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-26 17:55:11,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-26 17:55:11,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-26 17:55:11,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-26 17:55:11,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-26 17:55:11,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-26 17:55:11,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-26 17:55:11,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-26 17:55:11,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-26 17:55:11,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-07-26 17:55:11,074 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-26 17:55:11,074 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1044103912:90 , srv1050616662:90 , srv1060060671:90 , srv1077838132:90 , srv1091939767:90 , srv1145139215:90 , srv1205713548:90 , srv1290044191:90 , srv1312379250:90 , srv1371796405:90 , srv1458358942:90 , srv1503871296:90 , srv1573014693:90 , srv1764296428:90 , srv1802969309:90 , srv186523944:90 , srv1901724585:90 , srv1921519137:90 , srv2003735569:90 , srv2125109576:90 , srv256359477:90 , srv272354248:90 , srv280020253:90 , srv325549701:90 , srv745223936:90 , srv795134928:90 , srv850639223:90 , srv927789766:90 , srv932172374:90 , srv942517244:90 }
2023-07-26 17:55:11,092 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=240 (was 312), ProcessCount=172 (was 174), AvailableMemoryMB=8418 (was 8252) - AvailableMemoryMB LEAK? -