2023-07-29 03:53:02,802 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-07-29 03:53:03,079 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-07-29 03:53:03,196 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-07-29 03:53:03,255 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-07-29 03:53:03,263 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-07-29 03:53:03,278 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=455, ProcessCount=169, AvailableMemoryMB=5835
2023-07-29 03:53:03,279 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-07-29 03:53:03,280 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-07-29 03:53:03,394 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1022107977:84 , srv1028696183:84 , srv1051061101:84 , srv1069290571:84 , srv1108263209:84 , srv1108363646:84 , srv1137123457:84 , srv1150942728:84 , srv1155120454:84 , srv1199936786:84 , srv1202482966:84 , srv1214072763:84 , srv1219650413:84 , srv124519287:84 , srv1267939019:84 , srv128051250:84 , srv1296951358:84 , srv1297777085:84 , srv1319247638:84 , srv1336378072:84 , srv1389582771:84 , srv1405129998:84 , srv140834990:84 , srv1446264657:84 , srv147217393:84 , srv148733486:84 , srv1492384964:84 , srv1497922081:84 , srv1512727052:84 , srv1516419535:84 , srv1574490732:84 , srv1588754029:84 , srv1593181514:84 , srv1593330483:84 , srv1597891922:84 , srv1627090091:84 , srv1662669068:84 , srv1724467998:84 , srv1749548852:84 , srv1772663016:84 , srv17752822:84 , srv1787278202:84 , srv1808014848:84 , srv1855985468:84 , srv1863436427:84 , srv1879261399:84 , srv192772278:84 , srv1965434304:84 , srv1971984063:84 , srv2016842155:84 , srv2062379531:84 , srv2094281484:84 , srv2126675970:84 , srv2128614436:84 , srv2128808754:84 , srv2130114762:84 , srv2143852088:84 , srv215594311:84 , srv228091490:84 , srv256205547:84 , srv266118838:84 , srv267017812:84 , srv268310214:84 , srv283877757:84 , srv314276139:84 , srv340125613:84 , srv343966749:84 , srv398666017:84 , srv407145969:84 , srv407729221:84 , srv417299113:84 , srv474568463:84 , srv498042253:84 , srv535670786:84 , srv554656408:84 , srv580591755:84 , srv592337239:84 , srv606216029:84 , srv615630651:84 , srv6201317:84 , srv634203905:84 , srv656037803:84 , srv6641767:84 , srv684604166:84 , srv715083001:84 , srv719043454:84 , srv725945976:84 , srv757826745:84 , srv760016872:84 , srv783345958:84 , srv78690106:84 , srv814161843:84 , srv852348850:84 , srv862554555:84 , srv883954483:84 , srv905708676:84 , srv932140459:84 , srv952698178:84 , srv979803770:84 , srv547884750:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-07-29 03:53:03,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-29 03:53:03,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1267939019=14, srv1971984063=48, srv2094281484=51, srv684604166=84, srv6201317=80, srv1199936786=9, srv1787278202=41, srv1297777085=17, srv2143852088=56, srv979803770=99, srv1150942728=7, srv1336378072=19, srv474568463=71, srv498042253=72, srv1597891922=34, srv1855985468=43, srv615630651=79, srv2062379531=50, srv547884750=74, srv905708676=96, srv1512727052=28, srv606216029=78, srv554656408=75, srv266118838=60, srv1214072763=11, srv78690106=91, srv128051250=15, srv340125613=65, srv862554555=94, srv1492384964=26, srv2128614436=53, srv1202482966=10, srv417299113=70, srv1497922081=27, srv256205547=59, srv1772663016=39, srv2016842155=49, srv407145969=68, srv1446264657=23, srv783345958=90, srv1137123457=6, srv1108363646=5, srv725945976=87, srv1749548852=38, srv314276139=64, srv1808014848=42, srv1662669068=36, srv719043454=86, srv715083001=85, srv952698178=98, srv760016872=89, srv2128808754=54, srv852348850=93, srv634203905=81, srv283877757=63, srv1155120454=8, srv1593181514=32, srv814161843=92, srv215594311=57, srv757826745=88, srv1724467998=37, srv1574490732=30, srv140834990=22, srv407729221=69, srv1389582771=20, srv1879261399=45, srv1028696183=1, srv1069290571=3, srv932140459=97, srv883954483=95, srv228091490=58, srv1219650413=12, srv1863436427=44, srv398666017=67, srv580591755=76, srv124519287=13, srv1108263209=4, srv656037803=82, srv1405129998=21, srv1296951358=16, srv267017812=61, srv1965434304=47, srv1593330483=33, srv2126675970=52, srv192772278=46, srv1022107977=0, srv1588754029=31, srv17752822=40, srv6641767=83, srv268310214=62, srv148733486=25, srv343966749=66, srv592337239=77, srv2130114762=55, srv1627090091=35, srv1051061101=2, srv1516419535=29, srv535670786=73, srv147217393=24, srv1319247638=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-29 03:53:03,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-29 03:53:03,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-29 03:53:03,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-29 03:53:03,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-29 03:53:03,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-29 03:53:03,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-29 03:53:03,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-29 03:53:03,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-29 03:53:03,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-29 03:53:03,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-29 03:53:03,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-29 03:53:03,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-29 03:53:03,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-29 03:53:03,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-29 03:53:03,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-29 03:53:03,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-29 03:53:03,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-29 03:53:03,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-29 03:53:03,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-29 03:53:03,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-29 03:53:03,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-29 03:53:03,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-29 03:53:03,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-29 03:53:03,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-29 03:53:03,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-29 03:53:03,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-29 03:53:03,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-29 03:53:03,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-29 03:53:03,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-29 03:53:03,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-29 03:53:03,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-29 03:53:03,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-29 03:53:03,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-29 03:53:03,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-29 03:53:03,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-29 03:53:03,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-29 03:53:03,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-29 03:53:03,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-29 03:53:03,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-29 03:53:03,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-29 03:53:03,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-29 03:53:03,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-29 03:53:03,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-29 03:53:03,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-29 03:53:03,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-29 03:53:03,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-29 03:53:03,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-29 03:53:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-29 03:53:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-29 03:53:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-29 03:53:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-29 03:53:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-29 03:53:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-29 03:53:03,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-29 03:53:03,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-29 03:53:03,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-29 03:53:03,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-29 03:53:03,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-29 03:53:03,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-29 03:53:03,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-29 03:53:03,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-29 03:53:03,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-29 03:53:03,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-29 03:53:03,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-29 03:53:03,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-29 03:53:03,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-29 03:53:03,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-29 03:53:03,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-29 03:53:03,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-29 03:53:03,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-29 03:53:03,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-29 03:53:03,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-29 03:53:03,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-29 03:53:03,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-29 03:53:03,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-29 03:53:03,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-29 03:53:03,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-29 03:53:03,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-29 03:53:03,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-29 03:53:03,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-29 03:53:03,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-29 03:53:03,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-29 03:53:03,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-29 03:53:03,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-29 03:53:03,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-29 03:53:03,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-29 03:53:03,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-29 03:53:03,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-29 03:53:03,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-29 03:53:03,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-29 03:53:03,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-29 03:53:03,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-29 03:53:03,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-29 03:53:03,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-29 03:53:03,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-29 03:53:03,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-29 03:53:03,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-29 03:53:03,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-29 03:53:03,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-29 03:53:03,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-29 03:53:03,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-29 03:53:03,648 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-29 03:53:03,648 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-07-29 03:53:03,648 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-07-29 03:53:03,895 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 365 ms to try 5000 different iterations.  Found a solution that moves 4756 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17313752326738302. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04310640167735266, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05800460525855596, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5284444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04310640167735266, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16816666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23591666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-29 03:53:03,917 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1336378072:80 , srv1137123457:81 , srv1199936786:81 , srv1497922081:81 , srv340125613:81 , srv814161843:81 , srv1296951358:82 , srv1597891922:82 , srv1965434304:82 , srv760016872:82 , srv1787278202:83 , srv2094281484:83 , srv407729221:83 , srv580591755:83 , srv6641767:83 , srv684604166:83 , srv952698178:83 , srv1069290571:84 , srv1297777085:84 , srv1512727052:84 , srv1516419535:84 , srv1574490732:84 , srv1627090091:84 , srv17752822:84 , srv256205547:84 , srv267017812:84 , srv535670786:84 , srv1108363646:85 , srv140834990:85 , srv1724467998:85 , srv1772663016:85 , srv1879261399:85 , srv192772278:85 , srv2016842155:85 , srv615630651:85 , srv634203905:85 , srv719043454:85 , srv725945976:85 , srv757826745:85 , srv1022107977:86 , srv1028696183:86 , srv1214072763:86 , srv1405129998:86 , srv148733486:86 , srv1593181514:86 , srv1593330483:86 , srv1808014848:86 , srv2128614436:86 , srv2128808754:86 , srv2143852088:86 , srv283877757:86 , srv314276139:86 , srv398666017:86 , srv474568463:86 , srv592337239:86 , srv606216029:86 , srv656037803:86 , srv715083001:86 , srv78690106:86 , srv852348850:86 , srv979803770:86 , srv1051061101:87 , srv1108263209:87 , srv1219650413:87 , srv1319247638:87 , srv1446264657:87 , srv147217393:87 , srv1492384964:87 , srv1588754029:87 , srv407145969:87 , srv417299113:87 , srv6201317:87 , srv862554555:87 , srv883954483:87 , srv905708676:87 , srv1267939019:88 , srv2126675970:88 , srv2130114762:88 , srv498042253:88 , srv783345958:88 , srv1150942728:89 , srv1202482966:89 , srv128051250:89 , srv1863436427:89 , srv2062379531:89 , srv1155120454:90 , srv1389582771:90 , srv1662669068:90 , srv215594311:90 , srv266118838:90 , srv124519287:91 , srv1855985468:91 , srv1971984063:91 , srv228091490:91 , srv268310214:92 , srv554656408:92 , srv932140459:92 , srv343966749:93 , srv1749548852:95 , srv547884750:473 }
2023-07-29 03:53:03,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-29 03:53:03,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1267939019=14, srv1971984063=48, srv2094281484=51, srv684604166=84, srv6201317=80, srv1199936786=9, srv1787278202=41, srv1297777085=17, srv2143852088=56, srv979803770=99, srv1150942728=7, srv1336378072=19, srv474568463=71, srv498042253=72, srv1597891922=34, srv1855985468=43, srv615630651=79, srv2062379531=50, srv547884750=74, srv905708676=96, srv1512727052=28, srv606216029=78, srv554656408=75, srv266118838=60, srv1214072763=11, srv78690106=91, srv128051250=15, srv340125613=65, srv862554555=94, srv1492384964=26, srv2128614436=53, srv1202482966=10, srv417299113=70, srv1497922081=27, srv256205547=59, srv1772663016=39, srv2016842155=49, srv407145969=68, srv1446264657=23, srv783345958=90, srv1137123457=6, srv1108363646=5, srv725945976=87, srv1749548852=38, srv314276139=64, srv1808014848=42, srv1662669068=36, srv719043454=86, srv715083001=85, srv952698178=98, srv760016872=89, srv2128808754=54, srv852348850=93, srv634203905=81, srv283877757=63, srv1155120454=8, srv1593181514=32, srv814161843=92, srv215594311=57, srv757826745=88, srv1724467998=37, srv1574490732=30, srv140834990=22, srv407729221=69, srv1389582771=20, srv1879261399=45, srv1028696183=1, srv1069290571=3, srv932140459=97, srv883954483=95, srv228091490=58, srv1219650413=12, srv1863436427=44, srv398666017=67, srv580591755=76, srv124519287=13, srv1108263209=4, srv656037803=82, srv1405129998=21, srv1296951358=16, srv267017812=61, srv1965434304=47, srv1593330483=33, srv2126675970=52, srv192772278=46, srv1022107977=0, srv1588754029=31, srv17752822=40, srv6641767=83, srv268310214=62, srv148733486=25, srv343966749=66, srv592337239=77, srv2130114762=55, srv1627090091=35, srv1051061101=2, srv1516419535=29, srv535670786=73, srv147217393=24, srv1319247638=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-29 03:53:03,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-29 03:53:03,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-29 03:53:03,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-29 03:53:03,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-29 03:53:03,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-29 03:53:03,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-29 03:53:03,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-29 03:53:03,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-29 03:53:03,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-29 03:53:03,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-29 03:53:03,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-29 03:53:03,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-29 03:53:03,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-29 03:53:03,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-29 03:53:03,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-29 03:53:03,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-29 03:53:03,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-29 03:53:03,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-29 03:53:03,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-29 03:53:03,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-29 03:53:03,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-29 03:53:03,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-29 03:53:03,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-29 03:53:03,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-29 03:53:03,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-29 03:53:03,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-29 03:53:03,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-29 03:53:03,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-29 03:53:03,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-29 03:53:03,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-29 03:53:03,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-29 03:53:03,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-29 03:53:03,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-29 03:53:03,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-29 03:53:03,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-29 03:53:03,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-29 03:53:03,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-29 03:53:03,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-29 03:53:03,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-29 03:53:03,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-29 03:53:03,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-29 03:53:03,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-29 03:53:03,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-29 03:53:03,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-29 03:53:03,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-29 03:53:03,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-29 03:53:03,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-29 03:53:03,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-29 03:53:03,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-29 03:53:03,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-29 03:53:03,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-29 03:53:03,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-29 03:53:03,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-29 03:53:03,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-29 03:53:03,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-29 03:53:03,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-29 03:53:03,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-29 03:53:03,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-29 03:53:03,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-29 03:53:03,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-29 03:53:03,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-29 03:53:03,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-29 03:53:03,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-29 03:53:03,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-29 03:53:03,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-29 03:53:03,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-29 03:53:03,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-29 03:53:03,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-29 03:53:03,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-29 03:53:03,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-29 03:53:03,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-29 03:53:03,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-29 03:53:03,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-29 03:53:03,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-29 03:53:03,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-29 03:53:03,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-29 03:53:03,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-29 03:53:03,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-29 03:53:03,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-29 03:53:03,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-29 03:53:03,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-29 03:53:03,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-29 03:53:03,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-29 03:53:03,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-29 03:53:03,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-29 03:53:03,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-29 03:53:03,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-29 03:53:03,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-29 03:53:03,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-29 03:53:03,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-29 03:53:03,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-29 03:53:03,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-29 03:53:03,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-29 03:53:03,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-29 03:53:03,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-29 03:53:03,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-29 03:53:03,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-29 03:53:03,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-29 03:53:03,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-29 03:53:03,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-29 03:53:03,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-29 03:53:04,056 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04310640167735266, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05800460525855596, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04310640167735266, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16816666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23591666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-29 03:53:04,056 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-07-29 03:53:04,056 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17310422104360032, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04310640167735266, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05800460525855596, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04310640167735266, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16816666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23591666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-07-29 03:53:04,273 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 312 ms to try 5000 different iterations.  Found a solution that moves 2644 regions; Going from a computed imbalance of 0.17310422104360032 to a new imbalance of 0.015833389477577255. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023882428066894085); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03778371908123163, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2937777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023882428066894085); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043333333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-29 03:53:04,282 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1749548852:75 , srv1855985468:78 , srv268310214:79 , srv932140459:79 , srv1155120454:81 , srv124519287:81 , srv1389582771:81 , srv2062379531:81 , srv215594311:81 , srv554656408:81 , srv1150942728:82 , srv1202482966:82 , srv1219650413:82 , srv128051250:82 , srv1863436427:82 , srv266118838:82 , srv407145969:82 , srv1319247638:83 , srv148733486:83 , srv1662669068:83 , srv783345958:83 , srv1028696183:84 , srv1051061101:84 , srv1492384964:84 , srv1588754029:84 , srv1808014848:84 , srv2126675970:84 , srv2130114762:84 , srv283877757:84 , srv343966749:84 , srv417299113:84 , srv6201317:84 , srv719043454:84 , srv862554555:84 , srv883954483:84 , srv979803770:84 , srv1108263209:85 , srv1267939019:85 , srv140834990:85 , srv147217393:85 , srv1971984063:85 , srv2128808754:85 , srv267017812:85 , srv398666017:85 , srv474568463:85 , srv498042253:85 , srv606216029:85 , srv715083001:85 , srv757826745:85 , srv905708676:85 , srv1214072763:86 , srv17752822:86 , srv2128614436:86 , srv2143852088:86 , srv656037803:86 , srv852348850:86 , srv1022107977:87 , srv1405129998:87 , srv1593181514:87 , srv1627090091:87 , srv1772663016:87 , srv2016842155:87 , srv228091490:87 , srv256205547:87 , srv314276139:87 , srv615630651:87 , srv634203905:87 , srv725945976:87 , srv1297777085:88 , srv1446264657:88 , srv1512727052:88 , srv1593330483:88 , srv1597891922:88 , srv407729221:88 , srv535670786:88 , srv580591755:88 , srv592337239:88 , srv814161843:88 , srv1108363646:89 , srv1296951358:89 , srv1336378072:89 , srv1516419535:89 , srv1574490732:89 , srv1724467998:89 , srv2094281484:89 , srv684604166:89 , srv78690106:89 , srv1069290571:90 , srv1137123457:90 , srv1787278202:90 , srv1879261399:90 , srv1497922081:91 , srv1965434304:91 , srv340125613:91 , srv6641767:91 , srv952698178:91 , srv1199936786:92 , srv760016872:92 , srv192772278:94 , srv547884750:513 }
2023-07-29 03:53:04,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-29 03:53:04,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1267939019=14, srv1971984063=48, srv2094281484=51, srv684604166=84, srv6201317=80, srv1199936786=9, srv1787278202=41, srv1297777085=17, srv2143852088=56, srv979803770=99, srv1150942728=7, srv1336378072=19, srv474568463=71, srv498042253=72, srv1597891922=34, srv1855985468=43, srv615630651=79, srv2062379531=50, srv547884750=74, srv905708676=96, srv1512727052=28, srv606216029=78, srv554656408=75, srv266118838=60, srv1214072763=11, srv78690106=91, srv128051250=15, srv340125613=65, srv862554555=94, srv1492384964=26, srv2128614436=53, srv1202482966=10, srv417299113=70, srv1497922081=27, srv256205547=59, srv1772663016=39, srv2016842155=49, srv407145969=68, srv1446264657=23, srv783345958=90, srv1137123457=6, srv1108363646=5, srv725945976=87, srv1749548852=38, srv314276139=64, srv1808014848=42, srv1662669068=36, srv719043454=86, srv715083001=85, srv952698178=98, srv760016872=89, srv2128808754=54, srv852348850=93, srv634203905=81, srv283877757=63, srv1155120454=8, srv1593181514=32, srv814161843=92, srv215594311=57, srv757826745=88, srv1724467998=37, srv1574490732=30, srv140834990=22, srv407729221=69, srv1389582771=20, srv1879261399=45, srv1028696183=1, srv1069290571=3, srv932140459=97, srv883954483=95, srv228091490=58, srv1219650413=12, srv1863436427=44, srv398666017=67, srv580591755=76, srv124519287=13, srv1108263209=4, srv656037803=82, srv1405129998=21, srv1296951358=16, srv267017812=61, srv1965434304=47, srv1593330483=33, srv2126675970=52, srv192772278=46, srv1022107977=0, srv1588754029=31, srv17752822=40, srv6641767=83, srv268310214=62, srv148733486=25, srv343966749=66, srv592337239=77, srv2130114762=55, srv1627090091=35, srv1051061101=2, srv1516419535=29, srv535670786=73, srv147217393=24, srv1319247638=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-29 03:53:04,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-29 03:53:04,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-29 03:53:04,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-29 03:53:04,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-29 03:53:04,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-29 03:53:04,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-29 03:53:04,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-29 03:53:04,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-29 03:53:04,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-29 03:53:04,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-29 03:53:04,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-29 03:53:04,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-29 03:53:04,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-29 03:53:04,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-29 03:53:04,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-29 03:53:04,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-29 03:53:04,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-29 03:53:04,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-29 03:53:04,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-29 03:53:04,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-29 03:53:04,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-29 03:53:04,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-29 03:53:04,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-29 03:53:04,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-29 03:53:04,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-29 03:53:04,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-29 03:53:04,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-29 03:53:04,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-29 03:53:04,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-29 03:53:04,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-29 03:53:04,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-29 03:53:04,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-29 03:53:04,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-29 03:53:04,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-29 03:53:04,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-29 03:53:04,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-29 03:53:04,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-29 03:53:04,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-29 03:53:04,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-29 03:53:04,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-29 03:53:04,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-29 03:53:04,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-29 03:53:04,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-29 03:53:04,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-29 03:53:04,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-29 03:53:04,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-29 03:53:04,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-29 03:53:04,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-29 03:53:04,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-29 03:53:04,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-29 03:53:04,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-29 03:53:04,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-29 03:53:04,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-29 03:53:04,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-29 03:53:04,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-29 03:53:04,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-29 03:53:04,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-29 03:53:04,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-29 03:53:04,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-29 03:53:04,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-29 03:53:04,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-29 03:53:04,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-29 03:53:04,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-29 03:53:04,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-29 03:53:04,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-29 03:53:04,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-29 03:53:04,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-29 03:53:04,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-29 03:53:04,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-29 03:53:04,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-29 03:53:04,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-29 03:53:04,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-29 03:53:04,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-29 03:53:04,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-29 03:53:04,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-29 03:53:04,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-29 03:53:04,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-29 03:53:04,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-29 03:53:04,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-29 03:53:04,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-29 03:53:04,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-29 03:53:04,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-29 03:53:04,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-29 03:53:04,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-29 03:53:04,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-29 03:53:04,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-29 03:53:04,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-29 03:53:04,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-29 03:53:04,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-29 03:53:04,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-29 03:53:04,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-29 03:53:04,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-29 03:53:04,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-29 03:53:04,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-29 03:53:04,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-29 03:53:04,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-29 03:53:04,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-29 03:53:04,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-29 03:53:04,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-29 03:53:04,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-29 03:53:04,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-29 03:53:04,404 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023882428066894085); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03778371908123163, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023882428066894085); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043333333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-29 03:53:04,404 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-07-29 03:53:04,404 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015814875793876356, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023882428066894085); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03778371908123163, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023882428066894085); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043333333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-07-29 03:53:04,609 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 283 ms to try 5000 different iterations.  Found a solution that moves 988 regions; Going from a computed imbalance of 0.015814875793876356 to a new imbalance of 6.400881853621665E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006874790976200997); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009971902164189443); MoveCostFunction : (multiplier=7.0, imbalance=0.10977777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006874790976200997); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-29 03:53:04,611 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv192772278:78 , srv228091490:80 , srv343966749:81 , srv1446264657:82 , srv1971984063:82 , srv1069290571:83 , srv1593330483:83 , srv1879261399:83 , srv592337239:83 , srv78690106:83 , srv952698178:83 , srv1108363646:84 , srv1199936786:84 , srv1267939019:84 , srv1516419535:84 , srv1574490732:84 , srv1662669068:84 , srv1724467998:84 , srv1787278202:84 , srv498042253:84 , srv554656408:84 , srv6641767:84 , srv760016872:84 , srv1022107977:85 , srv1108263209:85 , srv1202482966:85 , srv1297777085:85 , srv1405129998:85 , srv1497922081:85 , srv1512727052:85 , srv1593181514:85 , srv2016842155:85 , srv2094281484:85 , srv2128614436:85 , srv2130114762:85 , srv314276139:85 , srv340125613:85 , srv615630651:85 , srv634203905:85 , srv656037803:85 , srv725945976:85 , srv852348850:85 , srv905708676:85 , srv1051061101:86 , srv1150942728:86 , srv1214072763:86 , srv124519287:86 , srv1296951358:86 , srv1389582771:86 , srv147217393:86 , srv1492384964:86 , srv1588754029:86 , srv1627090091:86 , srv1772663016:86 , srv1863436427:86 , srv1965434304:86 , srv2128808754:86 , srv2143852088:86 , srv256205547:86 , srv266118838:86 , srv398666017:86 , srv407729221:86 , srv535670786:86 , srv580591755:86 , srv606216029:86 , srv6201317:86 , srv783345958:86 , srv862554555:86 , srv883954483:86 , srv1028696183:87 , srv1137123457:87 , srv1155120454:87 , srv128051250:87 , srv1319247638:87 , srv140834990:87 , srv1597891922:87 , srv1749548852:87 , srv17752822:87 , srv1808014848:87 , srv2062379531:87 , srv2126675970:87 , srv215594311:87 , srv268310214:87 , srv283877757:87 , srv417299113:87 , srv474568463:87 , srv684604166:87 , srv715083001:87 , srv932140459:87 , srv979803770:87 , srv1219650413:88 , srv1336378072:88 , srv148733486:88 , srv1855985468:88 , srv407145969:88 , srv757826745:88 , srv267017812:89 , srv719043454:89 , srv814161843:89 , srv547884750:533 }
2023-07-29 03:53:04,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-29 03:53:04,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1267939019=14, srv1971984063=48, srv2094281484=51, srv684604166=84, srv6201317=80, srv1199936786=9, srv1787278202=41, srv1297777085=17, srv2143852088=56, srv979803770=99, srv1150942728=7, srv1336378072=19, srv474568463=71, srv498042253=72, srv1597891922=34, srv1855985468=43, srv615630651=79, srv2062379531=50, srv547884750=74, srv905708676=96, srv1512727052=28, srv606216029=78, srv554656408=75, srv266118838=60, srv1214072763=11, srv78690106=91, srv128051250=15, srv340125613=65, srv862554555=94, srv1492384964=26, srv2128614436=53, srv1202482966=10, srv417299113=70, srv1497922081=27, srv256205547=59, srv1772663016=39, srv2016842155=49, srv407145969=68, srv1446264657=23, srv783345958=90, srv1137123457=6, srv1108363646=5, srv725945976=87, srv1749548852=38, srv314276139=64, srv1808014848=42, srv1662669068=36, srv719043454=86, srv715083001=85, srv952698178=98, srv760016872=89, srv2128808754=54, srv852348850=93, srv634203905=81, srv283877757=63, srv1155120454=8, srv1593181514=32, srv814161843=92, srv215594311=57, srv757826745=88, srv1724467998=37, srv1574490732=30, srv140834990=22, srv407729221=69, srv1389582771=20, srv1879261399=45, srv1028696183=1, srv1069290571=3, srv932140459=97, srv883954483=95, srv228091490=58, srv1219650413=12, srv1863436427=44, srv398666017=67, srv580591755=76, srv124519287=13, srv1108263209=4, srv656037803=82, srv1405129998=21, srv1296951358=16, srv267017812=61, srv1965434304=47, srv1593330483=33, srv2126675970=52, srv192772278=46, srv1022107977=0, srv1588754029=31, srv17752822=40, srv6641767=83, srv268310214=62, srv148733486=25, srv343966749=66, srv592337239=77, srv2130114762=55, srv1627090091=35, srv1051061101=2, srv1516419535=29, srv535670786=73, srv147217393=24, srv1319247638=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-29 03:53:04,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-29 03:53:04,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-29 03:53:04,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-29 03:53:04,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-29 03:53:04,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-29 03:53:04,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-29 03:53:04,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-29 03:53:04,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-29 03:53:04,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-29 03:53:04,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-29 03:53:04,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-29 03:53:04,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-29 03:53:04,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-29 03:53:04,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-29 03:53:04,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-29 03:53:04,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-29 03:53:04,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-29 03:53:04,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-29 03:53:04,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-29 03:53:04,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-29 03:53:04,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-29 03:53:04,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-29 03:53:04,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-29 03:53:04,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-29 03:53:04,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-29 03:53:04,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-29 03:53:04,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-29 03:53:04,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-29 03:53:04,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-29 03:53:04,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-29 03:53:04,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-29 03:53:04,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-29 03:53:04,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-29 03:53:04,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-29 03:53:04,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-29 03:53:04,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-29 03:53:04,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-29 03:53:04,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-29 03:53:04,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-29 03:53:04,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-29 03:53:04,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-29 03:53:04,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-29 03:53:04,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-29 03:53:04,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-29 03:53:04,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-29 03:53:04,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-29 03:53:04,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-29 03:53:04,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-29 03:53:04,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-29 03:53:04,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-29 03:53:04,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-29 03:53:04,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-29 03:53:04,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-29 03:53:04,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-29 03:53:04,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-29 03:53:04,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-29 03:53:04,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-29 03:53:04,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-29 03:53:04,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-29 03:53:04,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-29 03:53:04,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-29 03:53:04,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-29 03:53:04,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-29 03:53:04,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-29 03:53:04,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-29 03:53:04,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-29 03:53:04,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-29 03:53:04,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-29 03:53:04,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-29 03:53:04,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-29 03:53:04,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-29 03:53:04,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-29 03:53:04,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-29 03:53:04,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-29 03:53:04,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-29 03:53:04,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-29 03:53:04,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-29 03:53:04,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-29 03:53:04,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-29 03:53:04,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-29 03:53:04,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-29 03:53:04,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-29 03:53:04,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-29 03:53:04,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-29 03:53:04,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-29 03:53:04,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-29 03:53:04,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-29 03:53:04,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-29 03:53:04,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-29 03:53:04,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-29 03:53:04,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-29 03:53:04,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-29 03:53:04,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-29 03:53:04,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-29 03:53:04,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-29 03:53:04,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-29 03:53:04,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-29 03:53:04,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-29 03:53:04,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-29 03:53:04,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-29 03:53:04,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-29 03:53:04,718 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006874790976200997); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009971902164189443); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006874790976200997); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-29 03:53:04,718 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-07-29 03:53:04,719 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.331700614981403E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006874790976200997); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009971902164189443); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006874790976200997); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-07-29 03:53:04,941 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 300 ms to try 5000 different iterations.  Found a solution that moves 289 regions; Going from a computed imbalance of 6.331700614981403E-4 to a new imbalance of 2.8563978347337335E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.03211111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-29 03:53:04,942 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1965434304:83 , srv228091490:83 , srv684604166:83 , srv1022107977:84 , srv1108363646:84 , srv1137123457:84 , srv1155120454:84 , srv1214072763:84 , srv124519287:84 , srv1267939019:84 , srv128051250:84 , srv1296951358:84 , srv1389582771:84 , srv1405129998:84 , srv140834990:84 , srv147217393:84 , srv1497922081:84 , srv1574490732:84 , srv1597891922:84 , srv1724467998:84 , srv1749548852:84 , srv1772663016:84 , srv1808014848:84 , srv1879261399:84 , srv1971984063:84 , srv2126675970:84 , srv2143852088:84 , srv215594311:84 , srv266118838:84 , srv267017812:84 , srv268310214:84 , srv343966749:84 , srv417299113:84 , srv474568463:84 , srv535670786:84 , srv606216029:84 , srv615630651:84 , srv656037803:84 , srv6641767:84 , srv719043454:84 , srv78690106:84 , srv814161843:84 , srv932140459:84 , srv1028696183:85 , srv1051061101:85 , srv1069290571:85 , srv1150942728:85 , srv1199936786:85 , srv1219650413:85 , srv1297777085:85 , srv1319247638:85 , srv1336378072:85 , srv1446264657:85 , srv148733486:85 , srv1492384964:85 , srv1512727052:85 , srv1588754029:85 , srv1593181514:85 , srv1593330483:85 , srv1627090091:85 , srv1662669068:85 , srv17752822:85 , srv1787278202:85 , srv1855985468:85 , srv192772278:85 , srv2016842155:85 , srv2062379531:85 , srv2094281484:85 , srv2128614436:85 , srv2128808754:85 , srv2130114762:85 , srv256205547:85 , srv283877757:85 , srv314276139:85 , srv340125613:85 , srv398666017:85 , srv407145969:85 , srv498042253:85 , srv554656408:85 , srv580591755:85 , srv634203905:85 , srv715083001:85 , srv725945976:85 , srv757826745:85 , srv760016872:85 , srv783345958:85 , srv862554555:85 , srv883954483:85 , srv905708676:85 , srv952698178:85 , srv979803770:85 , srv1108263209:86 , srv1202482966:86 , srv1516419535:86 , srv1863436427:86 , srv407729221:86 , srv592337239:86 , srv6201317:86 , srv852348850:86 , srv547884750:623 }
2023-07-29 03:53:04,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-29 03:53:04,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1267939019=14, srv1971984063=48, srv2094281484=51, srv684604166=84, srv6201317=80, srv1199936786=9, srv1787278202=41, srv1297777085=17, srv2143852088=56, srv979803770=99, srv1150942728=7, srv1336378072=19, srv474568463=71, srv498042253=72, srv1597891922=34, srv1855985468=43, srv615630651=79, srv2062379531=50, srv547884750=74, srv905708676=96, srv1512727052=28, srv606216029=78, srv554656408=75, srv266118838=60, srv1214072763=11, srv78690106=91, srv128051250=15, srv340125613=65, srv862554555=94, srv1492384964=26, srv2128614436=53, srv1202482966=10, srv417299113=70, srv1497922081=27, srv256205547=59, srv1772663016=39, srv2016842155=49, srv407145969=68, srv1446264657=23, srv783345958=90, srv1137123457=6, srv1108363646=5, srv725945976=87, srv1749548852=38, srv314276139=64, srv1808014848=42, srv1662669068=36, srv719043454=86, srv715083001=85, srv952698178=98, srv760016872=89, srv2128808754=54, srv852348850=93, srv634203905=81, srv283877757=63, srv1155120454=8, srv1593181514=32, srv814161843=92, srv215594311=57, srv757826745=88, srv1724467998=37, srv1574490732=30, srv140834990=22, srv407729221=69, srv1389582771=20, srv1879261399=45, srv1028696183=1, srv1069290571=3, srv932140459=97, srv883954483=95, srv228091490=58, srv1219650413=12, srv1863436427=44, srv398666017=67, srv580591755=76, srv124519287=13, srv1108263209=4, srv656037803=82, srv1405129998=21, srv1296951358=16, srv267017812=61, srv1965434304=47, srv1593330483=33, srv2126675970=52, srv192772278=46, srv1022107977=0, srv1588754029=31, srv17752822=40, srv6641767=83, srv268310214=62, srv148733486=25, srv343966749=66, srv592337239=77, srv2130114762=55, srv1627090091=35, srv1051061101=2, srv1516419535=29, srv535670786=73, srv147217393=24, srv1319247638=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-29 03:53:04,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-29 03:53:04,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-29 03:53:04,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-29 03:53:04,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-29 03:53:04,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-29 03:53:04,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-29 03:53:04,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-29 03:53:04,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-29 03:53:04,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-29 03:53:04,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-29 03:53:04,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-29 03:53:04,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-29 03:53:04,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-29 03:53:04,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-29 03:53:04,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-29 03:53:04,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-29 03:53:04,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-29 03:53:04,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-29 03:53:04,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-29 03:53:04,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-29 03:53:04,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-29 03:53:04,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-29 03:53:04,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-29 03:53:04,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-29 03:53:04,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-29 03:53:04,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-29 03:53:04,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-29 03:53:04,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-29 03:53:04,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-29 03:53:04,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-29 03:53:04,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-29 03:53:04,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-29 03:53:04,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-29 03:53:04,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-29 03:53:04,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-29 03:53:04,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-29 03:53:04,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-29 03:53:04,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-29 03:53:04,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-29 03:53:04,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-29 03:53:04,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-29 03:53:04,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-29 03:53:04,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-29 03:53:04,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-29 03:53:04,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-29 03:53:04,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-29 03:53:04,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-29 03:53:04,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-29 03:53:04,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-29 03:53:04,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-29 03:53:04,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-29 03:53:04,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-29 03:53:04,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-29 03:53:04,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-29 03:53:04,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-29 03:53:04,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-29 03:53:04,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-29 03:53:04,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-29 03:53:04,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-29 03:53:04,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-29 03:53:04,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-29 03:53:04,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-29 03:53:04,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-29 03:53:04,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-29 03:53:04,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-29 03:53:04,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-29 03:53:04,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-29 03:53:04,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-29 03:53:04,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-29 03:53:04,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-29 03:53:04,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-29 03:53:04,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-29 03:53:04,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-29 03:53:04,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-29 03:53:04,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-29 03:53:04,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-29 03:53:04,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-29 03:53:04,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-29 03:53:04,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-29 03:53:04,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-29 03:53:04,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-29 03:53:04,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-29 03:53:04,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-29 03:53:04,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-29 03:53:04,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-29 03:53:04,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-29 03:53:04,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-29 03:53:04,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-29 03:53:04,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-29 03:53:04,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-29 03:53:04,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-29 03:53:04,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-29 03:53:04,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-29 03:53:04,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-29 03:53:04,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-29 03:53:04,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-29 03:53:04,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-29 03:53:04,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-29 03:53:04,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-29 03:53:04,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-29 03:53:04,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-29 03:53:05,057 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-29 03:53:05,057 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-07-29 03:53:05,057 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.6540357095613055E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-07-29 03:53:05,204 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 225 ms to try 5000 different iterations.  Found a solution that moves 33 regions; Going from a computed imbalance of 2.6540357095613055E-5 to a new imbalance of 6.629056611047903E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0036666666666666666); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-29 03:53:05,205 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1108263209:83 , srv1516419535:83 , srv1593181514:83 , srv1863436427:83 , srv2126675970:83 , srv314276139:83 , srv407729221:83 , srv592337239:83 , srv6201317:83 , srv715083001:83 , srv757826745:83 , srv760016872:83 , srv852348850:83 , srv1022107977:84 , srv1028696183:84 , srv1051061101:84 , srv1069290571:84 , srv1108363646:84 , srv1137123457:84 , srv1150942728:84 , srv1155120454:84 , srv1199936786:84 , srv1202482966:84 , srv1214072763:84 , srv1219650413:84 , srv124519287:84 , srv128051250:84 , srv1297777085:84 , srv1319247638:84 , srv1336378072:84 , srv1405129998:84 , srv1446264657:84 , srv147217393:84 , srv148733486:84 , srv1492384964:84 , srv1512727052:84 , srv1588754029:84 , srv1593330483:84 , srv1627090091:84 , srv1662669068:84 , srv1724467998:84 , srv1772663016:84 , srv17752822:84 , srv1787278202:84 , srv1855985468:84 , srv1879261399:84 , srv192772278:84 , srv1965434304:84 , srv1971984063:84 , srv2016842155:84 , srv2062379531:84 , srv2094281484:84 , srv2128614436:84 , srv2128808754:84 , srv2130114762:84 , srv2143852088:84 , srv215594311:84 , srv256205547:84 , srv266118838:84 , srv267017812:84 , srv268310214:84 , srv283877757:84 , srv340125613:84 , srv343966749:84 , srv398666017:84 , srv407145969:84 , srv417299113:84 , srv474568463:84 , srv498042253:84 , srv535670786:84 , srv554656408:84 , srv580591755:84 , srv634203905:84 , srv6641767:84 , srv684604166:84 , srv719043454:84 , srv725945976:84 , srv783345958:84 , srv78690106:84 , srv814161843:84 , srv862554555:84 , srv883954483:84 , srv905708676:84 , srv932140459:84 , srv952698178:84 , srv979803770:84 , srv1267939019:85 , srv1296951358:85 , srv1389582771:85 , srv140834990:85 , srv1497922081:85 , srv1574490732:85 , srv1597891922:85 , srv1749548852:85 , srv1808014848:85 , srv228091490:85 , srv606216029:85 , srv615630651:85 , srv656037803:85 , srv547884750:684 }
2023-07-29 03:53:05,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-29 03:53:05,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1267939019=14, srv1971984063=48, srv2094281484=51, srv684604166=84, srv6201317=80, srv1199936786=9, srv1787278202=41, srv1297777085=17, srv2143852088=56, srv979803770=99, srv1150942728=7, srv1336378072=19, srv474568463=71, srv498042253=72, srv1597891922=34, srv1855985468=43, srv615630651=79, srv2062379531=50, srv547884750=74, srv905708676=96, srv1512727052=28, srv606216029=78, srv554656408=75, srv266118838=60, srv1214072763=11, srv78690106=91, srv128051250=15, srv340125613=65, srv862554555=94, srv1492384964=26, srv2128614436=53, srv1202482966=10, srv417299113=70, srv1497922081=27, srv256205547=59, srv1772663016=39, srv2016842155=49, srv407145969=68, srv1446264657=23, srv783345958=90, srv1137123457=6, srv1108363646=5, srv725945976=87, srv1749548852=38, srv314276139=64, srv1808014848=42, srv1662669068=36, srv719043454=86, srv715083001=85, srv952698178=98, srv760016872=89, srv2128808754=54, srv852348850=93, srv634203905=81, srv283877757=63, srv1155120454=8, srv1593181514=32, srv814161843=92, srv215594311=57, srv757826745=88, srv1724467998=37, srv1574490732=30, srv140834990=22, srv407729221=69, srv1389582771=20, srv1879261399=45, srv1028696183=1, srv1069290571=3, srv932140459=97, srv883954483=95, srv228091490=58, srv1219650413=12, srv1863436427=44, srv398666017=67, srv580591755=76, srv124519287=13, srv1108263209=4, srv656037803=82, srv1405129998=21, srv1296951358=16, srv267017812=61, srv1965434304=47, srv1593330483=33, srv2126675970=52, srv192772278=46, srv1022107977=0, srv1588754029=31, srv17752822=40, srv6641767=83, srv268310214=62, srv148733486=25, srv343966749=66, srv592337239=77, srv2130114762=55, srv1627090091=35, srv1051061101=2, srv1516419535=29, srv535670786=73, srv147217393=24, srv1319247638=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-29 03:53:05,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-29 03:53:05,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-29 03:53:05,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-29 03:53:05,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-29 03:53:05,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-29 03:53:05,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-29 03:53:05,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-29 03:53:05,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-29 03:53:05,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-29 03:53:05,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-29 03:53:05,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-29 03:53:05,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-29 03:53:05,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-29 03:53:05,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-29 03:53:05,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-29 03:53:05,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-29 03:53:05,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-29 03:53:05,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-29 03:53:05,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-29 03:53:05,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-29 03:53:05,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-29 03:53:05,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-29 03:53:05,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-29 03:53:05,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-29 03:53:05,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-29 03:53:05,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-29 03:53:05,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-29 03:53:05,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-29 03:53:05,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-29 03:53:05,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-29 03:53:05,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-29 03:53:05,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-29 03:53:05,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-29 03:53:05,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-29 03:53:05,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-29 03:53:05,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-29 03:53:05,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-29 03:53:05,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-29 03:53:05,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-29 03:53:05,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-29 03:53:05,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-29 03:53:05,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-29 03:53:05,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-29 03:53:05,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-29 03:53:05,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-29 03:53:05,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-29 03:53:05,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-29 03:53:05,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-29 03:53:05,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-29 03:53:05,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-29 03:53:05,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-29 03:53:05,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-29 03:53:05,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-29 03:53:05,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-29 03:53:05,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-29 03:53:05,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-29 03:53:05,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-29 03:53:05,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-29 03:53:05,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-29 03:53:05,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-29 03:53:05,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-29 03:53:05,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-29 03:53:05,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-29 03:53:05,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-29 03:53:05,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-29 03:53:05,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-29 03:53:05,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-29 03:53:05,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-29 03:53:05,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-29 03:53:05,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-29 03:53:05,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-29 03:53:05,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-29 03:53:05,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-29 03:53:05,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-29 03:53:05,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-29 03:53:05,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-29 03:53:05,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-29 03:53:05,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-29 03:53:05,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-29 03:53:05,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-29 03:53:05,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-29 03:53:05,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-29 03:53:05,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-29 03:53:05,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-29 03:53:05,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-29 03:53:05,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-29 03:53:05,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-29 03:53:05,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-29 03:53:05,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-29 03:53:05,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-29 03:53:05,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-29 03:53:05,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-29 03:53:05,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-29 03:53:05,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-29 03:53:05,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-29 03:53:05,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-29 03:53:05,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-29 03:53:05,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-29 03:53:05,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-29 03:53:05,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-29 03:53:05,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-29 03:53:05,315 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.397985672269699E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-29 03:53:05,315 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1108263209:83 , srv1516419535:83 , srv1593181514:83 , srv1863436427:83 , srv2126675970:83 , srv314276139:83 , srv407729221:83 , srv592337239:83 , srv6201317:83 , srv715083001:83 , srv757826745:83 , srv760016872:83 , srv852348850:83 , srv1022107977:84 , srv1028696183:84 , srv1051061101:84 , srv1069290571:84 , srv1108363646:84 , srv1137123457:84 , srv1150942728:84 , srv1155120454:84 , srv1199936786:84 , srv1202482966:84 , srv1214072763:84 , srv1219650413:84 , srv124519287:84 , srv128051250:84 , srv1297777085:84 , srv1319247638:84 , srv1336378072:84 , srv1405129998:84 , srv1446264657:84 , srv147217393:84 , srv148733486:84 , srv1492384964:84 , srv1512727052:84 , srv1588754029:84 , srv1593330483:84 , srv1627090091:84 , srv1662669068:84 , srv1724467998:84 , srv1772663016:84 , srv17752822:84 , srv1787278202:84 , srv1855985468:84 , srv1879261399:84 , srv192772278:84 , srv1965434304:84 , srv1971984063:84 , srv2016842155:84 , srv2062379531:84 , srv2094281484:84 , srv2128614436:84 , srv2128808754:84 , srv2130114762:84 , srv2143852088:84 , srv215594311:84 , srv256205547:84 , srv266118838:84 , srv267017812:84 , srv268310214:84 , srv283877757:84 , srv340125613:84 , srv343966749:84 , srv398666017:84 , srv407145969:84 , srv417299113:84 , srv474568463:84 , srv498042253:84 , srv535670786:84 , srv554656408:84 , srv580591755:84 , srv634203905:84 , srv6641767:84 , srv684604166:84 , srv719043454:84 , srv725945976:84 , srv783345958:84 , srv78690106:84 , srv814161843:84 , srv862554555:84 , srv883954483:84 , srv905708676:84 , srv932140459:84 , srv952698178:84 , srv979803770:84 , srv1267939019:85 , srv1296951358:85 , srv1389582771:85 , srv140834990:85 , srv1497922081:85 , srv1574490732:85 , srv1597891922:85 , srv1749548852:85 , srv1808014848:85 , srv228091490:85 , srv606216029:85 , srv615630651:85 , srv656037803:85 , srv547884750:684 }
2023-07-29 03:53:05,339 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=451 (was 455), ProcessCount=169 (was 169), AvailableMemoryMB=5316 (was 5835)
2023-07-29 03:53:05,346 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=451, ProcessCount=169, AvailableMemoryMB=5315
2023-07-29 03:53:05,347 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-07-29 03:53:05,347 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-07-29 03:53:05,358 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1063597048:84 , srv1069317429:84 , srv1088632287:84 , srv1289124362:84 , srv1430494068:84 , srv1477766163:84 , srv1529598925:84 , srv1532041681:84 , srv1579478189:84 , srv1621589770:84 , srv1690292650:84 , srv1778946117:84 , srv1885273525:84 , srv1946919055:84 , srv2007268033:84 , srv2066109842:84 , srv2108026747:84 , srv2110609828:84 , srv269862141:84 , srv384092500:84 , srv464107795:84 , srv47943661:84 , srv50407712:84 , srv600350611:84 , srv764674704:84 , srv784751377:84 , srv80684648:84 , srv828131576:84 , srv939324396:84 , srv646834324:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-07-29 03:53:05,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-29 03:53:05,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1069317429=1, srv1885273525=12, srv1063597048=0, srv2110609828=17, srv1477766163=5, srv2066109842=15, srv80684648=27, srv1529598925=6, srv1532041681=7, srv47943661=21, srv764674704=25, srv828131576=28, srv1088632287=2, srv1690292650=10, srv1579478189=8, srv1778946117=11, srv269862141=18, srv784751377=26, srv384092500=19, srv1430494068=4, srv2108026747=16, srv646834324=24, srv50407712=22, srv464107795=20, srv939324396=29, srv1621589770=9, srv2007268033=14, srv1289124362=3, srv1946919055=13, srv600350611=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-29 03:53:05,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-29 03:53:05,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-29 03:53:05,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-29 03:53:05,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-29 03:53:05,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-29 03:53:05,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-29 03:53:05,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-29 03:53:05,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-29 03:53:05,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-29 03:53:05,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-29 03:53:05,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-29 03:53:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-29 03:53:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-29 03:53:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-29 03:53:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-29 03:53:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-29 03:53:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-29 03:53:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-29 03:53:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-29 03:53:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-29 03:53:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-29 03:53:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-29 03:53:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-29 03:53:05,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-29 03:53:05,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-29 03:53:05,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-29 03:53:05,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-29 03:53:05,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-29 03:53:05,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-29 03:53:05,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-29 03:53:05,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-07-29 03:53:05,398 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1201789649541283, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-29 03:53:05,399 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912571723258646, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1201789649541283, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-07-29 03:55:05,379 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9557853 different iterations.  Found a solution that moves 1843 regions; Going from a computed imbalance of 0.9912571723258646 to a new imbalance of 4.3016539410932495E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6825925925925926, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-29 03:55:05,385 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1063597048:90 , srv1069317429:90 , srv1088632287:90 , srv1289124362:90 , srv1430494068:90 , srv1477766163:90 , srv1529598925:90 , srv1532041681:90 , srv1579478189:90 , srv1621589770:90 , srv1690292650:90 , srv1778946117:90 , srv1885273525:90 , srv1946919055:90 , srv2007268033:90 , srv2066109842:90 , srv2108026747:90 , srv2110609828:90 , srv269862141:90 , srv384092500:90 , srv464107795:90 , srv47943661:90 , srv50407712:90 , srv600350611:90 , srv646834324:90 , srv764674704:90 , srv784751377:90 , srv80684648:90 , srv828131576:90 , srv939324396:90 }
2023-07-29 03:55:05,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-29 03:55:05,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1069317429=1, srv1885273525=12, srv1063597048=0, srv2110609828=17, srv1477766163=5, srv2066109842=15, srv80684648=27, srv1529598925=6, srv1532041681=7, srv47943661=21, srv764674704=25, srv828131576=28, srv1088632287=2, srv1690292650=10, srv1579478189=8, srv1778946117=11, srv269862141=18, srv784751377=26, srv384092500=19, srv1430494068=4, srv2108026747=16, srv646834324=24, srv50407712=22, srv464107795=20, srv939324396=29, srv1621589770=9, srv2007268033=14, srv1289124362=3, srv1946919055=13, srv600350611=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-29 03:55:05,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-29 03:55:05,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-29 03:55:05,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-29 03:55:05,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-29 03:55:05,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-29 03:55:05,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-29 03:55:05,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-29 03:55:05,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-29 03:55:05,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-29 03:55:05,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-29 03:55:05,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-29 03:55:05,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-29 03:55:05,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-29 03:55:05,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-29 03:55:05,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-29 03:55:05,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-29 03:55:05,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-29 03:55:05,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-29 03:55:05,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-29 03:55:05,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-29 03:55:05,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-29 03:55:05,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-29 03:55:05,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-29 03:55:05,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-29 03:55:05,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-29 03:55:05,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-29 03:55:05,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-29 03:55:05,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-29 03:55:05,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-29 03:55:05,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-29 03:55:05,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-07-29 03:55:05,401 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-29 03:55:05,401 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1063597048:90 , srv1069317429:90 , srv1088632287:90 , srv1289124362:90 , srv1430494068:90 , srv1477766163:90 , srv1529598925:90 , srv1532041681:90 , srv1579478189:90 , srv1621589770:90 , srv1690292650:90 , srv1778946117:90 , srv1885273525:90 , srv1946919055:90 , srv2007268033:90 , srv2066109842:90 , srv2108026747:90 , srv2110609828:90 , srv269862141:90 , srv384092500:90 , srv464107795:90 , srv47943661:90 , srv50407712:90 , srv600350611:90 , srv646834324:90 , srv764674704:90 , srv784751377:90 , srv80684648:90 , srv828131576:90 , srv939324396:90 }
2023-07-29 03:55:05,413 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=225 (was 451), ProcessCount=169 (was 169), AvailableMemoryMB=5352 (was 5315) - AvailableMemoryMB LEAK? -