2023-09-19 18:53:15,006 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-09-19 18:53:15,338 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-09-19 18:53:15,448 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-09-19 18:53:15,512 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-19 18:53:15,520 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-09-19 18:53:15,536 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=329, ProcessCount=170, AvailableMemoryMB=10873
2023-09-19 18:53:15,539 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-19 18:53:15,539 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-09-19 18:53:15,708 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1012033732:84 , srv1022984818:84 , srv1081491078:84 , srv1092280711:84 , srv1148888610:84 , srv1149106617:84 , srv1161070906:84 , srv1223353036:84 , srv1243375015:84 , srv1285566568:84 , srv1313575335:84 , srv1320387121:84 , srv1353016920:84 , srv1354946971:84 , srv1380128512:84 , srv1396387944:84 , srv1396841407:84 , srv1429280213:84 , srv1475985655:84 , srv1504945885:84 , srv1520886771:84 , srv1535291656:84 , srv1558686173:84 , srv1574281322:84 , srv1588568497:84 , srv1655603269:84 , srv1658581221:84 , srv1660784708:84 , srv1674264870:84 , srv1723541328:84 , srv173443327:84 , srv1769941960:84 , srv1777660793:84 , srv177804677:84 , srv1793317836:84 , srv1823051578:84 , srv1826601990:84 , srv1849677421:84 , srv1867372740:84 , srv1879344381:84 , srv1885975192:84 , srv1892817040:84 , srv1928593708:84 , srv1928983533:84 , srv1936818513:84 , srv1944278169:84 , srv1945233430:84 , srv2006647050:84 , srv2029682497:84 , srv2051670451:84 , srv2052562646:84 , srv2061256881:84 , srv2086360369:84 , srv2091465655:84 , srv2096359115:84 , srv211216482:84 , srv2118762326:84 , srv2147487:84 , srv227969079:84 , srv23807051:84 , srv246705155:84 , srv253508893:84 , srv255131969:84 , srv264292260:84 , srv266056583:84 , srv270314685:84 , srv308533524:84 , srv347816101:84 , srv364485066:84 , srv371877567:84 , srv390912166:84 , srv411175945:84 , srv420445910:84 , srv42341017:84 , srv480679644:84 , srv492990629:84 , srv522294035:84 , srv527733695:84 , srv556035015:84 , srv563084574:84 , srv573838363:84 , srv584745130:84 , srv607916192:84 , srv624669950:84 , srv674534534:84 , srv692150254:84 , srv708656159:84 , srv714729314:84 , srv72514527:84 , srv744507574:84 , srv791125553:84 , srv794783195:84 , srv818593136:84 , srv85230110:84 , srv868571020:84 , srv901538778:84 , srv926835027:84 , srv966868655:84 , srv978049915:84 , srv691112898:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-09-19 18:53:15,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-19 18:53:15,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2029682497=48, srv1161070906=6, srv1223353036=7, srv1504945885=19, srv1520886771=20, srv573838363=80, srv1655603269=25, srv1945233430=46, srv744507574=90, srv211216482=55, srv674534534=84, srv1148888610=4, srv692150254=86, srv420445910=72, srv1535291656=21, srv1380128512=14, srv1588568497=24, srv522294035=76, srv556035015=78, srv2118762326=56, srv364485066=68, srv390912166=70, srv72514527=89, srv2096359115=54, srv868571020=95, srv1769941960=31, srv1149106617=5, srv1823051578=35, srv1022984818=1, srv1285566568=9, srv1826601990=36, srv1660784708=27, srv1879344381=39, srv1396841407=16, srv371877567=69, srv1429280213=17, srv1081491078=2, srv563084574=79, srv255131969=62, srv347816101=67, srv1243375015=8, srv1777660793=32, srv177804677=33, srv1928983533=43, srv246705155=60, srv1674264870=28, srv2051670451=49, srv818593136=93, srv794783195=92, srv1353016920=12, srv1658581221=26, srv901538778=96, srv1396387944=15, srv2091465655=53, srv1092280711=3, srv1944278169=45, srv266056583=64, srv1723541328=29, srv1928593708=42, srv23807051=59, srv1313575335=10, srv1475985655=18, srv966868655=98, srv2061256881=51, srv1892817040=41, srv173443327=30, srv492990629=75, srv1574281322=23, srv2006647050=47, srv1354946971=13, srv2052562646=50, srv1793317836=34, srv480679644=74, srv2086360369=52, srv270314685=65, srv1936818513=44, srv691112898=85, srv264292260=63, srv624669950=83, srv926835027=97, srv1558686173=22, srv607916192=82, srv1867372740=38, srv411175945=71, srv791125553=91, srv85230110=94, srv227969079=58, srv978049915=99, srv1885975192=40, srv2147487=57, srv1012033732=0, srv714729314=88, srv527733695=77, srv253508893=61, srv308533524=66, srv1849677421=37, srv42341017=73, srv584745130=81, srv708656159=87, srv1320387121=11} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-19 18:53:15,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-19 18:53:15,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-19 18:53:15,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-19 18:53:15,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-19 18:53:15,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-19 18:53:15,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-19 18:53:15,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-19 18:53:15,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-19 18:53:15,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-19 18:53:15,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-19 18:53:15,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-19 18:53:15,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-19 18:53:15,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-19 18:53:15,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-19 18:53:15,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-19 18:53:15,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-19 18:53:15,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-19 18:53:15,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-19 18:53:15,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-19 18:53:15,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-19 18:53:15,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-19 18:53:15,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-19 18:53:15,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-19 18:53:15,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-19 18:53:15,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-19 18:53:15,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-19 18:53:15,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-19 18:53:15,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-19 18:53:15,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-19 18:53:15,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-19 18:53:15,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-19 18:53:15,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-19 18:53:15,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-19 18:53:15,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-19 18:53:15,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-19 18:53:15,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-19 18:53:15,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-19 18:53:15,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-19 18:53:15,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-19 18:53:15,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-19 18:53:15,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-19 18:53:15,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-19 18:53:15,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-19 18:53:15,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-19 18:53:15,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-19 18:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-19 18:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-19 18:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-19 18:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-19 18:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-19 18:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-19 18:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-19 18:53:15,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-19 18:53:15,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-19 18:53:15,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-19 18:53:15,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-19 18:53:15,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-19 18:53:15,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-19 18:53:15,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-19 18:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-19 18:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-19 18:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-19 18:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-19 18:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-19 18:53:15,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-19 18:53:15,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-19 18:53:15,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-19 18:53:15,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-19 18:53:15,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-19 18:53:15,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-19 18:53:15,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-19 18:53:15,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-19 18:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-19 18:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-19 18:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-19 18:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-19 18:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-19 18:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-19 18:53:15,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-19 18:53:15,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-19 18:53:15,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-19 18:53:15,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-19 18:53:15,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-19 18:53:15,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-19 18:53:15,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-19 18:53:15,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-19 18:53:15,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-19 18:53:15,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-19 18:53:15,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-19 18:53:15,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-19 18:53:15,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-19 18:53:15,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-19 18:53:15,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-19 18:53:15,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-19 18:53:15,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-19 18:53:15,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-19 18:53:15,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-19 18:53:15,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-19 18:53:15,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-19 18:53:15,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-19 18:53:15,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-19 18:53:16,112 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-19 18:53:16,113 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-19 18:53:16,113 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-19 18:53:16,512 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 640 ms to try 5000 different iterations.  Found a solution that moves 4695 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1791175570897017. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039379461496813595, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055575753903998056, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5216666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039379461496813595, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24266666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-19 18:53:16,524 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1081491078:81 , srv1223353036:82 , srv1879344381:82 , srv411175945:82 , srv674534534:82 , srv1396841407:83 , srv2086360369:83 , srv624669950:83 , srv818593136:83 , srv1149106617:84 , srv1588568497:84 , srv1660784708:84 , srv1793317836:84 , srv2006647050:84 , srv255131969:84 , srv264292260:84 , srv556035015:84 , srv714729314:84 , srv744507574:84 , srv1161070906:85 , srv1243375015:85 , srv1574281322:85 , srv1769941960:85 , srv1849677421:85 , srv2052562646:85 , srv211216482:85 , srv2147487:85 , srv364485066:85 , srv371877567:85 , srv563084574:85 , srv573838363:85 , srv584745130:85 , srv692150254:85 , srv72514527:85 , srv1429280213:86 , srv1504945885:86 , srv1520886771:86 , srv1658581221:86 , srv1674264870:86 , srv1723541328:86 , srv1777660793:86 , srv1867372740:86 , srv1928983533:86 , srv1936818513:86 , srv1945233430:86 , srv2096359115:86 , srv23807051:86 , srv308533524:86 , srv390912166:86 , srv420445910:86 , srv480679644:86 , srv607916192:86 , srv708656159:86 , srv794783195:86 , srv978049915:86 , srv1092280711:87 , srv1313575335:87 , srv1353016920:87 , srv1380128512:87 , srv1558686173:87 , srv1655603269:87 , srv177804677:87 , srv1823051578:87 , srv1928593708:87 , srv2029682497:87 , srv2061256881:87 , srv266056583:87 , srv270314685:87 , srv522294035:87 , srv527733695:87 , srv85230110:87 , srv901538778:87 , srv1012033732:88 , srv1396387944:88 , srv1535291656:88 , srv173443327:88 , srv1826601990:88 , srv1944278169:88 , srv2118762326:88 , srv492990629:88 , srv926835027:88 , srv1022984818:89 , srv1285566568:89 , srv1354946971:89 , srv1475985655:89 , srv246705155:89 , srv253508893:89 , srv347816101:89 , srv868571020:89 , srv966868655:89 , srv1320387121:90 , srv1885975192:90 , srv2091465655:90 , srv227969079:91 , srv1148888610:92 , srv1892817040:92 , srv42341017:93 , srv791125553:93 , srv2051670451:95 , srv691112898:440 }
2023-09-19 18:53:16,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-19 18:53:16,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2029682497=48, srv1161070906=6, srv1223353036=7, srv1504945885=19, srv1520886771=20, srv573838363=80, srv1655603269=25, srv1945233430=46, srv744507574=90, srv211216482=55, srv674534534=84, srv1148888610=4, srv692150254=86, srv420445910=72, srv1535291656=21, srv1380128512=14, srv1588568497=24, srv522294035=76, srv556035015=78, srv2118762326=56, srv364485066=68, srv390912166=70, srv72514527=89, srv2096359115=54, srv868571020=95, srv1769941960=31, srv1149106617=5, srv1823051578=35, srv1022984818=1, srv1285566568=9, srv1826601990=36, srv1660784708=27, srv1879344381=39, srv1396841407=16, srv371877567=69, srv1429280213=17, srv1081491078=2, srv563084574=79, srv255131969=62, srv347816101=67, srv1243375015=8, srv1777660793=32, srv177804677=33, srv1928983533=43, srv246705155=60, srv1674264870=28, srv2051670451=49, srv818593136=93, srv794783195=92, srv1353016920=12, srv1658581221=26, srv901538778=96, srv1396387944=15, srv2091465655=53, srv1092280711=3, srv1944278169=45, srv266056583=64, srv1723541328=29, srv1928593708=42, srv23807051=59, srv1313575335=10, srv1475985655=18, srv966868655=98, srv2061256881=51, srv1892817040=41, srv173443327=30, srv492990629=75, srv1574281322=23, srv2006647050=47, srv1354946971=13, srv2052562646=50, srv1793317836=34, srv480679644=74, srv2086360369=52, srv270314685=65, srv1936818513=44, srv691112898=85, srv264292260=63, srv624669950=83, srv926835027=97, srv1558686173=22, srv607916192=82, srv1867372740=38, srv411175945=71, srv791125553=91, srv85230110=94, srv227969079=58, srv978049915=99, srv1885975192=40, srv2147487=57, srv1012033732=0, srv714729314=88, srv527733695=77, srv253508893=61, srv308533524=66, srv1849677421=37, srv42341017=73, srv584745130=81, srv708656159=87, srv1320387121=11} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-19 18:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-19 18:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-19 18:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-19 18:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-19 18:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-19 18:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-19 18:53:16,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-19 18:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-19 18:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-19 18:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-19 18:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-19 18:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-19 18:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-19 18:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-19 18:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-19 18:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-19 18:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-19 18:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-19 18:53:16,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-19 18:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-19 18:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-19 18:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-19 18:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-19 18:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-19 18:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-19 18:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-19 18:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-19 18:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-19 18:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-19 18:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-19 18:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-19 18:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-19 18:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-19 18:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-19 18:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-19 18:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-19 18:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-19 18:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-19 18:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-19 18:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-19 18:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-19 18:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-19 18:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-19 18:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-19 18:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-19 18:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-19 18:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-19 18:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-19 18:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-19 18:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-19 18:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-19 18:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-19 18:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-19 18:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-19 18:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-19 18:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-19 18:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-19 18:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-19 18:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-19 18:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-19 18:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-19 18:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-19 18:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-19 18:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-19 18:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-19 18:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-19 18:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-19 18:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-19 18:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-19 18:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-19 18:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-19 18:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-19 18:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-19 18:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-19 18:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-19 18:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-19 18:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-19 18:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-19 18:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-19 18:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-19 18:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-19 18:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-19 18:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-19 18:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-19 18:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-19 18:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-19 18:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-19 18:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-19 18:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-19 18:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-19 18:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-19 18:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-19 18:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-19 18:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-19 18:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-19 18:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-19 18:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-19 18:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-19 18:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-19 18:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-19 18:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-19 18:53:16,739 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039379461496813595, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055575753903998056, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039379461496813595, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24266666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-19 18:53:16,739 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-19 18:53:16,740 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17908468199704827, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039379461496813595, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055575753903998056, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039379461496813595, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24266666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-19 18:53:17,109 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 519 ms to try 5000 different iterations.  Found a solution that moves 2663 regions; Going from a computed imbalance of 0.17908468199704827 to a new imbalance of 0.014921689984732539. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021821299191187506); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03242188228925722, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29588888888888887, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021821299191187506); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01175); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04525, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-19 18:53:17,123 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1892817040:78 , srv2051670451:78 , srv1148888610:80 , srv2091465655:80 , srv227969079:80 , srv791125553:80 , srv1885975192:81 , srv42341017:81 , srv1320387121:82 , srv1475985655:83 , srv1674264870:83 , srv527733695:83 , srv868571020:83 , srv901538778:83 , srv926835027:83 , srv1092280711:84 , srv1354946971:84 , srv1380128512:84 , srv1535291656:84 , srv1558686173:84 , srv1944278169:84 , srv2029682497:84 , srv2061256881:84 , srv2096359115:84 , srv246705155:84 , srv253508893:84 , srv266056583:84 , srv270314685:84 , srv492990629:84 , srv692150254:84 , srv966868655:84 , srv1012033732:85 , srv1022984818:85 , srv1285566568:85 , srv1520886771:85 , srv1655603269:85 , srv1723541328:85 , srv1777660793:85 , srv177804677:85 , srv1826601990:85 , srv1936818513:85 , srv211216482:85 , srv2118762326:85 , srv23807051:85 , srv347816101:85 , srv72514527:85 , srv85230110:85 , srv978049915:85 , srv1243375015:86 , srv1313575335:86 , srv1396387944:86 , srv1504945885:86 , srv1588568497:86 , srv173443327:86 , srv1823051578:86 , srv1867372740:86 , srv1928983533:86 , srv1945233430:86 , srv2006647050:86 , srv255131969:86 , srv308533524:86 , srv420445910:86 , srv522294035:86 , srv563084574:86 , srv584745130:86 , srv1161070906:87 , srv1353016920:87 , srv1429280213:87 , srv1574281322:87 , srv1928593708:87 , srv2052562646:87 , srv2147487:87 , srv264292260:87 , srv364485066:87 , srv371877567:87 , srv390912166:87 , srv708656159:87 , srv1149106617:88 , srv1396841407:88 , srv1658581221:88 , srv1660784708:88 , srv1769941960:88 , srv1793317836:88 , srv1849677421:88 , srv2086360369:88 , srv480679644:88 , srv573838363:88 , srv607916192:88 , srv1223353036:89 , srv556035015:89 , srv794783195:89 , srv818593136:89 , srv624669950:90 , srv674534534:90 , srv714729314:90 , srv744507574:90 , srv1879344381:91 , srv411175945:91 , srv1081491078:93 , srv691112898:528 }
2023-09-19 18:53:17,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-19 18:53:17,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2029682497=48, srv1161070906=6, srv1223353036=7, srv1504945885=19, srv1520886771=20, srv573838363=80, srv1655603269=25, srv1945233430=46, srv744507574=90, srv211216482=55, srv674534534=84, srv1148888610=4, srv692150254=86, srv420445910=72, srv1535291656=21, srv1380128512=14, srv1588568497=24, srv522294035=76, srv556035015=78, srv2118762326=56, srv364485066=68, srv390912166=70, srv72514527=89, srv2096359115=54, srv868571020=95, srv1769941960=31, srv1149106617=5, srv1823051578=35, srv1022984818=1, srv1285566568=9, srv1826601990=36, srv1660784708=27, srv1879344381=39, srv1396841407=16, srv371877567=69, srv1429280213=17, srv1081491078=2, srv563084574=79, srv255131969=62, srv347816101=67, srv1243375015=8, srv1777660793=32, srv177804677=33, srv1928983533=43, srv246705155=60, srv1674264870=28, srv2051670451=49, srv818593136=93, srv794783195=92, srv1353016920=12, srv1658581221=26, srv901538778=96, srv1396387944=15, srv2091465655=53, srv1092280711=3, srv1944278169=45, srv266056583=64, srv1723541328=29, srv1928593708=42, srv23807051=59, srv1313575335=10, srv1475985655=18, srv966868655=98, srv2061256881=51, srv1892817040=41, srv173443327=30, srv492990629=75, srv1574281322=23, srv2006647050=47, srv1354946971=13, srv2052562646=50, srv1793317836=34, srv480679644=74, srv2086360369=52, srv270314685=65, srv1936818513=44, srv691112898=85, srv264292260=63, srv624669950=83, srv926835027=97, srv1558686173=22, srv607916192=82, srv1867372740=38, srv411175945=71, srv791125553=91, srv85230110=94, srv227969079=58, srv978049915=99, srv1885975192=40, srv2147487=57, srv1012033732=0, srv714729314=88, srv527733695=77, srv253508893=61, srv308533524=66, srv1849677421=37, srv42341017=73, srv584745130=81, srv708656159=87, srv1320387121=11} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-19 18:53:17,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-19 18:53:17,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-19 18:53:17,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-19 18:53:17,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-19 18:53:17,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-19 18:53:17,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-19 18:53:17,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-19 18:53:17,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-19 18:53:17,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-19 18:53:17,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-19 18:53:17,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-19 18:53:17,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-19 18:53:17,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-19 18:53:17,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-19 18:53:17,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-19 18:53:17,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-19 18:53:17,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-19 18:53:17,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-19 18:53:17,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-19 18:53:17,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-19 18:53:17,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-19 18:53:17,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-19 18:53:17,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-19 18:53:17,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-19 18:53:17,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-19 18:53:17,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-19 18:53:17,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-19 18:53:17,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-19 18:53:17,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-19 18:53:17,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-19 18:53:17,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-19 18:53:17,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-19 18:53:17,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-19 18:53:17,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-19 18:53:17,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-19 18:53:17,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-19 18:53:17,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-19 18:53:17,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-19 18:53:17,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-19 18:53:17,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-19 18:53:17,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-19 18:53:17,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-19 18:53:17,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-19 18:53:17,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-19 18:53:17,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-19 18:53:17,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-19 18:53:17,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-19 18:53:17,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-19 18:53:17,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-19 18:53:17,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-19 18:53:17,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-19 18:53:17,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-19 18:53:17,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-19 18:53:17,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-19 18:53:17,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-19 18:53:17,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-19 18:53:17,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-19 18:53:17,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-19 18:53:17,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-19 18:53:17,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-19 18:53:17,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-19 18:53:17,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-19 18:53:17,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-19 18:53:17,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-19 18:53:17,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-19 18:53:17,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-19 18:53:17,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-19 18:53:17,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-19 18:53:17,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-19 18:53:17,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-19 18:53:17,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-19 18:53:17,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-19 18:53:17,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-19 18:53:17,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-19 18:53:17,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-19 18:53:17,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-19 18:53:17,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-19 18:53:17,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-19 18:53:17,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-19 18:53:17,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-19 18:53:17,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-19 18:53:17,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-19 18:53:17,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-19 18:53:17,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-19 18:53:17,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-19 18:53:17,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-19 18:53:17,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-19 18:53:17,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-19 18:53:17,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-19 18:53:17,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-19 18:53:17,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-19 18:53:17,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-19 18:53:17,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-19 18:53:17,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-19 18:53:17,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-19 18:53:17,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-19 18:53:17,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-19 18:53:17,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-19 18:53:17,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-19 18:53:17,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-19 18:53:17,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-19 18:53:17,333 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021821299191187506); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03242188228925722, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021821299191187506); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01175); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04525, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-19 18:53:17,334 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-19 18:53:17,334 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014903043260188103, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021821299191187506); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03242188228925722, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021821299191187506); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01175); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04525, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-19 18:53:17,713 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 517 ms to try 5000 different iterations.  Found a solution that moves 965 regions; Going from a computed imbalance of 0.014903043260188103 to a new imbalance of 6.418866329816614E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0067834882385377025); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008838239537061558); MoveCostFunction : (multiplier=7.0, imbalance=0.10722222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0067834882385377025); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-19 18:53:17,716 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv347816101:82 , srv1022984818:83 , srv1081491078:83 , srv1285566568:83 , srv1353016920:83 , srv42341017:83 , srv480679644:83 , srv714729314:83 , srv794783195:83 , srv1012033732:84 , srv1313575335:84 , srv1354946971:84 , srv1396387944:84 , srv1429280213:84 , srv173443327:84 , srv1769941960:84 , srv177804677:84 , srv1826601990:84 , srv1928593708:84 , srv2051670451:84 , srv246705155:84 , srv253508893:84 , srv522294035:84 , srv556035015:84 , srv573838363:84 , srv624669950:84 , srv708656159:84 , srv744507574:84 , srv1148888610:85 , srv1149106617:85 , srv1320387121:85 , srv1475985655:85 , srv1504945885:85 , srv1535291656:85 , srv1655603269:85 , srv1658581221:85 , srv1793317836:85 , srv1823051578:85 , srv1879344381:85 , srv1928983533:85 , srv1944278169:85 , srv1945233430:85 , srv2052562646:85 , srv2118762326:85 , srv2147487:85 , srv266056583:85 , srv308533524:85 , srv364485066:85 , srv390912166:85 , srv411175945:85 , srv420445910:85 , srv607916192:85 , srv674534534:85 , srv818593136:85 , srv85230110:85 , srv868571020:85 , srv966868655:85 , srv978049915:85 , srv1092280711:86 , srv1161070906:86 , srv1243375015:86 , srv1396841407:86 , srv1520886771:86 , srv1558686173:86 , srv1574281322:86 , srv1660784708:86 , srv1723541328:86 , srv1777660793:86 , srv1849677421:86 , srv1867372740:86 , srv1885975192:86 , srv2029682497:86 , srv371877567:86 , srv563084574:86 , srv901538778:86 , srv1380128512:87 , srv1588568497:87 , srv1892817040:87 , srv1936818513:87 , srv2006647050:87 , srv2061256881:87 , srv2086360369:87 , srv2096359115:87 , srv227969079:87 , srv23807051:87 , srv264292260:87 , srv270314685:87 , srv492990629:87 , srv527733695:87 , srv584745130:87 , srv791125553:87 , srv926835027:87 , srv1674264870:88 , srv2091465655:88 , srv211216482:88 , srv255131969:88 , srv692150254:88 , srv72514527:88 , srv1223353036:89 , srv691112898:550 }
2023-09-19 18:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-19 18:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2029682497=48, srv1161070906=6, srv1223353036=7, srv1504945885=19, srv1520886771=20, srv573838363=80, srv1655603269=25, srv1945233430=46, srv744507574=90, srv211216482=55, srv674534534=84, srv1148888610=4, srv692150254=86, srv420445910=72, srv1535291656=21, srv1380128512=14, srv1588568497=24, srv522294035=76, srv556035015=78, srv2118762326=56, srv364485066=68, srv390912166=70, srv72514527=89, srv2096359115=54, srv868571020=95, srv1769941960=31, srv1149106617=5, srv1823051578=35, srv1022984818=1, srv1285566568=9, srv1826601990=36, srv1660784708=27, srv1879344381=39, srv1396841407=16, srv371877567=69, srv1429280213=17, srv1081491078=2, srv563084574=79, srv255131969=62, srv347816101=67, srv1243375015=8, srv1777660793=32, srv177804677=33, srv1928983533=43, srv246705155=60, srv1674264870=28, srv2051670451=49, srv818593136=93, srv794783195=92, srv1353016920=12, srv1658581221=26, srv901538778=96, srv1396387944=15, srv2091465655=53, srv1092280711=3, srv1944278169=45, srv266056583=64, srv1723541328=29, srv1928593708=42, srv23807051=59, srv1313575335=10, srv1475985655=18, srv966868655=98, srv2061256881=51, srv1892817040=41, srv173443327=30, srv492990629=75, srv1574281322=23, srv2006647050=47, srv1354946971=13, srv2052562646=50, srv1793317836=34, srv480679644=74, srv2086360369=52, srv270314685=65, srv1936818513=44, srv691112898=85, srv264292260=63, srv624669950=83, srv926835027=97, srv1558686173=22, srv607916192=82, srv1867372740=38, srv411175945=71, srv791125553=91, srv85230110=94, srv227969079=58, srv978049915=99, srv1885975192=40, srv2147487=57, srv1012033732=0, srv714729314=88, srv527733695=77, srv253508893=61, srv308533524=66, srv1849677421=37, srv42341017=73, srv584745130=81, srv708656159=87, srv1320387121=11} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-19 18:53:17,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-19 18:53:17,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-19 18:53:17,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-19 18:53:17,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-19 18:53:17,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-19 18:53:17,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-19 18:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-19 18:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-19 18:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-19 18:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-19 18:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-19 18:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-19 18:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-19 18:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-19 18:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-19 18:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-19 18:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-19 18:53:17,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-19 18:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-19 18:53:17,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-19 18:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-19 18:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-19 18:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-19 18:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-19 18:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-19 18:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-19 18:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-19 18:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-19 18:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-19 18:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-19 18:53:17,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-19 18:53:17,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-19 18:53:17,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-19 18:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-19 18:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-19 18:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-19 18:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-19 18:53:17,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-19 18:53:17,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-19 18:53:17,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-19 18:53:17,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-19 18:53:17,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-19 18:53:17,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-19 18:53:17,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-19 18:53:17,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-19 18:53:17,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-19 18:53:17,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-19 18:53:17,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-19 18:53:17,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-19 18:53:17,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-19 18:53:17,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-19 18:53:17,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-19 18:53:17,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-19 18:53:17,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-19 18:53:17,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-19 18:53:17,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-19 18:53:17,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-19 18:53:17,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-19 18:53:17,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-19 18:53:17,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-19 18:53:17,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-19 18:53:17,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-19 18:53:17,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-19 18:53:17,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-19 18:53:17,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-19 18:53:17,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-19 18:53:17,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-19 18:53:17,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-19 18:53:17,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-19 18:53:17,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-19 18:53:17,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-19 18:53:17,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-19 18:53:17,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-19 18:53:17,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-19 18:53:17,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-19 18:53:17,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-19 18:53:17,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-19 18:53:17,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-19 18:53:17,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-19 18:53:17,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-19 18:53:17,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-19 18:53:17,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-19 18:53:17,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-19 18:53:17,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-19 18:53:17,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-19 18:53:17,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-19 18:53:17,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-19 18:53:17,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-19 18:53:17,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-19 18:53:17,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-19 18:53:17,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-19 18:53:17,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-19 18:53:17,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-19 18:53:17,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-19 18:53:17,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-19 18:53:17,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-19 18:53:17,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-19 18:53:17,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-19 18:53:17,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-19 18:53:17,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-19 18:53:17,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-19 18:53:17,940 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0067834882385377025); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008838239537061558); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0067834882385377025); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-19 18:53:17,941 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-19 18:53:17,943 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.351295585598139E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0067834882385377025); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008838239537061558); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0067834882385377025); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-19 18:53:18,333 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 532 ms to try 5000 different iterations.  Found a solution that moves 282 regions; Going from a computed imbalance of 6.351295585598139E-4 to a new imbalance of 2.2330160892854874E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023210354127426377); MoveCostFunction : (multiplier=7.0, imbalance=0.03133333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-19 18:53:18,334 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1223353036:83 , srv1658581221:83 , srv1849677421:83 , srv390912166:83 , srv492990629:83 , srv607916192:83 , srv791125553:83 , srv926835027:83 , srv1161070906:84 , srv1380128512:84 , srv1396387944:84 , srv1429280213:84 , srv1535291656:84 , srv1588568497:84 , srv1660784708:84 , srv173443327:84 , srv1793317836:84 , srv1823051578:84 , srv1867372740:84 , srv1892817040:84 , srv1928593708:84 , srv1936818513:84 , srv2061256881:84 , srv2086360369:84 , srv2091465655:84 , srv211216482:84 , srv227969079:84 , srv23807051:84 , srv255131969:84 , srv264292260:84 , srv270314685:84 , srv371877567:84 , srv420445910:84 , srv584745130:84 , srv72514527:84 , srv744507574:84 , srv966868655:84 , srv1022984818:85 , srv1081491078:85 , srv1092280711:85 , srv1148888610:85 , srv1149106617:85 , srv1243375015:85 , srv1285566568:85 , srv1313575335:85 , srv1320387121:85 , srv1353016920:85 , srv1354946971:85 , srv1396841407:85 , srv1475985655:85 , srv1504945885:85 , srv1520886771:85 , srv1558686173:85 , srv1574281322:85 , srv1655603269:85 , srv1674264870:85 , srv1723541328:85 , srv1769941960:85 , srv1777660793:85 , srv177804677:85 , srv1826601990:85 , srv1879344381:85 , srv1885975192:85 , srv1928983533:85 , srv1944278169:85 , srv1945233430:85 , srv2006647050:85 , srv2029682497:85 , srv2051670451:85 , srv2052562646:85 , srv2096359115:85 , srv2118762326:85 , srv2147487:85 , srv246705155:85 , srv253508893:85 , srv308533524:85 , srv347816101:85 , srv364485066:85 , srv411175945:85 , srv42341017:85 , srv480679644:85 , srv522294035:85 , srv527733695:85 , srv556035015:85 , srv573838363:85 , srv624669950:85 , srv674534534:85 , srv692150254:85 , srv708656159:85 , srv714729314:85 , srv794783195:85 , srv818593136:85 , srv85230110:85 , srv868571020:85 , srv1012033732:86 , srv266056583:86 , srv563084574:86 , srv901538778:86 , srv978049915:87 , srv691112898:624 }
2023-09-19 18:53:18,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-19 18:53:18,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2029682497=48, srv1161070906=6, srv1223353036=7, srv1504945885=19, srv1520886771=20, srv573838363=80, srv1655603269=25, srv1945233430=46, srv744507574=90, srv211216482=55, srv674534534=84, srv1148888610=4, srv692150254=86, srv420445910=72, srv1535291656=21, srv1380128512=14, srv1588568497=24, srv522294035=76, srv556035015=78, srv2118762326=56, srv364485066=68, srv390912166=70, srv72514527=89, srv2096359115=54, srv868571020=95, srv1769941960=31, srv1149106617=5, srv1823051578=35, srv1022984818=1, srv1285566568=9, srv1826601990=36, srv1660784708=27, srv1879344381=39, srv1396841407=16, srv371877567=69, srv1429280213=17, srv1081491078=2, srv563084574=79, srv255131969=62, srv347816101=67, srv1243375015=8, srv1777660793=32, srv177804677=33, srv1928983533=43, srv246705155=60, srv1674264870=28, srv2051670451=49, srv818593136=93, srv794783195=92, srv1353016920=12, srv1658581221=26, srv901538778=96, srv1396387944=15, srv2091465655=53, srv1092280711=3, srv1944278169=45, srv266056583=64, srv1723541328=29, srv1928593708=42, srv23807051=59, srv1313575335=10, srv1475985655=18, srv966868655=98, srv2061256881=51, srv1892817040=41, srv173443327=30, srv492990629=75, srv1574281322=23, srv2006647050=47, srv1354946971=13, srv2052562646=50, srv1793317836=34, srv480679644=74, srv2086360369=52, srv270314685=65, srv1936818513=44, srv691112898=85, srv264292260=63, srv624669950=83, srv926835027=97, srv1558686173=22, srv607916192=82, srv1867372740=38, srv411175945=71, srv791125553=91, srv85230110=94, srv227969079=58, srv978049915=99, srv1885975192=40, srv2147487=57, srv1012033732=0, srv714729314=88, srv527733695=77, srv253508893=61, srv308533524=66, srv1849677421=37, srv42341017=73, srv584745130=81, srv708656159=87, srv1320387121=11} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-19 18:53:18,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-19 18:53:18,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-19 18:53:18,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-19 18:53:18,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-19 18:53:18,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-19 18:53:18,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-19 18:53:18,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-19 18:53:18,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-19 18:53:18,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-19 18:53:18,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-19 18:53:18,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-19 18:53:18,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-19 18:53:18,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-19 18:53:18,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-19 18:53:18,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-19 18:53:18,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-19 18:53:18,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-19 18:53:18,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-19 18:53:18,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-19 18:53:18,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-19 18:53:18,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-19 18:53:18,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-19 18:53:18,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-19 18:53:18,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-19 18:53:18,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-19 18:53:18,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-19 18:53:18,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-19 18:53:18,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-19 18:53:18,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-19 18:53:18,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-19 18:53:18,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-19 18:53:18,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-19 18:53:18,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-19 18:53:18,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-19 18:53:18,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-19 18:53:18,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-19 18:53:18,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-19 18:53:18,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-19 18:53:18,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-19 18:53:18,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-19 18:53:18,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-19 18:53:18,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-19 18:53:18,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-19 18:53:18,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-19 18:53:18,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-19 18:53:18,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-19 18:53:18,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-19 18:53:18,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-19 18:53:18,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-19 18:53:18,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-19 18:53:18,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-19 18:53:18,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-19 18:53:18,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-19 18:53:18,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-19 18:53:18,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-19 18:53:18,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-19 18:53:18,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-19 18:53:18,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-19 18:53:18,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-19 18:53:18,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-19 18:53:18,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-19 18:53:18,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-19 18:53:18,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-19 18:53:18,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-19 18:53:18,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-19 18:53:18,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-19 18:53:18,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-19 18:53:18,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-19 18:53:18,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-19 18:53:18,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-19 18:53:18,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-19 18:53:18,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-19 18:53:18,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-19 18:53:18,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-19 18:53:18,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-19 18:53:18,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-19 18:53:18,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-19 18:53:18,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-19 18:53:18,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-19 18:53:18,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-19 18:53:18,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-19 18:53:18,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-19 18:53:18,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-19 18:53:18,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-19 18:53:18,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-19 18:53:18,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-19 18:53:18,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-19 18:53:18,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-19 18:53:18,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-19 18:53:18,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-19 18:53:18,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-19 18:53:18,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-19 18:53:18,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-19 18:53:18,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-19 18:53:18,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-19 18:53:18,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-19 18:53:18,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-19 18:53:18,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-19 18:53:18,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-19 18:53:18,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-19 18:53:18,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-19 18:53:18,477 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=2.035555468875021E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023210354127426377); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-19 18:53:18,477 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1223353036:83 , srv1658581221:83 , srv1849677421:83 , srv390912166:83 , srv492990629:83 , srv607916192:83 , srv791125553:83 , srv926835027:83 , srv1161070906:84 , srv1380128512:84 , srv1396387944:84 , srv1429280213:84 , srv1535291656:84 , srv1588568497:84 , srv1660784708:84 , srv173443327:84 , srv1793317836:84 , srv1823051578:84 , srv1867372740:84 , srv1892817040:84 , srv1928593708:84 , srv1936818513:84 , srv2061256881:84 , srv2086360369:84 , srv2091465655:84 , srv211216482:84 , srv227969079:84 , srv23807051:84 , srv255131969:84 , srv264292260:84 , srv270314685:84 , srv371877567:84 , srv420445910:84 , srv584745130:84 , srv72514527:84 , srv744507574:84 , srv966868655:84 , srv1022984818:85 , srv1081491078:85 , srv1092280711:85 , srv1148888610:85 , srv1149106617:85 , srv1243375015:85 , srv1285566568:85 , srv1313575335:85 , srv1320387121:85 , srv1353016920:85 , srv1354946971:85 , srv1396841407:85 , srv1475985655:85 , srv1504945885:85 , srv1520886771:85 , srv1558686173:85 , srv1574281322:85 , srv1655603269:85 , srv1674264870:85 , srv1723541328:85 , srv1769941960:85 , srv1777660793:85 , srv177804677:85 , srv1826601990:85 , srv1879344381:85 , srv1885975192:85 , srv1928983533:85 , srv1944278169:85 , srv1945233430:85 , srv2006647050:85 , srv2029682497:85 , srv2051670451:85 , srv2052562646:85 , srv2096359115:85 , srv2118762326:85 , srv2147487:85 , srv246705155:85 , srv253508893:85 , srv308533524:85 , srv347816101:85 , srv364485066:85 , srv411175945:85 , srv42341017:85 , srv480679644:85 , srv522294035:85 , srv527733695:85 , srv556035015:85 , srv573838363:85 , srv624669950:85 , srv674534534:85 , srv692150254:85 , srv708656159:85 , srv714729314:85 , srv794783195:85 , srv818593136:85 , srv85230110:85 , srv868571020:85 , srv1012033732:86 , srv266056583:86 , srv563084574:86 , srv901538778:86 , srv978049915:87 , srv691112898:624 }
2023-09-19 18:53:18,506 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=329 (was 329), ProcessCount=170 (was 170), AvailableMemoryMB=10240 (was 10873)
2023-09-19 18:53:18,516 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=329, ProcessCount=170, AvailableMemoryMB=10240
2023-09-19 18:53:18,517 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-19 18:53:18,517 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-09-19 18:53:18,526 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1060610781:84 , srv1129437566:84 , srv1158017689:84 , srv1239739004:84 , srv1341267710:84 , srv1552762508:84 , srv1658903039:84 , srv168712784:84 , srv1709806371:84 , srv1718145802:84 , srv1759441941:84 , srv1898021845:84 , srv1909113756:84 , srv2023326476:84 , srv231747407:84 , srv278282874:84 , srv306976778:84 , srv356815753:84 , srv503200429:84 , srv513668860:84 , srv515761900:84 , srv632735461:84 , srv633876524:84 , srv690535392:84 , srv78305533:84 , srv792291054:84 , srv833382154:84 , srv889001803:84 , srv995069160:84 , srv1593411614:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-09-19 18:53:18,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-19 18:53:18,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1658903039=7, srv1239739004=3, srv78305533=25, srv1898021845=12, srv515761900=21, srv1759441941=11, srv995069160=29, srv1060610781=0, srv1129437566=1, srv1709806371=9, srv1593411614=6, srv356815753=18, srv1909113756=13, srv1341267710=4, srv690535392=24, srv513668860=20, srv833382154=27, srv1158017689=2, srv231747407=15, srv1718145802=10, srv503200429=19, srv792291054=26, srv633876524=23, srv306976778=17, srv889001803=28, srv168712784=8, srv632735461=22, srv2023326476=14, srv278282874=16, srv1552762508=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-19 18:53:18,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-19 18:53:18,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-19 18:53:18,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-19 18:53:18,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-19 18:53:18,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-19 18:53:18,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-19 18:53:18,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-19 18:53:18,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-19 18:53:18,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-19 18:53:18,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-19 18:53:18,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-19 18:53:18,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-19 18:53:18,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-19 18:53:18,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-19 18:53:18,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-19 18:53:18,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-19 18:53:18,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-19 18:53:18,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-19 18:53:18,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-19 18:53:18,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-19 18:53:18,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-19 18:53:18,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-19 18:53:18,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-19 18:53:18,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-19 18:53:18,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-19 18:53:18,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-19 18:53:18,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-19 18:53:18,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-19 18:53:18,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-19 18:53:18,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-19 18:53:18,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-09-19 18:53:18,553 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1222872639712458, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-19 18:53:18,553 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912578366440007, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1222872639712458, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-09-19 18:55:18,540 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6302639 different iterations.  Found a solution that moves 1847 regions; Going from a computed imbalance of 0.9912578366440007 to a new imbalance of 4.3109901406398434E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6840740740740741, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-19 18:55:18,547 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1060610781:90 , srv1129437566:90 , srv1158017689:90 , srv1239739004:90 , srv1341267710:90 , srv1552762508:90 , srv1593411614:90 , srv1658903039:90 , srv168712784:90 , srv1709806371:90 , srv1718145802:90 , srv1759441941:90 , srv1898021845:90 , srv1909113756:90 , srv2023326476:90 , srv231747407:90 , srv278282874:90 , srv306976778:90 , srv356815753:90 , srv503200429:90 , srv513668860:90 , srv515761900:90 , srv632735461:90 , srv633876524:90 , srv690535392:90 , srv78305533:90 , srv792291054:90 , srv833382154:90 , srv889001803:90 , srv995069160:90 }
2023-09-19 18:55:18,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-19 18:55:18,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1658903039=7, srv1239739004=3, srv78305533=25, srv1898021845=12, srv515761900=21, srv1759441941=11, srv995069160=29, srv1060610781=0, srv1129437566=1, srv1709806371=9, srv1593411614=6, srv356815753=18, srv1909113756=13, srv1341267710=4, srv690535392=24, srv513668860=20, srv833382154=27, srv1158017689=2, srv231747407=15, srv1718145802=10, srv503200429=19, srv792291054=26, srv633876524=23, srv306976778=17, srv889001803=28, srv168712784=8, srv632735461=22, srv2023326476=14, srv278282874=16, srv1552762508=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-19 18:55:18,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-19 18:55:18,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-19 18:55:18,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-19 18:55:18,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-19 18:55:18,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-19 18:55:18,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-19 18:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-19 18:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-19 18:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-19 18:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-19 18:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-19 18:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-19 18:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-19 18:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-19 18:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-19 18:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-19 18:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-19 18:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-19 18:55:18,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-19 18:55:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-19 18:55:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-19 18:55:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-19 18:55:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-19 18:55:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-19 18:55:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-19 18:55:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-19 18:55:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-19 18:55:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-19 18:55:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-19 18:55:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-19 18:55:18,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-09-19 18:55:18,574 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-19 18:55:18,574 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1060610781:90 , srv1129437566:90 , srv1158017689:90 , srv1239739004:90 , srv1341267710:90 , srv1552762508:90 , srv1593411614:90 , srv1658903039:90 , srv168712784:90 , srv1709806371:90 , srv1718145802:90 , srv1759441941:90 , srv1898021845:90 , srv1909113756:90 , srv2023326476:90 , srv231747407:90 , srv278282874:90 , srv306976778:90 , srv356815753:90 , srv503200429:90 , srv513668860:90 , srv515761900:90 , srv632735461:90 , srv633876524:90 , srv690535392:90 , srv78305533:90 , srv792291054:90 , srv833382154:90 , srv889001803:90 , srv995069160:90 }
2023-09-19 18:55:18,592 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=168 (was 329), ProcessCount=170 (was 170), AvailableMemoryMB=9309 (was 10240)