2024-01-31 16:53:13,072 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-01-31 16:53:13,391 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-01-31 16:53:13,501 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-01-31 16:53:13,571 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-31 16:53:13,580 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-01-31 16:53:13,594 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=306, ProcessCount=167, AvailableMemoryMB=3054
2024-01-31 16:53:13,596 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-31 16:53:13,597 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-01-31 16:53:13,732 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1038261537:84 , srv1054214181:84 , srv1072873562:84 , srv1087525022:84 , srv1092149880:84 , srv1108037988:84 , srv1164845371:84 , srv1233056705:84 , srv1261586357:84 , srv1271801144:84 , srv1278890675:84 , srv1307033251:84 , srv1319567252:84 , srv1341472810:84 , srv1351316178:84 , srv137055914:84 , srv1452685344:84 , srv1476793088:84 , srv1551503910:84 , srv156073290:84 , srv1562214876:84 , srv1563585266:84 , srv1584343140:84 , srv1638432043:84 , srv1640248584:84 , srv1657920974:84 , srv1658867946:84 , srv1664445860:84 , srv1682404772:84 , srv1779922430:84 , srv1787573198:84 , srv1802948672:84 , srv1815880153:84 , srv1825379912:84 , srv1833444822:84 , srv1834348804:84 , srv1854335043:84 , srv1855876248:84 , srv1861484130:84 , srv1886878609:84 , srv1900966925:84 , srv1916996444:84 , srv1924804582:84 , srv1933602213:84 , srv1945737090:84 , srv1967095426:84 , srv2009314919:84 , srv2016760998:84 , srv2023462623:84 , srv2025868579:84 , srv2075800721:84 , srv2086756999:84 , srv2090190009:84 , srv2111971988:84 , srv2117787106:84 , srv2134005892:84 , srv2143241311:84 , srv214659769:84 , srv22120413:84 , srv24143842:84 , srv241452772:84 , srv249574648:84 , srv314149193:84 , srv39636658:84 , srv406250673:84 , srv419277614:84 , srv445082767:84 , srv462607891:84 , srv487156848:84 , srv489301168:84 , srv511517736:84 , srv540353265:84 , srv541129735:84 , srv547636694:84 , srv555405438:84 , srv578298160:84 , srv586421228:84 , srv59861328:84 , srv634102670:84 , srv642717356:84 , srv663199257:84 , srv673484328:84 , srv675499916:84 , srv683255119:84 , srv688397750:84 , srv746544298:84 , srv753454706:84 , srv76936333:84 , srv770514170:84 , srv787312366:84 , srv793026348:84 , srv811091279:84 , srv822682891:84 , srv829740617:84 , srv846423186:84 , srv861268426:84 , srv893781475:84 , srv896893690:84 , srv936893534:84 , srv1409499610:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-01-31 16:53:13,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-31 16:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1802948672=32, srv2016760998=48, srv1087525022=3, srv683255119=84, srv1815880153=33, srv1886878609=40, srv2134005892=56, srv1278890675=10, srv2143241311=57, srv1563585266=22, srv1584343140=23, srv511517736=71, srv770514170=89, srv1861484130=39, srv555405438=75, srv1562214876=21, srv137055914=15, srv893781475=97, srv846423186=95, srv1900966925=41, srv1038261537=0, srv1825379912=34, srv675499916=83, srv642717356=80, srv1072873562=2, srv787312366=90, srv1341472810=13, srv314149193=63, srv811091279=92, srv406250673=65, srv793026348=91, srv896893690=98, srv861268426=96, srv445082767=67, srv586421228=77, srv1307033251=11, srv2009314919=47, srv39636658=64, srv1092149880=4, srv1664445860=28, srv1682404772=29, srv249574648=62, srv753454706=87, srv1409499610=16, srv1054214181=1, srv1551503910=19, srv419277614=66, srv487156848=69, srv1916996444=42, srv1945737090=45, srv214659769=58, srv1638432043=24, srv462607891=68, srv541129735=73, srv578298160=76, srv547636694=74, srv673484328=82, srv1779922430=30, srv1933602213=44, srv76936333=88, srv663199257=81, srv634102670=79, srv22120413=59, srv1476793088=18, srv1319567252=12, srv1657920974=26, srv2117787106=55, srv489301168=70, srv829740617=94, srv24143842=60, srv1108037988=5, srv2075800721=51, srv1233056705=7, srv746544298=86, srv1351316178=14, srv2111971988=54, srv822682891=93, srv1271801144=9, srv1640248584=25, srv2025868579=50, srv1967095426=46, srv540353265=72, srv2090190009=53, srv2086756999=52, srv1854335043=37, srv2023462623=49, srv241452772=61, srv1833444822=35, srv688397750=85, srv1452685344=17, srv1787573198=31, srv1164845371=6, srv1658867946=27, srv1855876248=38, srv1834348804=36, srv1261586357=8, srv1924804582=43, srv156073290=20, srv936893534=99, srv59861328=78} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-31 16:53:13,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-31 16:53:13,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-31 16:53:13,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-31 16:53:13,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-31 16:53:13,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-31 16:53:13,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-31 16:53:13,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-31 16:53:13,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-31 16:53:13,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-31 16:53:13,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-31 16:53:13,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-31 16:53:13,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-31 16:53:13,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-31 16:53:13,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-31 16:53:13,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-31 16:53:13,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-31 16:53:13,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-31 16:53:13,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-31 16:53:13,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-31 16:53:13,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-31 16:53:13,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-31 16:53:13,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-31 16:53:13,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-31 16:53:13,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-31 16:53:13,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-31 16:53:13,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-31 16:53:13,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-31 16:53:13,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-31 16:53:13,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-31 16:53:13,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-31 16:53:13,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-31 16:53:13,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-31 16:53:13,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-31 16:53:13,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-31 16:53:13,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-31 16:53:13,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-31 16:53:13,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-31 16:53:13,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-31 16:53:13,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-31 16:53:13,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-31 16:53:13,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-31 16:53:13,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-31 16:53:13,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-31 16:53:13,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-31 16:53:13,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-31 16:53:13,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-31 16:53:13,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-31 16:53:13,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-31 16:53:13,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-31 16:53:13,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-31 16:53:13,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-31 16:53:13,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-31 16:53:13,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-31 16:53:13,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-31 16:53:13,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-31 16:53:13,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-31 16:53:13,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-31 16:53:13,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-31 16:53:13,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-31 16:53:13,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-31 16:53:13,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-31 16:53:13,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-31 16:53:13,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-31 16:53:13,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-31 16:53:13,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-31 16:53:13,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-31 16:53:13,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-31 16:53:13,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-31 16:53:13,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-31 16:53:13,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-31 16:53:13,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-31 16:53:13,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-31 16:53:13,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-31 16:53:13,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-31 16:53:13,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-31 16:53:13,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-31 16:53:13,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-31 16:53:13,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-31 16:53:13,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-31 16:53:13,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-31 16:53:13,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-31 16:53:13,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-31 16:53:13,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-31 16:53:13,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-31 16:53:13,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-31 16:53:13,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-31 16:53:13,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-31 16:53:13,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-31 16:53:13,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-31 16:53:13,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-31 16:53:13,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-31 16:53:13,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-31 16:53:13,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-31 16:53:13,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-31 16:53:13,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-31 16:53:13,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-31 16:53:13,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-31 16:53:13,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-31 16:53:13,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-31 16:53:13,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-31 16:53:13,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-31 16:53:14,048 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-31 16:53:14,049 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-31 16:53:14,049 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-31 16:53:14,412 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 530 ms to try 5000 different iterations.  Found a solution that moves 4719 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1771202643945048. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039468030127811624, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05840375739338646, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5243333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039468030127811624, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17233333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23866666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-31 16:53:14,445 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1072873562:83 , srv1233056705:83 , srv1815880153:83 , srv487156848:83 , srv541129735:83 , srv688397750:83 , srv1261586357:84 , srv1271801144:84 , srv1278890675:84 , srv1341472810:84 , srv1452685344:84 , srv1551503910:84 , srv1584343140:84 , srv1640248584:84 , srv1900966925:84 , srv1967095426:84 , srv2025868579:84 , srv2111971988:84 , srv2117787106:84 , srv22120413:84 , srv314149193:84 , srv39636658:84 , srv936893534:84 , srv1164845371:85 , srv1886878609:85 , srv1924804582:85 , srv24143842:85 , srv445082767:85 , srv489301168:85 , srv547636694:85 , srv59861328:85 , srv683255119:85 , srv746544298:85 , srv822682891:85 , srv861268426:85 , srv1054214181:86 , srv1307033251:86 , srv1563585266:86 , srv1658867946:86 , srv1802948672:86 , srv1855876248:86 , srv1861484130:86 , srv1916996444:86 , srv1945737090:86 , srv2023462623:86 , srv241452772:86 , srv511517736:86 , srv642717356:86 , srv753454706:86 , srv770514170:86 , srv829740617:86 , srv1092149880:87 , srv1319567252:87 , srv1351316178:87 , srv137055914:87 , srv1476793088:87 , srv1664445860:87 , srv1682404772:87 , srv1787573198:87 , srv1834348804:87 , srv2009314919:87 , srv2075800721:87 , srv2086756999:87 , srv2090190009:87 , srv406250673:87 , srv462607891:87 , srv540353265:87 , srv555405438:87 , srv578298160:87 , srv634102670:87 , srv663199257:87 , srv787312366:87 , srv793026348:87 , srv1038261537:88 , srv1108037988:88 , srv156073290:88 , srv1657920974:88 , srv1779922430:88 , srv1833444822:88 , srv1854335043:88 , srv2143241311:88 , srv249574648:88 , srv419277614:88 , srv1087525022:89 , srv1562214876:89 , srv1933602213:89 , srv896893690:89 , srv1825379912:90 , srv2134005892:90 , srv675499916:90 , srv76936333:90 , srv846423186:90 , srv893781475:90 , srv2016760998:91 , srv214659769:91 , srv586421228:91 , srv673484328:91 , srv811091279:91 , srv1638432043:92 , srv1409499610:441 }
2024-01-31 16:53:14,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-31 16:53:14,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1802948672=32, srv2016760998=48, srv1087525022=3, srv683255119=84, srv1815880153=33, srv1886878609=40, srv2134005892=56, srv1278890675=10, srv2143241311=57, srv1563585266=22, srv1584343140=23, srv511517736=71, srv770514170=89, srv1861484130=39, srv555405438=75, srv1562214876=21, srv137055914=15, srv893781475=97, srv846423186=95, srv1900966925=41, srv1038261537=0, srv1825379912=34, srv675499916=83, srv642717356=80, srv1072873562=2, srv787312366=90, srv1341472810=13, srv314149193=63, srv811091279=92, srv406250673=65, srv793026348=91, srv896893690=98, srv861268426=96, srv445082767=67, srv586421228=77, srv1307033251=11, srv2009314919=47, srv39636658=64, srv1092149880=4, srv1664445860=28, srv1682404772=29, srv249574648=62, srv753454706=87, srv1409499610=16, srv1054214181=1, srv1551503910=19, srv419277614=66, srv487156848=69, srv1916996444=42, srv1945737090=45, srv214659769=58, srv1638432043=24, srv462607891=68, srv541129735=73, srv578298160=76, srv547636694=74, srv673484328=82, srv1779922430=30, srv1933602213=44, srv76936333=88, srv663199257=81, srv634102670=79, srv22120413=59, srv1476793088=18, srv1319567252=12, srv1657920974=26, srv2117787106=55, srv489301168=70, srv829740617=94, srv24143842=60, srv1108037988=5, srv2075800721=51, srv1233056705=7, srv746544298=86, srv1351316178=14, srv2111971988=54, srv822682891=93, srv1271801144=9, srv1640248584=25, srv2025868579=50, srv1967095426=46, srv540353265=72, srv2090190009=53, srv2086756999=52, srv1854335043=37, srv2023462623=49, srv241452772=61, srv1833444822=35, srv688397750=85, srv1452685344=17, srv1787573198=31, srv1164845371=6, srv1658867946=27, srv1855876248=38, srv1834348804=36, srv1261586357=8, srv1924804582=43, srv156073290=20, srv936893534=99, srv59861328=78} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-31 16:53:14,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-31 16:53:14,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-31 16:53:14,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-31 16:53:14,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-31 16:53:14,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-31 16:53:14,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-31 16:53:14,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-31 16:53:14,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-31 16:53:14,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-31 16:53:14,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-31 16:53:14,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-31 16:53:14,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-31 16:53:14,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-31 16:53:14,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-31 16:53:14,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-31 16:53:14,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-31 16:53:14,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-31 16:53:14,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-31 16:53:14,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-31 16:53:14,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-31 16:53:14,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-31 16:53:14,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-31 16:53:14,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-31 16:53:14,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-31 16:53:14,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-31 16:53:14,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-31 16:53:14,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-31 16:53:14,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-31 16:53:14,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-31 16:53:14,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-31 16:53:14,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-31 16:53:14,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-31 16:53:14,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-31 16:53:14,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-31 16:53:14,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-31 16:53:14,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-31 16:53:14,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-31 16:53:14,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-31 16:53:14,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-31 16:53:14,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-31 16:53:14,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-31 16:53:14,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-31 16:53:14,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-31 16:53:14,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-31 16:53:14,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-31 16:53:14,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-31 16:53:14,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-31 16:53:14,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-31 16:53:14,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-31 16:53:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-31 16:53:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-31 16:53:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-31 16:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-31 16:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-31 16:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-31 16:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-31 16:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-31 16:53:14,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-31 16:53:14,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-31 16:53:14,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-31 16:53:14,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-31 16:53:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-31 16:53:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-31 16:53:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-31 16:53:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-31 16:53:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-31 16:53:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-31 16:53:14,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-31 16:53:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-31 16:53:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-31 16:53:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-31 16:53:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-31 16:53:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-31 16:53:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-31 16:53:14,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-31 16:53:14,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-31 16:53:14,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-31 16:53:14,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-31 16:53:14,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-31 16:53:14,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-31 16:53:14,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-31 16:53:14,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-31 16:53:14,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-31 16:53:14,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-31 16:53:14,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-31 16:53:14,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-31 16:53:14,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-31 16:53:14,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-31 16:53:14,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-31 16:53:14,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-31 16:53:14,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-31 16:53:14,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-31 16:53:14,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-31 16:53:14,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-31 16:53:14,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-31 16:53:14,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-31 16:53:14,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-31 16:53:14,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-31 16:53:14,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-31 16:53:14,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-31 16:53:14,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-31 16:53:14,683 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039468030127811624, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05840375739338646, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039468030127811624, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17233333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23866666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-31 16:53:14,683 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-31 16:53:14,684 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17708722125025947, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039468030127811624, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05840375739338646, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039468030127811624, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17233333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23866666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-31 16:53:15,009 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 475 ms to try 5000 different iterations.  Found a solution that moves 2602 regions; Going from a computed imbalance of 0.17708722125025947 to a new imbalance of 0.017907053900982482. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02329080620272224); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039154934126393315, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2891111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02329080620272224); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.014833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04716666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-31 16:53:15,025 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv214659769:78 , srv2016760998:79 , srv1638432043:80 , srv673484328:81 , srv1038261537:82 , srv1108037988:82 , srv156073290:82 , srv1825379912:82 , srv1854335043:82 , srv1933602213:82 , srv419277614:82 , srv586421228:82 , srv675499916:82 , srv76936333:82 , srv1563585266:83 , srv1657920974:83 , srv1779922430:83 , srv2086756999:83 , srv2143241311:83 , srv893781475:83 , srv1476793088:84 , srv1562214876:84 , srv1682404772:84 , srv1833444822:84 , srv1916996444:84 , srv1945737090:84 , srv2075800721:84 , srv2134005892:84 , srv249574648:84 , srv540353265:84 , srv634102670:84 , srv683255119:84 , srv793026348:84 , srv811091279:84 , srv846423186:84 , srv896893690:84 , srv1087525022:85 , srv1092149880:85 , srv1319567252:85 , srv1341472810:85 , srv1351316178:85 , srv137055914:85 , srv1452685344:85 , srv1658867946:85 , srv1834348804:85 , srv1855876248:85 , srv2009314919:85 , srv2090190009:85 , srv241452772:85 , srv39636658:85 , srv406250673:85 , srv753454706:85 , srv1787573198:86 , srv1861484130:86 , srv1900966925:86 , srv314149193:86 , srv462607891:86 , srv555405438:86 , srv578298160:86 , srv642717356:86 , srv663199257:86 , srv746544298:86 , srv770514170:86 , srv787312366:86 , srv829740617:86 , srv1164845371:87 , srv1261586357:87 , srv1307033251:87 , srv1802948672:87 , srv2025868579:87 , srv2111971988:87 , srv489301168:87 , srv547636694:87 , srv1054214181:88 , srv1072873562:88 , srv1271801144:88 , srv1278890675:88 , srv1584343140:88 , srv1640248584:88 , srv1664445860:88 , srv1886878609:88 , srv1967095426:88 , srv2023462623:88 , srv22120413:88 , srv445082767:88 , srv487156848:88 , srv511517736:88 , srv822682891:88 , srv861268426:88 , srv936893534:88 , srv1551503910:89 , srv2117787106:89 , srv24143842:89 , srv688397750:89 , srv1233056705:90 , srv1924804582:90 , srv59861328:90 , srv1815880153:92 , srv541129735:92 , srv1409499610:540 }
2024-01-31 16:53:15,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-31 16:53:15,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1802948672=32, srv2016760998=48, srv1087525022=3, srv683255119=84, srv1815880153=33, srv1886878609=40, srv2134005892=56, srv1278890675=10, srv2143241311=57, srv1563585266=22, srv1584343140=23, srv511517736=71, srv770514170=89, srv1861484130=39, srv555405438=75, srv1562214876=21, srv137055914=15, srv893781475=97, srv846423186=95, srv1900966925=41, srv1038261537=0, srv1825379912=34, srv675499916=83, srv642717356=80, srv1072873562=2, srv787312366=90, srv1341472810=13, srv314149193=63, srv811091279=92, srv406250673=65, srv793026348=91, srv896893690=98, srv861268426=96, srv445082767=67, srv586421228=77, srv1307033251=11, srv2009314919=47, srv39636658=64, srv1092149880=4, srv1664445860=28, srv1682404772=29, srv249574648=62, srv753454706=87, srv1409499610=16, srv1054214181=1, srv1551503910=19, srv419277614=66, srv487156848=69, srv1916996444=42, srv1945737090=45, srv214659769=58, srv1638432043=24, srv462607891=68, srv541129735=73, srv578298160=76, srv547636694=74, srv673484328=82, srv1779922430=30, srv1933602213=44, srv76936333=88, srv663199257=81, srv634102670=79, srv22120413=59, srv1476793088=18, srv1319567252=12, srv1657920974=26, srv2117787106=55, srv489301168=70, srv829740617=94, srv24143842=60, srv1108037988=5, srv2075800721=51, srv1233056705=7, srv746544298=86, srv1351316178=14, srv2111971988=54, srv822682891=93, srv1271801144=9, srv1640248584=25, srv2025868579=50, srv1967095426=46, srv540353265=72, srv2090190009=53, srv2086756999=52, srv1854335043=37, srv2023462623=49, srv241452772=61, srv1833444822=35, srv688397750=85, srv1452685344=17, srv1787573198=31, srv1164845371=6, srv1658867946=27, srv1855876248=38, srv1834348804=36, srv1261586357=8, srv1924804582=43, srv156073290=20, srv936893534=99, srv59861328=78} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-31 16:53:15,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-31 16:53:15,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-31 16:53:15,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-31 16:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-31 16:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-31 16:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-31 16:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-31 16:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-31 16:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-31 16:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-31 16:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-31 16:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-31 16:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-31 16:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-31 16:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-31 16:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-31 16:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-31 16:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-31 16:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-31 16:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-31 16:53:15,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-31 16:53:15,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-31 16:53:15,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-31 16:53:15,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-31 16:53:15,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-31 16:53:15,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-31 16:53:15,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-31 16:53:15,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-31 16:53:15,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-31 16:53:15,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-31 16:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-31 16:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-31 16:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-31 16:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-31 16:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-31 16:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-31 16:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-31 16:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-31 16:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-31 16:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-31 16:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-31 16:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-31 16:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-31 16:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-31 16:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-31 16:53:15,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-31 16:53:15,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-31 16:53:15,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-31 16:53:15,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-31 16:53:15,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-31 16:53:15,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-31 16:53:15,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-31 16:53:15,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-31 16:53:15,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-31 16:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-31 16:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-31 16:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-31 16:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-31 16:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-31 16:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-31 16:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-31 16:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-31 16:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-31 16:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-31 16:53:15,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-31 16:53:15,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-31 16:53:15,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-31 16:53:15,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-31 16:53:15,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-31 16:53:15,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-31 16:53:15,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-31 16:53:15,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-31 16:53:15,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-31 16:53:15,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-31 16:53:15,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-31 16:53:15,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-31 16:53:15,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-31 16:53:15,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-31 16:53:15,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-31 16:53:15,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-31 16:53:15,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-31 16:53:15,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-31 16:53:15,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-31 16:53:15,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-31 16:53:15,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-31 16:53:15,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-31 16:53:15,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-31 16:53:15,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-31 16:53:15,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-31 16:53:15,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-31 16:53:15,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-31 16:53:15,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-31 16:53:15,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-31 16:53:15,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-31 16:53:15,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-31 16:53:15,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-31 16:53:15,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-31 16:53:15,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-31 16:53:15,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-31 16:53:15,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-31 16:53:15,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-31 16:53:15,193 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02329080620272224); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039154934126393315, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02329080620272224); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.014833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04716666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-31 16:53:15,193 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-31 16:53:15,194 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0178888343075673, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02329080620272224); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039154934126393315, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02329080620272224); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.014833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04716666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-31 16:53:15,536 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 455 ms to try 5000 different iterations.  Found a solution that moves 1033 regions; Going from a computed imbalance of 0.0178888343075673 to a new imbalance of 9.169774783255345E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00531581484978049); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00808207060844344); MoveCostFunction : (multiplier=7.0, imbalance=0.11477777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00531581484978049); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-31 16:53:15,538 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1664445860:82 , srv1924804582:82 , srv541129735:82 , srv59861328:82 , srv811091279:82 , srv1087525022:83 , srv1815880153:83 , srv2023462623:83 , srv24143842:83 , srv1054214181:84 , srv1319567252:84 , srv1562214876:84 , srv1802948672:84 , srv1886878609:84 , srv2134005892:84 , srv511517736:84 , srv586421228:84 , srv642717356:84 , srv663199257:84 , srv846423186:84 , srv861268426:84 , srv893781475:84 , srv896893690:84 , srv1092149880:85 , srv1164845371:85 , srv1233056705:85 , srv1278890675:85 , srv1307033251:85 , srv1351316178:85 , srv1551503910:85 , srv1787573198:85 , srv1825379912:85 , srv1834348804:85 , srv2009314919:85 , srv2117787106:85 , srv445082767:85 , srv462607891:85 , srv489301168:85 , srv555405438:85 , srv578298160:85 , srv673484328:85 , srv76936333:85 , srv787312366:85 , srv822682891:85 , srv1261586357:86 , srv1271801144:86 , srv137055914:86 , srv1584343140:86 , srv1638432043:86 , srv1657920974:86 , srv1779922430:86 , srv1833444822:86 , srv1861484130:86 , srv1967095426:86 , srv2075800721:86 , srv2143241311:86 , srv22120413:86 , srv249574648:86 , srv406250673:86 , srv487156848:86 , srv547636694:86 , srv675499916:86 , srv688397750:86 , srv770514170:86 , srv829740617:86 , srv936893534:86 , srv1038261537:87 , srv1072873562:87 , srv1108037988:87 , srv1476793088:87 , srv1640248584:87 , srv1658867946:87 , srv1682404772:87 , srv1854335043:87 , srv1855876248:87 , srv1900966925:87 , srv1916996444:87 , srv1933602213:87 , srv1945737090:87 , srv2016760998:87 , srv2025868579:87 , srv2086756999:87 , srv2090190009:87 , srv2111971988:87 , srv241452772:87 , srv314149193:87 , srv419277614:87 , srv634102670:87 , srv746544298:87 , srv753454706:87 , srv793026348:87 , srv1341472810:88 , srv156073290:88 , srv1563585266:88 , srv39636658:88 , srv540353265:88 , srv683255119:88 , srv1452685344:89 , srv214659769:89 , srv1409499610:524 }
2024-01-31 16:53:15,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-31 16:53:15,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1802948672=32, srv2016760998=48, srv1087525022=3, srv683255119=84, srv1815880153=33, srv1886878609=40, srv2134005892=56, srv1278890675=10, srv2143241311=57, srv1563585266=22, srv1584343140=23, srv511517736=71, srv770514170=89, srv1861484130=39, srv555405438=75, srv1562214876=21, srv137055914=15, srv893781475=97, srv846423186=95, srv1900966925=41, srv1038261537=0, srv1825379912=34, srv675499916=83, srv642717356=80, srv1072873562=2, srv787312366=90, srv1341472810=13, srv314149193=63, srv811091279=92, srv406250673=65, srv793026348=91, srv896893690=98, srv861268426=96, srv445082767=67, srv586421228=77, srv1307033251=11, srv2009314919=47, srv39636658=64, srv1092149880=4, srv1664445860=28, srv1682404772=29, srv249574648=62, srv753454706=87, srv1409499610=16, srv1054214181=1, srv1551503910=19, srv419277614=66, srv487156848=69, srv1916996444=42, srv1945737090=45, srv214659769=58, srv1638432043=24, srv462607891=68, srv541129735=73, srv578298160=76, srv547636694=74, srv673484328=82, srv1779922430=30, srv1933602213=44, srv76936333=88, srv663199257=81, srv634102670=79, srv22120413=59, srv1476793088=18, srv1319567252=12, srv1657920974=26, srv2117787106=55, srv489301168=70, srv829740617=94, srv24143842=60, srv1108037988=5, srv2075800721=51, srv1233056705=7, srv746544298=86, srv1351316178=14, srv2111971988=54, srv822682891=93, srv1271801144=9, srv1640248584=25, srv2025868579=50, srv1967095426=46, srv540353265=72, srv2090190009=53, srv2086756999=52, srv1854335043=37, srv2023462623=49, srv241452772=61, srv1833444822=35, srv688397750=85, srv1452685344=17, srv1787573198=31, srv1164845371=6, srv1658867946=27, srv1855876248=38, srv1834348804=36, srv1261586357=8, srv1924804582=43, srv156073290=20, srv936893534=99, srv59861328=78} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-31 16:53:15,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-31 16:53:15,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-31 16:53:15,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-31 16:53:15,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-31 16:53:15,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-31 16:53:15,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-31 16:53:15,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-31 16:53:15,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-31 16:53:15,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-31 16:53:15,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-31 16:53:15,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-31 16:53:15,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-31 16:53:15,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-31 16:53:15,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-31 16:53:15,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-31 16:53:15,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-31 16:53:15,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-31 16:53:15,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-31 16:53:15,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-31 16:53:15,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-31 16:53:15,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-31 16:53:15,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-31 16:53:15,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-31 16:53:15,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-31 16:53:15,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-31 16:53:15,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-31 16:53:15,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-31 16:53:15,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-31 16:53:15,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-31 16:53:15,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-31 16:53:15,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-31 16:53:15,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-31 16:53:15,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-31 16:53:15,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-31 16:53:15,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-31 16:53:15,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-31 16:53:15,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-31 16:53:15,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-31 16:53:15,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-31 16:53:15,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-31 16:53:15,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-31 16:53:15,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-31 16:53:15,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-31 16:53:15,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-31 16:53:15,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-31 16:53:15,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-31 16:53:15,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-31 16:53:15,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-31 16:53:15,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-31 16:53:15,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-31 16:53:15,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-31 16:53:15,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-31 16:53:15,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-31 16:53:15,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-31 16:53:15,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-31 16:53:15,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-31 16:53:15,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-31 16:53:15,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-31 16:53:15,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-31 16:53:15,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-31 16:53:15,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-31 16:53:15,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-31 16:53:15,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-31 16:53:15,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-31 16:53:15,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-31 16:53:15,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-31 16:53:15,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-31 16:53:15,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-31 16:53:15,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-31 16:53:15,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-31 16:53:15,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-31 16:53:15,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-31 16:53:15,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-31 16:53:15,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-31 16:53:15,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-31 16:53:15,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-31 16:53:15,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-31 16:53:15,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-31 16:53:15,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-31 16:53:15,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-31 16:53:15,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-31 16:53:15,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-31 16:53:15,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-31 16:53:15,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-31 16:53:15,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-31 16:53:15,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-31 16:53:15,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-31 16:53:15,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-31 16:53:15,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-31 16:53:15,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-31 16:53:15,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-31 16:53:15,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-31 16:53:15,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-31 16:53:15,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-31 16:53:15,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-31 16:53:15,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-31 16:53:15,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-31 16:53:15,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-31 16:53:15,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-31 16:53:15,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-31 16:53:15,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-31 16:53:15,683 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00531581484978049); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00808207060844344); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00531581484978049); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-31 16:53:15,683 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-31 16:53:15,683 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.097442577268107E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00531581484978049); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00808207060844344); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00531581484978049); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-31 16:53:15,956 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 384 ms to try 5000 different iterations.  Found a solution that moves 244 regions; Going from a computed imbalance of 9.097442577268107E-4 to a new imbalance of 2.8402126269875486E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.02711111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-31 16:53:15,957 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1072873562:83 , srv1640248584:83 , srv1933602213:83 , srv2090190009:83 , srv2134005892:83 , srv540353265:83 , srv1054214181:84 , srv1233056705:84 , srv1261586357:84 , srv1271801144:84 , srv1307033251:84 , srv137055914:84 , srv1452685344:84 , srv1476793088:84 , srv156073290:84 , srv1638432043:84 , srv1658867946:84 , srv1664445860:84 , srv1682404772:84 , srv1787573198:84 , srv1815880153:84 , srv1833444822:84 , srv1855876248:84 , srv1861484130:84 , srv1916996444:84 , srv1967095426:84 , srv2025868579:84 , srv2111971988:84 , srv2117787106:84 , srv2143241311:84 , srv214659769:84 , srv22120413:84 , srv241452772:84 , srv445082767:84 , srv462607891:84 , srv547636694:84 , srv555405438:84 , srv578298160:84 , srv634102670:84 , srv675499916:84 , srv688397750:84 , srv746544298:84 , srv770514170:84 , srv787312366:84 , srv793026348:84 , srv822682891:84 , srv829740617:84 , srv846423186:84 , srv936893534:84 , srv1038261537:85 , srv1092149880:85 , srv1108037988:85 , srv1164845371:85 , srv1278890675:85 , srv1341472810:85 , srv1351316178:85 , srv1551503910:85 , srv1562214876:85 , srv1563585266:85 , srv1584343140:85 , srv1657920974:85 , srv1779922430:85 , srv1802948672:85 , srv1825379912:85 , srv1834348804:85 , srv1854335043:85 , srv1886878609:85 , srv1900966925:85 , srv1924804582:85 , srv1945737090:85 , srv2009314919:85 , srv2016760998:85 , srv2023462623:85 , srv2075800721:85 , srv2086756999:85 , srv24143842:85 , srv249574648:85 , srv314149193:85 , srv39636658:85 , srv406250673:85 , srv419277614:85 , srv487156848:85 , srv489301168:85 , srv511517736:85 , srv541129735:85 , srv586421228:85 , srv59861328:85 , srv642717356:85 , srv663199257:85 , srv673484328:85 , srv683255119:85 , srv753454706:85 , srv76936333:85 , srv861268426:85 , srv893781475:85 , srv896893690:85 , srv1087525022:86 , srv1319567252:86 , srv811091279:86 , srv1409499610:637 }
2024-01-31 16:53:15,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-31 16:53:15,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1802948672=32, srv2016760998=48, srv1087525022=3, srv683255119=84, srv1815880153=33, srv1886878609=40, srv2134005892=56, srv1278890675=10, srv2143241311=57, srv1563585266=22, srv1584343140=23, srv511517736=71, srv770514170=89, srv1861484130=39, srv555405438=75, srv1562214876=21, srv137055914=15, srv893781475=97, srv846423186=95, srv1900966925=41, srv1038261537=0, srv1825379912=34, srv675499916=83, srv642717356=80, srv1072873562=2, srv787312366=90, srv1341472810=13, srv314149193=63, srv811091279=92, srv406250673=65, srv793026348=91, srv896893690=98, srv861268426=96, srv445082767=67, srv586421228=77, srv1307033251=11, srv2009314919=47, srv39636658=64, srv1092149880=4, srv1664445860=28, srv1682404772=29, srv249574648=62, srv753454706=87, srv1409499610=16, srv1054214181=1, srv1551503910=19, srv419277614=66, srv487156848=69, srv1916996444=42, srv1945737090=45, srv214659769=58, srv1638432043=24, srv462607891=68, srv541129735=73, srv578298160=76, srv547636694=74, srv673484328=82, srv1779922430=30, srv1933602213=44, srv76936333=88, srv663199257=81, srv634102670=79, srv22120413=59, srv1476793088=18, srv1319567252=12, srv1657920974=26, srv2117787106=55, srv489301168=70, srv829740617=94, srv24143842=60, srv1108037988=5, srv2075800721=51, srv1233056705=7, srv746544298=86, srv1351316178=14, srv2111971988=54, srv822682891=93, srv1271801144=9, srv1640248584=25, srv2025868579=50, srv1967095426=46, srv540353265=72, srv2090190009=53, srv2086756999=52, srv1854335043=37, srv2023462623=49, srv241452772=61, srv1833444822=35, srv688397750=85, srv1452685344=17, srv1787573198=31, srv1164845371=6, srv1658867946=27, srv1855876248=38, srv1834348804=36, srv1261586357=8, srv1924804582=43, srv156073290=20, srv936893534=99, srv59861328=78} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-31 16:53:15,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-31 16:53:15,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-31 16:53:15,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-31 16:53:15,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-31 16:53:15,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-31 16:53:15,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-31 16:53:15,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-31 16:53:15,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-31 16:53:15,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-31 16:53:15,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-31 16:53:15,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-31 16:53:15,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-31 16:53:15,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-31 16:53:15,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-31 16:53:15,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-31 16:53:15,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-31 16:53:15,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-31 16:53:15,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-31 16:53:15,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-31 16:53:15,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-31 16:53:15,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-31 16:53:15,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-31 16:53:15,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-31 16:53:15,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-31 16:53:15,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-31 16:53:15,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-31 16:53:15,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-31 16:53:15,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-31 16:53:15,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-31 16:53:15,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-31 16:53:15,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-31 16:53:15,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-31 16:53:15,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-31 16:53:15,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-31 16:53:15,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-31 16:53:15,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-31 16:53:15,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-31 16:53:15,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-31 16:53:15,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-31 16:53:15,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-31 16:53:15,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-31 16:53:15,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-31 16:53:15,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-31 16:53:15,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-31 16:53:15,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-31 16:53:15,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-31 16:53:15,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-31 16:53:15,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-31 16:53:15,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-31 16:53:15,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-31 16:53:15,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-31 16:53:15,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-31 16:53:15,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-31 16:53:15,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-31 16:53:15,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-31 16:53:15,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-31 16:53:15,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-31 16:53:15,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-31 16:53:15,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-31 16:53:15,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-31 16:53:15,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-31 16:53:15,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-31 16:53:15,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-31 16:53:15,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-31 16:53:15,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-31 16:53:15,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-31 16:53:15,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-31 16:53:15,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-31 16:53:15,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-31 16:53:15,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-31 16:53:15,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-31 16:53:15,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-31 16:53:15,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-31 16:53:15,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-31 16:53:15,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-31 16:53:15,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-31 16:53:15,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-31 16:53:15,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-31 16:53:15,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-31 16:53:15,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-31 16:53:15,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-31 16:53:15,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-31 16:53:15,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-31 16:53:15,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-31 16:53:15,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-31 16:53:15,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-31 16:53:15,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-31 16:53:15,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-31 16:53:15,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-31 16:53:15,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-31 16:53:15,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-31 16:53:15,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-31 16:53:15,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-31 16:53:15,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-31 16:53:15,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-31 16:53:15,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-31 16:53:15,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-31 16:53:15,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-31 16:53:15,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-31 16:53:15,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-31 16:53:15,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-31 16:53:16,134 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-31 16:53:16,134 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-31 16:53:16,134 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.6693601752848763E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-31 16:53:16,368 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 373 ms to try 5000 different iterations.  Found a solution that moves 30 regions; Going from a computed imbalance of 2.6693601752848763E-5 to a new imbalance of 7.283303786127816E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0033333333333333335); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-31 16:53:16,369 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1087525022:83 , srv1551503910:83 , srv1584343140:83 , srv249574648:83 , srv406250673:83 , srv511517736:83 , srv753454706:83 , srv811091279:83 , srv1038261537:84 , srv1054214181:84 , srv1092149880:84 , srv1108037988:84 , srv1164845371:84 , srv1233056705:84 , srv1271801144:84 , srv1278890675:84 , srv1307033251:84 , srv1319567252:84 , srv1341472810:84 , srv1351316178:84 , srv137055914:84 , srv1452685344:84 , srv1476793088:84 , srv156073290:84 , srv1562214876:84 , srv1563585266:84 , srv1638432043:84 , srv1640248584:84 , srv1657920974:84 , srv1658867946:84 , srv1682404772:84 , srv1779922430:84 , srv1787573198:84 , srv1802948672:84 , srv1815880153:84 , srv1825379912:84 , srv1833444822:84 , srv1834348804:84 , srv1854335043:84 , srv1855876248:84 , srv1861484130:84 , srv1886878609:84 , srv1900966925:84 , srv1924804582:84 , srv1945737090:84 , srv1967095426:84 , srv2009314919:84 , srv2016760998:84 , srv2023462623:84 , srv2025868579:84 , srv2075800721:84 , srv2086756999:84 , srv2090190009:84 , srv2111971988:84 , srv2117787106:84 , srv214659769:84 , srv22120413:84 , srv24143842:84 , srv241452772:84 , srv314149193:84 , srv39636658:84 , srv419277614:84 , srv445082767:84 , srv462607891:84 , srv487156848:84 , srv489301168:84 , srv540353265:84 , srv541129735:84 , srv547636694:84 , srv555405438:84 , srv578298160:84 , srv586421228:84 , srv59861328:84 , srv634102670:84 , srv663199257:84 , srv673484328:84 , srv675499916:84 , srv683255119:84 , srv688397750:84 , srv746544298:84 , srv76936333:84 , srv770514170:84 , srv787312366:84 , srv793026348:84 , srv822682891:84 , srv829740617:84 , srv846423186:84 , srv861268426:84 , srv893781475:84 , srv896893690:84 , srv936893534:84 , srv1072873562:85 , srv1261586357:85 , srv1664445860:85 , srv1916996444:85 , srv1933602213:85 , srv2134005892:85 , srv2143241311:85 , srv642717356:85 , srv1409499610:684 }
2024-01-31 16:53:16,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-31 16:53:16,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1802948672=32, srv2016760998=48, srv1087525022=3, srv683255119=84, srv1815880153=33, srv1886878609=40, srv2134005892=56, srv1278890675=10, srv2143241311=57, srv1563585266=22, srv1584343140=23, srv511517736=71, srv770514170=89, srv1861484130=39, srv555405438=75, srv1562214876=21, srv137055914=15, srv893781475=97, srv846423186=95, srv1900966925=41, srv1038261537=0, srv1825379912=34, srv675499916=83, srv642717356=80, srv1072873562=2, srv787312366=90, srv1341472810=13, srv314149193=63, srv811091279=92, srv406250673=65, srv793026348=91, srv896893690=98, srv861268426=96, srv445082767=67, srv586421228=77, srv1307033251=11, srv2009314919=47, srv39636658=64, srv1092149880=4, srv1664445860=28, srv1682404772=29, srv249574648=62, srv753454706=87, srv1409499610=16, srv1054214181=1, srv1551503910=19, srv419277614=66, srv487156848=69, srv1916996444=42, srv1945737090=45, srv214659769=58, srv1638432043=24, srv462607891=68, srv541129735=73, srv578298160=76, srv547636694=74, srv673484328=82, srv1779922430=30, srv1933602213=44, srv76936333=88, srv663199257=81, srv634102670=79, srv22120413=59, srv1476793088=18, srv1319567252=12, srv1657920974=26, srv2117787106=55, srv489301168=70, srv829740617=94, srv24143842=60, srv1108037988=5, srv2075800721=51, srv1233056705=7, srv746544298=86, srv1351316178=14, srv2111971988=54, srv822682891=93, srv1271801144=9, srv1640248584=25, srv2025868579=50, srv1967095426=46, srv540353265=72, srv2090190009=53, srv2086756999=52, srv1854335043=37, srv2023462623=49, srv241452772=61, srv1833444822=35, srv688397750=85, srv1452685344=17, srv1787573198=31, srv1164845371=6, srv1658867946=27, srv1855876248=38, srv1834348804=36, srv1261586357=8, srv1924804582=43, srv156073290=20, srv936893534=99, srv59861328=78} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-31 16:53:16,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-31 16:53:16,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-31 16:53:16,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-31 16:53:16,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-31 16:53:16,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-31 16:53:16,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-31 16:53:16,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-31 16:53:16,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-31 16:53:16,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-31 16:53:16,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-31 16:53:16,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-31 16:53:16,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-31 16:53:16,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-31 16:53:16,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-31 16:53:16,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-31 16:53:16,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-31 16:53:16,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-31 16:53:16,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-31 16:53:16,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-31 16:53:16,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-31 16:53:16,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-31 16:53:16,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-31 16:53:16,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-31 16:53:16,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-31 16:53:16,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-31 16:53:16,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-31 16:53:16,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-31 16:53:16,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-31 16:53:16,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-31 16:53:16,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-31 16:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-31 16:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-31 16:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-31 16:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-31 16:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-31 16:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-31 16:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-31 16:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-31 16:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-31 16:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-31 16:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-31 16:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-31 16:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-31 16:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-31 16:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-31 16:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-31 16:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-31 16:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-31 16:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-31 16:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-31 16:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-31 16:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-31 16:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-31 16:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-31 16:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-31 16:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-31 16:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-31 16:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-31 16:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-31 16:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-31 16:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-31 16:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-31 16:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-31 16:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-31 16:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-31 16:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-31 16:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-31 16:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-31 16:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-31 16:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-31 16:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-31 16:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-31 16:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-31 16:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-31 16:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-31 16:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-31 16:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-31 16:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-31 16:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-31 16:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-31 16:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-31 16:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-31 16:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-31 16:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-31 16:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-31 16:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-31 16:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-31 16:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-31 16:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-31 16:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-31 16:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-31 16:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-31 16:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-31 16:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-31 16:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-31 16:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-31 16:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-31 16:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-31 16:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-31 16:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-31 16:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-31 16:53:16,494 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.073239296329447E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-31 16:53:16,495 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1087525022:83 , srv1551503910:83 , srv1584343140:83 , srv249574648:83 , srv406250673:83 , srv511517736:83 , srv753454706:83 , srv811091279:83 , srv1038261537:84 , srv1054214181:84 , srv1092149880:84 , srv1108037988:84 , srv1164845371:84 , srv1233056705:84 , srv1271801144:84 , srv1278890675:84 , srv1307033251:84 , srv1319567252:84 , srv1341472810:84 , srv1351316178:84 , srv137055914:84 , srv1452685344:84 , srv1476793088:84 , srv156073290:84 , srv1562214876:84 , srv1563585266:84 , srv1638432043:84 , srv1640248584:84 , srv1657920974:84 , srv1658867946:84 , srv1682404772:84 , srv1779922430:84 , srv1787573198:84 , srv1802948672:84 , srv1815880153:84 , srv1825379912:84 , srv1833444822:84 , srv1834348804:84 , srv1854335043:84 , srv1855876248:84 , srv1861484130:84 , srv1886878609:84 , srv1900966925:84 , srv1924804582:84 , srv1945737090:84 , srv1967095426:84 , srv2009314919:84 , srv2016760998:84 , srv2023462623:84 , srv2025868579:84 , srv2075800721:84 , srv2086756999:84 , srv2090190009:84 , srv2111971988:84 , srv2117787106:84 , srv214659769:84 , srv22120413:84 , srv24143842:84 , srv241452772:84 , srv314149193:84 , srv39636658:84 , srv419277614:84 , srv445082767:84 , srv462607891:84 , srv487156848:84 , srv489301168:84 , srv540353265:84 , srv541129735:84 , srv547636694:84 , srv555405438:84 , srv578298160:84 , srv586421228:84 , srv59861328:84 , srv634102670:84 , srv663199257:84 , srv673484328:84 , srv675499916:84 , srv683255119:84 , srv688397750:84 , srv746544298:84 , srv76936333:84 , srv770514170:84 , srv787312366:84 , srv793026348:84 , srv822682891:84 , srv829740617:84 , srv846423186:84 , srv861268426:84 , srv893781475:84 , srv896893690:84 , srv936893534:84 , srv1072873562:85 , srv1261586357:85 , srv1664445860:85 , srv1916996444:85 , srv1933602213:85 , srv2134005892:85 , srv2143241311:85 , srv642717356:85 , srv1409499610:684 }
2024-01-31 16:53:16,529 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=322 (was 306) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=2131 (was 3054)
2024-01-31 16:53:16,536 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=322, ProcessCount=167, AvailableMemoryMB=2130
2024-01-31 16:53:16,536 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-31 16:53:16,537 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-01-31 16:53:16,548 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1092710055:84 , srv1191987281:84 , srv1283041766:84 , srv1361790262:84 , srv1372214440:84 , srv137875594:84 , srv1625631529:84 , srv1694191183:84 , srv1739383266:84 , srv1788443534:84 , srv1845243739:84 , srv1879538342:84 , srv1895641611:84 , srv1975719820:84 , srv2104368503:84 , srv238102381:84 , srv35786073:84 , srv41382444:84 , srv422405634:84 , srv480671071:84 , srv506031520:84 , srv578815052:84 , srv612282621:84 , srv700294883:84 , srv729350422:84 , srv882100407:84 , srv91908068:84 , srv940292425:84 , srv995743387:84 , srv2063424930:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-01-31 16:53:16,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-31 16:53:16,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv700294883=24, srv1361790262=3, srv506031520=21, srv422405634=19, srv882100407=26, srv91908068=27, srv1191987281=1, srv1788443534=9, srv2063424930=14, srv1975719820=13, srv1625631529=6, srv480671071=20, srv35786073=17, srv940292425=28, srv137875594=5, srv995743387=29, srv1694191183=7, srv41382444=18, srv1372214440=4, srv1092710055=0, srv1739383266=8, srv1879538342=11, srv1895641611=12, srv578815052=22, srv729350422=25, srv1283041766=2, srv2104368503=15, srv238102381=16, srv612282621=23, srv1845243739=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-31 16:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-31 16:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-31 16:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-31 16:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-31 16:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-31 16:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-31 16:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-31 16:53:16,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-31 16:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-31 16:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-31 16:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-31 16:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-31 16:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-31 16:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-31 16:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-31 16:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-31 16:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-31 16:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-31 16:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-31 16:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-31 16:53:16,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-31 16:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-31 16:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-31 16:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-31 16:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-31 16:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-31 16:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-31 16:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-31 16:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-31 16:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-31 16:53:16,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-01-31 16:53:16,585 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1406586987823992, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-31 16:53:16,585 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912636254231214, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1406586987823992, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-01-31 16:55:16,565 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6982338 different iterations.  Found a solution that moves 1843 regions; Going from a computed imbalance of 0.9912636254231214 to a new imbalance of 4.3016539410932495E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6825925925925926, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-31 16:55:16,573 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1092710055:90 , srv1191987281:90 , srv1283041766:90 , srv1361790262:90 , srv1372214440:90 , srv137875594:90 , srv1625631529:90 , srv1694191183:90 , srv1739383266:90 , srv1788443534:90 , srv1845243739:90 , srv1879538342:90 , srv1895641611:90 , srv1975719820:90 , srv2063424930:90 , srv2104368503:90 , srv238102381:90 , srv35786073:90 , srv41382444:90 , srv422405634:90 , srv480671071:90 , srv506031520:90 , srv578815052:90 , srv612282621:90 , srv700294883:90 , srv729350422:90 , srv882100407:90 , srv91908068:90 , srv940292425:90 , srv995743387:90 }
2024-01-31 16:55:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-31 16:55:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv700294883=24, srv1361790262=3, srv506031520=21, srv422405634=19, srv882100407=26, srv91908068=27, srv1191987281=1, srv1788443534=9, srv2063424930=14, srv1975719820=13, srv1625631529=6, srv480671071=20, srv35786073=17, srv940292425=28, srv137875594=5, srv995743387=29, srv1694191183=7, srv41382444=18, srv1372214440=4, srv1092710055=0, srv1739383266=8, srv1879538342=11, srv1895641611=12, srv578815052=22, srv729350422=25, srv1283041766=2, srv2104368503=15, srv238102381=16, srv612282621=23, srv1845243739=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-31 16:55:16,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-31 16:55:16,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-31 16:55:16,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-31 16:55:16,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-31 16:55:16,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-31 16:55:16,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-31 16:55:16,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-31 16:55:16,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-31 16:55:16,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-31 16:55:16,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-31 16:55:16,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-31 16:55:16,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-31 16:55:16,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-31 16:55:16,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-31 16:55:16,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-31 16:55:16,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-31 16:55:16,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-31 16:55:16,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-31 16:55:16,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-31 16:55:16,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-31 16:55:16,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-31 16:55:16,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-31 16:55:16,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-31 16:55:16,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-31 16:55:16,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-31 16:55:16,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-31 16:55:16,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-31 16:55:16,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-31 16:55:16,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-31 16:55:16,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-31 16:55:16,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-01-31 16:55:16,592 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-31 16:55:16,592 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1092710055:90 , srv1191987281:90 , srv1283041766:90 , srv1361790262:90 , srv1372214440:90 , srv137875594:90 , srv1625631529:90 , srv1694191183:90 , srv1739383266:90 , srv1788443534:90 , srv1845243739:90 , srv1879538342:90 , srv1895641611:90 , srv1975719820:90 , srv2063424930:90 , srv2104368503:90 , srv238102381:90 , srv35786073:90 , srv41382444:90 , srv422405634:90 , srv480671071:90 , srv506031520:90 , srv578815052:90 , srv612282621:90 , srv700294883:90 , srv729350422:90 , srv882100407:90 , srv91908068:90 , srv940292425:90 , srv995743387:90 }
2024-01-31 16:55:16,604 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=172 (was 322), ProcessCount=167 (was 167), AvailableMemoryMB=2296 (was 2130) - AvailableMemoryMB LEAK? -