2024-02-08 18:54:18,440 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-02-08 18:54:18,699 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-02-08 18:54:18,794 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-02-08 18:54:18,849 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-02-08 18:54:18,856 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-02-08 18:54:18,869 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=231, ProcessCount=176, AvailableMemoryMB=2255
2024-02-08 18:54:18,872 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-02-08 18:54:18,872 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-02-08 18:54:18,988 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1015032081:84 , srv1029436307:84 , srv1091543353:84 , srv1093025275:84 , srv1099148575:84 , srv1165227440:84 , srv1165283720:84 , srv1227038938:84 , srv1227622242:84 , srv1251047279:84 , srv1263547578:84 , srv12717803:84 , srv1335167883:84 , srv1352199971:84 , srv1358303938:84 , srv1371320942:84 , srv1385515533:84 , srv1433342324:84 , srv1446895722:84 , srv1487286882:84 , srv150299660:84 , srv1510389266:84 , srv1531890363:84 , srv1545763870:84 , srv1546890970:84 , srv1601784604:84 , srv1627566024:84 , srv1645195934:84 , srv1654251301:84 , srv166002060:84 , srv1664204365:84 , srv1690085143:84 , srv171615464:84 , srv1730530190:84 , srv1780763216:84 , srv1784716098:84 , srv1790423519:84 , srv1799210384:84 , srv1815320582:84 , srv1815384190:84 , srv1820738364:84 , srv1845335775:84 , srv184720967:84 , srv1883531129:84 , srv1903998857:84 , srv1934871472:84 , srv1953952627:84 , srv196552656:84 , srv1984715017:84 , srv1993544607:84 , srv1995640818:84 , srv2026955242:84 , srv2055367488:84 , srv2060836616:84 , srv2082975282:84 , srv2111674424:84 , srv2118571088:84 , srv2125112280:84 , srv254946460:84 , srv258310192:84 , srv264958943:84 , srv291842323:84 , srv295923873:84 , srv369611767:84 , srv383918136:84 , srv406797463:84 , srv42189848:84 , srv437805336:84 , srv438230874:84 , srv454899917:84 , srv472208133:84 , srv473492188:84 , srv491920910:84 , srv534320921:84 , srv536317065:84 , srv556630341:84 , srv571515858:84 , srv582105459:84 , srv600557937:84 , srv611218482:84 , srv616159782:84 , srv63718912:84 , srv6812572:84 , srv70340170:84 , srv704662528:84 , srv705561700:84 , srv732320460:84 , srv747857555:84 , srv748811689:84 , srv78775113:84 , srv791292305:84 , srv80068901:84 , srv800954771:84 , srv861560231:84 , srv880591357:84 , srv899904709:84 , srv905409948:84 , srv918237231:84 , srv928817944:84 , srv1325392829:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-02-08 18:54:18,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-08 18:54:19,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv42189848=67, srv491920910=73, srv1780763216=35, srv78775113=90, srv1385515533=17, srv70340170=84, srv1487286882=20, srv2125112280=58, srv1690085143=32, srv861560231=94, srv295923873=63, srv166002060=30, srv258310192=60, srv536317065=75, srv556630341=76, srv1263547578=10, srv150299660=21, srv747857555=88, srv1093025275=3, srv1815320582=39, srv928817944=99, srv918237231=98, srv1546890970=25, srv1335167883=13, srv1358303938=15, srv1601784604=26, srv1993544607=50, srv1325392829=12, srv732320460=87, srv600557937=79, srv196552656=48, srv1934871472=46, srv2082975282=55, srv1446895722=19, srv1953952627=47, srv1091543353=2, srv383918136=65, srv1784716098=36, srv1664204365=31, srv2055367488=53, srv12717803=11, srv1227622242=8, srv1645195934=28, srv1015032081=0, srv1984715017=49, srv254946460=59, srv899904709=96, srv616159782=81, srv582105459=78, srv2118571088=57, srv1165283720=6, srv534320921=74, srv880591357=95, srv369611767=64, srv1227038938=7, srv1545763870=24, srv406797463=66, srv6812572=83, srv905409948=97, srv1903998857=45, srv1820738364=41, srv748811689=89, srv1251047279=9, srv1627566024=27, srv1510389266=22, srv63718912=82, srv184720967=43, srv1654251301=29, srv171615464=33, srv1371320942=16, srv705561700=86, srv1790423519=37, srv438230874=69, srv2060836616=54, srv1845335775=42, srv1883531129=44, srv472208133=71, srv571515858=77, srv1029436307=1, srv704662528=85, srv611218482=80, srv1165227440=5, srv1433342324=18, srv2026955242=52, srv80068901=92, srv454899917=70, srv1815384190=40, srv2111674424=56, srv1352199971=14, srv264958943=61, srv1799210384=38, srv291842323=62, srv473492188=72, srv1531890363=23, srv1730530190=34, srv791292305=91, srv1099148575=4, srv800954771=93, srv1995640818=51, srv437805336=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-08 18:54:19,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-08 18:54:19,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-08 18:54:19,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-08 18:54:19,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-08 18:54:19,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-08 18:54:19,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-08 18:54:19,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-08 18:54:19,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-08 18:54:19,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-08 18:54:19,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-08 18:54:19,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-08 18:54:19,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-08 18:54:19,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-08 18:54:19,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-08 18:54:19,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-08 18:54:19,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-08 18:54:19,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-08 18:54:19,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-08 18:54:19,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-08 18:54:19,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-08 18:54:19,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-08 18:54:19,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-08 18:54:19,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-08 18:54:19,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-08 18:54:19,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-08 18:54:19,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-08 18:54:19,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-08 18:54:19,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-08 18:54:19,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-08 18:54:19,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-08 18:54:19,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-08 18:54:19,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-08 18:54:19,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-08 18:54:19,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-08 18:54:19,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-08 18:54:19,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-08 18:54:19,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-08 18:54:19,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-08 18:54:19,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-08 18:54:19,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-08 18:54:19,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-08 18:54:19,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-08 18:54:19,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-08 18:54:19,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-08 18:54:19,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-08 18:54:19,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-08 18:54:19,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-08 18:54:19,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-08 18:54:19,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-08 18:54:19,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-08 18:54:19,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-08 18:54:19,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-08 18:54:19,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-08 18:54:19,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-08 18:54:19,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-08 18:54:19,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-08 18:54:19,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-08 18:54:19,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-08 18:54:19,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-08 18:54:19,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-08 18:54:19,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-08 18:54:19,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-08 18:54:19,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-08 18:54:19,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-08 18:54:19,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-08 18:54:19,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-08 18:54:19,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-08 18:54:19,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-08 18:54:19,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-08 18:54:19,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-08 18:54:19,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-08 18:54:19,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-08 18:54:19,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-08 18:54:19,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-08 18:54:19,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-08 18:54:19,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-08 18:54:19,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-08 18:54:19,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-08 18:54:19,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-08 18:54:19,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-08 18:54:19,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-08 18:54:19,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-08 18:54:19,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-08 18:54:19,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-08 18:54:19,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-08 18:54:19,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-08 18:54:19,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-08 18:54:19,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-08 18:54:19,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-08 18:54:19,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-08 18:54:19,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-08 18:54:19,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-08 18:54:19,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-08 18:54:19,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-08 18:54:19,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-08 18:54:19,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-08 18:54:19,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-08 18:54:19,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-08 18:54:19,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-08 18:54:19,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-08 18:54:19,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-08 18:54:19,422 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-08 18:54:19,422 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-08 18:54:19,423 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-08 18:54:19,818 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 705 ms to try 5000 different iterations.  Found a solution that moves 4675 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18114868441351997. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04038998071875813, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05741533560981273, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5194444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04038998071875813, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17633333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24341666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-08 18:54:19,864 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1385515533:81 , srv1546890970:81 , srv571515858:81 , srv42189848:82 , srv1029436307:83 , srv1165283720:83 , srv1227038938:83 , srv1335167883:83 , srv1780763216:83 , srv383918136:83 , srv70340170:83 , srv704662528:83 , srv705561700:83 , srv78775113:83 , srv1903998857:84 , srv1934871472:84 , srv2026955242:84 , srv473492188:84 , srv536317065:84 , srv1251047279:85 , srv1371320942:85 , srv1487286882:85 , srv150299660:85 , srv1654251301:85 , srv1664204365:85 , srv1815320582:85 , srv1820738364:85 , srv1984715017:85 , srv2125112280:85 , srv406797463:85 , srv582105459:85 , srv6812572:85 , srv732320460:85 , srv791292305:85 , srv928817944:85 , srv1015032081:86 , srv1091543353:86 , srv1093025275:86 , srv1227622242:86 , srv1263547578:86 , srv1358303938:86 , srv1446895722:86 , srv1730530190:86 , srv184720967:86 , srv1953952627:86 , srv264958943:86 , srv454899917:86 , srv491920910:86 , srv616159782:86 , srv748811689:86 , srv861560231:86 , srv899904709:86 , srv12717803:87 , srv1433342324:87 , srv1510389266:87 , srv1531890363:87 , srv166002060:87 , srv171615464:87 , srv1784716098:87 , srv1790423519:87 , srv1883531129:87 , srv196552656:87 , srv1993544607:87 , srv2111674424:87 , srv254946460:87 , srv291842323:87 , srv295923873:87 , srv472208133:87 , srv534320921:87 , srv600557937:87 , srv611218482:87 , srv747857555:87 , srv80068901:87 , srv918237231:87 , srv1165227440:88 , srv1545763870:88 , srv1645195934:88 , srv1690085143:88 , srv1815384190:88 , srv1845335775:88 , srv1995640818:88 , srv2060836616:88 , srv2082975282:88 , srv369611767:88 , srv800954771:88 , srv1099148575:89 , srv1601784604:89 , srv1799210384:89 , srv2118571088:89 , srv1627566024:90 , srv437805336:90 , srv2055367488:91 , srv556630341:91 , srv880591357:91 , srv905409948:91 , srv63718912:92 , srv1352199971:93 , srv438230874:93 , srv258310192:94 , srv1325392829:449 }
2024-02-08 18:54:19,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-08 18:54:19,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv42189848=67, srv491920910=73, srv1780763216=35, srv78775113=90, srv1385515533=17, srv70340170=84, srv1487286882=20, srv2125112280=58, srv1690085143=32, srv861560231=94, srv295923873=63, srv166002060=30, srv258310192=60, srv536317065=75, srv556630341=76, srv1263547578=10, srv150299660=21, srv747857555=88, srv1093025275=3, srv1815320582=39, srv928817944=99, srv918237231=98, srv1546890970=25, srv1335167883=13, srv1358303938=15, srv1601784604=26, srv1993544607=50, srv1325392829=12, srv732320460=87, srv600557937=79, srv196552656=48, srv1934871472=46, srv2082975282=55, srv1446895722=19, srv1953952627=47, srv1091543353=2, srv383918136=65, srv1784716098=36, srv1664204365=31, srv2055367488=53, srv12717803=11, srv1227622242=8, srv1645195934=28, srv1015032081=0, srv1984715017=49, srv254946460=59, srv899904709=96, srv616159782=81, srv582105459=78, srv2118571088=57, srv1165283720=6, srv534320921=74, srv880591357=95, srv369611767=64, srv1227038938=7, srv1545763870=24, srv406797463=66, srv6812572=83, srv905409948=97, srv1903998857=45, srv1820738364=41, srv748811689=89, srv1251047279=9, srv1627566024=27, srv1510389266=22, srv63718912=82, srv184720967=43, srv1654251301=29, srv171615464=33, srv1371320942=16, srv705561700=86, srv1790423519=37, srv438230874=69, srv2060836616=54, srv1845335775=42, srv1883531129=44, srv472208133=71, srv571515858=77, srv1029436307=1, srv704662528=85, srv611218482=80, srv1165227440=5, srv1433342324=18, srv2026955242=52, srv80068901=92, srv454899917=70, srv1815384190=40, srv2111674424=56, srv1352199971=14, srv264958943=61, srv1799210384=38, srv291842323=62, srv473492188=72, srv1531890363=23, srv1730530190=34, srv791292305=91, srv1099148575=4, srv800954771=93, srv1995640818=51, srv437805336=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-08 18:54:19,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-08 18:54:19,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-08 18:54:19,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-08 18:54:19,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-08 18:54:19,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-08 18:54:19,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-08 18:54:19,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-08 18:54:19,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-08 18:54:19,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-08 18:54:19,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-08 18:54:19,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-08 18:54:19,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-08 18:54:19,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-08 18:54:19,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-08 18:54:19,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-08 18:54:19,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-08 18:54:19,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-08 18:54:19,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-08 18:54:19,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-08 18:54:19,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-08 18:54:19,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-08 18:54:19,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-08 18:54:19,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-08 18:54:19,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-08 18:54:19,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-08 18:54:19,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-08 18:54:19,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-08 18:54:19,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-08 18:54:19,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-08 18:54:19,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-08 18:54:19,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-08 18:54:19,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-08 18:54:19,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-08 18:54:19,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-08 18:54:19,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-08 18:54:19,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-08 18:54:19,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-08 18:54:19,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-08 18:54:19,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-08 18:54:19,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-08 18:54:19,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-08 18:54:19,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-08 18:54:19,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-08 18:54:19,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-08 18:54:19,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-08 18:54:19,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-08 18:54:19,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-08 18:54:19,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-08 18:54:19,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-08 18:54:19,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-08 18:54:19,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-08 18:54:19,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-08 18:54:19,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-08 18:54:19,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-08 18:54:19,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-08 18:54:19,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-08 18:54:19,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-08 18:54:19,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-08 18:54:19,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-08 18:54:19,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-08 18:54:19,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-08 18:54:19,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-08 18:54:19,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-08 18:54:19,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-08 18:54:19,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-08 18:54:19,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-08 18:54:19,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-08 18:54:19,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-08 18:54:19,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-08 18:54:19,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-08 18:54:19,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-08 18:54:19,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-08 18:54:19,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-08 18:54:19,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-08 18:54:19,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-08 18:54:19,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-08 18:54:19,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-08 18:54:19,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-08 18:54:19,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-08 18:54:19,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-08 18:54:19,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-08 18:54:19,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-08 18:54:19,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-08 18:54:19,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-08 18:54:19,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-08 18:54:19,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-08 18:54:19,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-08 18:54:19,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-08 18:54:19,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-08 18:54:19,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-08 18:54:19,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-08 18:54:19,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-08 18:54:19,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-08 18:54:19,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-08 18:54:19,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-08 18:54:19,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-08 18:54:19,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-08 18:54:19,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-08 18:54:19,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-08 18:54:19,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-08 18:54:19,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-08 18:54:20,068 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04038998071875813, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05741533560981273, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04038998071875813, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17633333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24341666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-08 18:54:20,069 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-08 18:54:20,069 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18111594936385972, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04038998071875813, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05741533560981273, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04038998071875813, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17633333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24341666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-08 18:54:20,310 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 365 ms to try 5000 different iterations.  Found a solution that moves 2655 regions; Going from a computed imbalance of 0.18111594936385972 to a new imbalance of 0.01643267572133507. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022283305498805854); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035944838647081985, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.295, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022283305498805854); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01325); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04683333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-08 18:54:20,324 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv258310192:77 , srv1352199971:79 , srv437805336:80 , srv556630341:80 , srv905409948:80 , srv2055367488:81 , srv63718912:81 , srv1627566024:82 , srv1799210384:82 , srv2082975282:82 , srv880591357:82 , srv1099148575:83 , srv1510389266:83 , srv1601784604:83 , srv1690085143:83 , srv1883531129:83 , srv1993544607:83 , srv1995640818:83 , srv369611767:83 , srv438230874:83 , srv747857555:83 , srv800954771:83 , srv1091543353:84 , srv12717803:84 , srv1545763870:84 , srv166002060:84 , srv1784716098:84 , srv2111674424:84 , srv472208133:84 , srv491920910:84 , srv534320921:84 , srv1165227440:85 , srv1227622242:85 , srv1358303938:85 , srv1433342324:85 , srv150299660:85 , srv171615464:85 , srv1790423519:85 , srv1815384190:85 , srv1845335775:85 , srv295923873:85 , srv454899917:85 , srv582105459:85 , srv611218482:85 , srv732320460:85 , srv1093025275:86 , srv1263547578:86 , srv1446895722:86 , srv1487286882:86 , srv1531890363:86 , srv1664204365:86 , srv1730530190:86 , srv184720967:86 , srv1953952627:86 , srv196552656:86 , srv2060836616:86 , srv406797463:86 , srv473492188:86 , srv616159782:86 , srv748811689:86 , srv80068901:86 , srv918237231:86 , srv1165283720:87 , srv1251047279:87 , srv1371320942:87 , srv1645195934:87 , srv1654251301:87 , srv1815320582:87 , srv1820738364:87 , srv2118571088:87 , srv2125112280:87 , srv264958943:87 , srv600557937:87 , srv6812572:87 , srv704662528:87 , srv791292305:87 , srv899904709:87 , srv928817944:87 , srv1015032081:88 , srv291842323:88 , srv70340170:88 , srv861560231:88 , srv1335167883:89 , srv1546890970:89 , srv1780763216:89 , srv1903998857:89 , srv1934871472:89 , srv2026955242:89 , srv254946460:89 , srv383918136:89 , srv571515858:89 , srv78775113:89 , srv1385515533:90 , srv1984715017:90 , srv536317065:90 , srv705561700:90 , srv1029436307:91 , srv1227038938:92 , srv42189848:93 , srv1325392829:523 }
2024-02-08 18:54:20,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-08 18:54:20,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv42189848=67, srv491920910=73, srv1780763216=35, srv78775113=90, srv1385515533=17, srv70340170=84, srv1487286882=20, srv2125112280=58, srv1690085143=32, srv861560231=94, srv295923873=63, srv166002060=30, srv258310192=60, srv536317065=75, srv556630341=76, srv1263547578=10, srv150299660=21, srv747857555=88, srv1093025275=3, srv1815320582=39, srv928817944=99, srv918237231=98, srv1546890970=25, srv1335167883=13, srv1358303938=15, srv1601784604=26, srv1993544607=50, srv1325392829=12, srv732320460=87, srv600557937=79, srv196552656=48, srv1934871472=46, srv2082975282=55, srv1446895722=19, srv1953952627=47, srv1091543353=2, srv383918136=65, srv1784716098=36, srv1664204365=31, srv2055367488=53, srv12717803=11, srv1227622242=8, srv1645195934=28, srv1015032081=0, srv1984715017=49, srv254946460=59, srv899904709=96, srv616159782=81, srv582105459=78, srv2118571088=57, srv1165283720=6, srv534320921=74, srv880591357=95, srv369611767=64, srv1227038938=7, srv1545763870=24, srv406797463=66, srv6812572=83, srv905409948=97, srv1903998857=45, srv1820738364=41, srv748811689=89, srv1251047279=9, srv1627566024=27, srv1510389266=22, srv63718912=82, srv184720967=43, srv1654251301=29, srv171615464=33, srv1371320942=16, srv705561700=86, srv1790423519=37, srv438230874=69, srv2060836616=54, srv1845335775=42, srv1883531129=44, srv472208133=71, srv571515858=77, srv1029436307=1, srv704662528=85, srv611218482=80, srv1165227440=5, srv1433342324=18, srv2026955242=52, srv80068901=92, srv454899917=70, srv1815384190=40, srv2111674424=56, srv1352199971=14, srv264958943=61, srv1799210384=38, srv291842323=62, srv473492188=72, srv1531890363=23, srv1730530190=34, srv791292305=91, srv1099148575=4, srv800954771=93, srv1995640818=51, srv437805336=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-08 18:54:20,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-08 18:54:20,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-08 18:54:20,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-08 18:54:20,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-08 18:54:20,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-08 18:54:20,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-08 18:54:20,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-08 18:54:20,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-08 18:54:20,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-08 18:54:20,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-08 18:54:20,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-08 18:54:20,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-08 18:54:20,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-08 18:54:20,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-08 18:54:20,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-08 18:54:20,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-08 18:54:20,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-08 18:54:20,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-08 18:54:20,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-08 18:54:20,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-08 18:54:20,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-08 18:54:20,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-08 18:54:20,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-08 18:54:20,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-08 18:54:20,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-08 18:54:20,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-08 18:54:20,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-08 18:54:20,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-08 18:54:20,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-08 18:54:20,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-08 18:54:20,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-08 18:54:20,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-08 18:54:20,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-08 18:54:20,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-08 18:54:20,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-08 18:54:20,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-08 18:54:20,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-08 18:54:20,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-08 18:54:20,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-08 18:54:20,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-08 18:54:20,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-08 18:54:20,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-08 18:54:20,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-08 18:54:20,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-08 18:54:20,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-08 18:54:20,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-08 18:54:20,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-08 18:54:20,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-08 18:54:20,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-08 18:54:20,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-08 18:54:20,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-08 18:54:20,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-08 18:54:20,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-08 18:54:20,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-08 18:54:20,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-08 18:54:20,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-08 18:54:20,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-08 18:54:20,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-08 18:54:20,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-08 18:54:20,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-08 18:54:20,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-08 18:54:20,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-08 18:54:20,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-08 18:54:20,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-08 18:54:20,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-08 18:54:20,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-08 18:54:20,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-08 18:54:20,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-08 18:54:20,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-08 18:54:20,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-08 18:54:20,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-08 18:54:20,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-08 18:54:20,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-08 18:54:20,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-08 18:54:20,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-08 18:54:20,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-08 18:54:20,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-08 18:54:20,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-08 18:54:20,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-08 18:54:20,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-08 18:54:20,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-08 18:54:20,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-08 18:54:20,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-08 18:54:20,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-08 18:54:20,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-08 18:54:20,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-08 18:54:20,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-08 18:54:20,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-08 18:54:20,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-08 18:54:20,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-08 18:54:20,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-08 18:54:20,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-08 18:54:20,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-08 18:54:20,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-08 18:54:20,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-08 18:54:20,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-08 18:54:20,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-08 18:54:20,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-08 18:54:20,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-08 18:54:20,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-08 18:54:20,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-08 18:54:20,491 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022283305498805854); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035944838647081985, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022283305498805854); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01325); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04683333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-08 18:54:20,492 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-08 18:54:20,492 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016414085013987913, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022283305498805854); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035944838647081985, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022283305498805854); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01325); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04683333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-08 18:54:20,721 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 338 ms to try 5000 different iterations.  Found a solution that moves 962 regions; Going from a computed imbalance of 0.016414085013987913 to a new imbalance of 9.132100278716315E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0063288916631981705); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011271568674394604); MoveCostFunction : (multiplier=7.0, imbalance=0.1068888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0063288916631981705); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-08 18:54:20,723 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1645195934:82 , srv2060836616:82 , srv254946460:82 , srv291842323:82 , srv42189848:82 , srv438230874:82 , srv1227038938:83 , srv1845335775:83 , srv1984715017:83 , srv2118571088:83 , srv536317065:83 , srv918237231:83 , srv1015032081:84 , srv1029436307:84 , srv1352199971:84 , srv1531890363:84 , srv1815384190:84 , srv1934871472:84 , srv2026955242:84 , srv600557937:84 , srv705561700:84 , srv861560231:84 , srv899904709:84 , srv1099148575:85 , srv1165227440:85 , srv1251047279:85 , srv1371320942:85 , srv1433342324:85 , srv1601784604:85 , srv1627566024:85 , srv1654251301:85 , srv1730530190:85 , srv1790423519:85 , srv1820738364:85 , srv1903998857:85 , srv196552656:85 , srv616159782:85 , srv63718912:85 , srv6812572:85 , srv70340170:85 , srv78775113:85 , srv80068901:85 , srv880591357:85 , srv1093025275:86 , srv1263547578:86 , srv12717803:86 , srv1335167883:86 , srv1358303938:86 , srv1385515533:86 , srv1446895722:86 , srv1487286882:86 , srv1545763870:86 , srv166002060:86 , srv1664204365:86 , srv1690085143:86 , srv171615464:86 , srv1780763216:86 , srv1784716098:86 , srv1799210384:86 , srv184720967:86 , srv1953952627:86 , srv1995640818:86 , srv2055367488:86 , srv2111674424:86 , srv2125112280:86 , srv258310192:86 , srv264958943:86 , srv295923873:86 , srv369611767:86 , srv383918136:86 , srv454899917:86 , srv534320921:86 , srv611218482:86 , srv800954771:86 , srv928817944:86 , srv1091543353:87 , srv1165283720:87 , srv1227622242:87 , srv1510389266:87 , srv1546890970:87 , srv1815320582:87 , srv1993544607:87 , srv2082975282:87 , srv406797463:87 , srv437805336:87 , srv472208133:87 , srv473492188:87 , srv491920910:87 , srv556630341:87 , srv582105459:87 , srv704662528:87 , srv747857555:87 , srv748811689:87 , srv791292305:87 , srv905409948:87 , srv150299660:88 , srv1883531129:88 , srv571515858:88 , srv732320460:88 , srv1325392829:542 }
2024-02-08 18:54:20,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-08 18:54:20,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv42189848=67, srv491920910=73, srv1780763216=35, srv78775113=90, srv1385515533=17, srv70340170=84, srv1487286882=20, srv2125112280=58, srv1690085143=32, srv861560231=94, srv295923873=63, srv166002060=30, srv258310192=60, srv536317065=75, srv556630341=76, srv1263547578=10, srv150299660=21, srv747857555=88, srv1093025275=3, srv1815320582=39, srv928817944=99, srv918237231=98, srv1546890970=25, srv1335167883=13, srv1358303938=15, srv1601784604=26, srv1993544607=50, srv1325392829=12, srv732320460=87, srv600557937=79, srv196552656=48, srv1934871472=46, srv2082975282=55, srv1446895722=19, srv1953952627=47, srv1091543353=2, srv383918136=65, srv1784716098=36, srv1664204365=31, srv2055367488=53, srv12717803=11, srv1227622242=8, srv1645195934=28, srv1015032081=0, srv1984715017=49, srv254946460=59, srv899904709=96, srv616159782=81, srv582105459=78, srv2118571088=57, srv1165283720=6, srv534320921=74, srv880591357=95, srv369611767=64, srv1227038938=7, srv1545763870=24, srv406797463=66, srv6812572=83, srv905409948=97, srv1903998857=45, srv1820738364=41, srv748811689=89, srv1251047279=9, srv1627566024=27, srv1510389266=22, srv63718912=82, srv184720967=43, srv1654251301=29, srv171615464=33, srv1371320942=16, srv705561700=86, srv1790423519=37, srv438230874=69, srv2060836616=54, srv1845335775=42, srv1883531129=44, srv472208133=71, srv571515858=77, srv1029436307=1, srv704662528=85, srv611218482=80, srv1165227440=5, srv1433342324=18, srv2026955242=52, srv80068901=92, srv454899917=70, srv1815384190=40, srv2111674424=56, srv1352199971=14, srv264958943=61, srv1799210384=38, srv291842323=62, srv473492188=72, srv1531890363=23, srv1730530190=34, srv791292305=91, srv1099148575=4, srv800954771=93, srv1995640818=51, srv437805336=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-08 18:54:20,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-08 18:54:20,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-08 18:54:20,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-08 18:54:20,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-08 18:54:20,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-08 18:54:20,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-08 18:54:20,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-08 18:54:20,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-08 18:54:20,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-08 18:54:20,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-08 18:54:20,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-08 18:54:20,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-08 18:54:20,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-08 18:54:20,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-08 18:54:20,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-08 18:54:20,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-08 18:54:20,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-08 18:54:20,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-08 18:54:20,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-08 18:54:20,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-08 18:54:20,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-08 18:54:20,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-08 18:54:20,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-08 18:54:20,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-08 18:54:20,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-08 18:54:20,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-08 18:54:20,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-08 18:54:20,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-08 18:54:20,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-08 18:54:20,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-08 18:54:20,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-08 18:54:20,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-08 18:54:20,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-08 18:54:20,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-08 18:54:20,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-08 18:54:20,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-08 18:54:20,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-08 18:54:20,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-08 18:54:20,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-08 18:54:20,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-08 18:54:20,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-08 18:54:20,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-08 18:54:20,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-08 18:54:20,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-08 18:54:20,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-08 18:54:20,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-08 18:54:20,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-08 18:54:20,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-08 18:54:20,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-08 18:54:20,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-08 18:54:20,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-08 18:54:20,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-08 18:54:20,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-08 18:54:20,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-08 18:54:20,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-08 18:54:20,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-08 18:54:20,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-08 18:54:20,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-08 18:54:20,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-08 18:54:20,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-08 18:54:20,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-08 18:54:20,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-08 18:54:20,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-08 18:54:20,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-08 18:54:20,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-08 18:54:20,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-08 18:54:20,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-08 18:54:20,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-08 18:54:20,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-08 18:54:20,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-08 18:54:20,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-08 18:54:20,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-08 18:54:20,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-08 18:54:20,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-08 18:54:20,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-08 18:54:20,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-08 18:54:20,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-08 18:54:20,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-08 18:54:20,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-08 18:54:20,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-08 18:54:20,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-08 18:54:20,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-08 18:54:20,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-08 18:54:20,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-08 18:54:20,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-08 18:54:20,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-08 18:54:20,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-08 18:54:20,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-08 18:54:20,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-08 18:54:20,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-08 18:54:20,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-08 18:54:20,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-08 18:54:20,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-08 18:54:20,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-08 18:54:20,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-08 18:54:20,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-08 18:54:20,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-08 18:54:20,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-08 18:54:20,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-08 18:54:20,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-08 18:54:20,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-08 18:54:20,829 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0063288916631981705); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011271568674394604); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0063288916631981705); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-08 18:54:20,829 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-08 18:54:20,829 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.064739598987639E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0063288916631981705); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011271568674394604); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0063288916631981705); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-08 18:54:21,058 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 307 ms to try 5000 different iterations.  Found a solution that moves 287 regions; Going from a computed imbalance of 9.064739598987639E-4 to a new imbalance of 2.031759956649894E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.03188888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-08 18:54:21,059 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1227622242:83 , srv1815320582:83 , srv2118571088:83 , srv264958943:83 , srv748811689:83 , srv791292305:83 , srv80068901:83 , srv1015032081:84 , srv1029436307:84 , srv1093025275:84 , srv1165227440:84 , srv1263547578:84 , srv1335167883:84 , srv1371320942:84 , srv1446895722:84 , srv150299660:84 , srv1545763870:84 , srv171615464:84 , srv1780763216:84 , srv184720967:84 , srv1883531129:84 , srv1903998857:84 , srv1953952627:84 , srv196552656:84 , srv1984715017:84 , srv2055367488:84 , srv2125112280:84 , srv254946460:84 , srv258310192:84 , srv295923873:84 , srv369611767:84 , srv383918136:84 , srv406797463:84 , srv438230874:84 , srv472208133:84 , srv556630341:84 , srv600557937:84 , srv611218482:84 , srv63718912:84 , srv732320460:84 , srv880591357:84 , srv905409948:84 , srv928817944:84 , srv1091543353:85 , srv1099148575:85 , srv1165283720:85 , srv1227038938:85 , srv1251047279:85 , srv1358303938:85 , srv1385515533:85 , srv1487286882:85 , srv1510389266:85 , srv1531890363:85 , srv1546890970:85 , srv1601784604:85 , srv1645195934:85 , srv1654251301:85 , srv166002060:85 , srv1664204365:85 , srv1690085143:85 , srv1730530190:85 , srv1784716098:85 , srv1790423519:85 , srv1799210384:85 , srv1815384190:85 , srv1820738364:85 , srv1934871472:85 , srv1993544607:85 , srv1995640818:85 , srv2026955242:85 , srv2060836616:85 , srv2082975282:85 , srv2111674424:85 , srv291842323:85 , srv42189848:85 , srv437805336:85 , srv454899917:85 , srv473492188:85 , srv491920910:85 , srv534320921:85 , srv571515858:85 , srv582105459:85 , srv616159782:85 , srv6812572:85 , srv70340170:85 , srv704662528:85 , srv705561700:85 , srv747857555:85 , srv78775113:85 , srv800954771:85 , srv861560231:85 , srv899904709:85 , srv12717803:86 , srv1352199971:86 , srv1433342324:86 , srv1627566024:86 , srv1845335775:86 , srv536317065:86 , srv918237231:86 , srv1325392829:628 }
2024-02-08 18:54:21,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-08 18:54:21,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv42189848=67, srv491920910=73, srv1780763216=35, srv78775113=90, srv1385515533=17, srv70340170=84, srv1487286882=20, srv2125112280=58, srv1690085143=32, srv861560231=94, srv295923873=63, srv166002060=30, srv258310192=60, srv536317065=75, srv556630341=76, srv1263547578=10, srv150299660=21, srv747857555=88, srv1093025275=3, srv1815320582=39, srv928817944=99, srv918237231=98, srv1546890970=25, srv1335167883=13, srv1358303938=15, srv1601784604=26, srv1993544607=50, srv1325392829=12, srv732320460=87, srv600557937=79, srv196552656=48, srv1934871472=46, srv2082975282=55, srv1446895722=19, srv1953952627=47, srv1091543353=2, srv383918136=65, srv1784716098=36, srv1664204365=31, srv2055367488=53, srv12717803=11, srv1227622242=8, srv1645195934=28, srv1015032081=0, srv1984715017=49, srv254946460=59, srv899904709=96, srv616159782=81, srv582105459=78, srv2118571088=57, srv1165283720=6, srv534320921=74, srv880591357=95, srv369611767=64, srv1227038938=7, srv1545763870=24, srv406797463=66, srv6812572=83, srv905409948=97, srv1903998857=45, srv1820738364=41, srv748811689=89, srv1251047279=9, srv1627566024=27, srv1510389266=22, srv63718912=82, srv184720967=43, srv1654251301=29, srv171615464=33, srv1371320942=16, srv705561700=86, srv1790423519=37, srv438230874=69, srv2060836616=54, srv1845335775=42, srv1883531129=44, srv472208133=71, srv571515858=77, srv1029436307=1, srv704662528=85, srv611218482=80, srv1165227440=5, srv1433342324=18, srv2026955242=52, srv80068901=92, srv454899917=70, srv1815384190=40, srv2111674424=56, srv1352199971=14, srv264958943=61, srv1799210384=38, srv291842323=62, srv473492188=72, srv1531890363=23, srv1730530190=34, srv791292305=91, srv1099148575=4, srv800954771=93, srv1995640818=51, srv437805336=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-08 18:54:21,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-08 18:54:21,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-08 18:54:21,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-08 18:54:21,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-08 18:54:21,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-08 18:54:21,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-08 18:54:21,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-08 18:54:21,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-08 18:54:21,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-08 18:54:21,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-08 18:54:21,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-08 18:54:21,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-08 18:54:21,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-08 18:54:21,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-08 18:54:21,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-08 18:54:21,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-08 18:54:21,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-08 18:54:21,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-08 18:54:21,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-08 18:54:21,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-08 18:54:21,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-08 18:54:21,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-08 18:54:21,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-08 18:54:21,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-08 18:54:21,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-08 18:54:21,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-08 18:54:21,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-08 18:54:21,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-08 18:54:21,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-08 18:54:21,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-08 18:54:21,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-08 18:54:21,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-08 18:54:21,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-08 18:54:21,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-08 18:54:21,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-08 18:54:21,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-08 18:54:21,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-08 18:54:21,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-08 18:54:21,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-08 18:54:21,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-08 18:54:21,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-08 18:54:21,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-08 18:54:21,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-08 18:54:21,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-08 18:54:21,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-08 18:54:21,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-08 18:54:21,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-08 18:54:21,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-08 18:54:21,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-08 18:54:21,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-08 18:54:21,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-08 18:54:21,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-08 18:54:21,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-08 18:54:21,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-08 18:54:21,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-08 18:54:21,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-08 18:54:21,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-08 18:54:21,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-08 18:54:21,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-08 18:54:21,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-08 18:54:21,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-08 18:54:21,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-08 18:54:21,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-08 18:54:21,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-08 18:54:21,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-08 18:54:21,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-08 18:54:21,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-08 18:54:21,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-08 18:54:21,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-08 18:54:21,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-08 18:54:21,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-08 18:54:21,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-08 18:54:21,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-08 18:54:21,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-08 18:54:21,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-08 18:54:21,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-08 18:54:21,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-08 18:54:21,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-08 18:54:21,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-08 18:54:21,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-08 18:54:21,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-08 18:54:21,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-08 18:54:21,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-08 18:54:21,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-08 18:54:21,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-08 18:54:21,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-08 18:54:21,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-08 18:54:21,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-08 18:54:21,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-08 18:54:21,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-08 18:54:21,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-08 18:54:21,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-08 18:54:21,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-08 18:54:21,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-08 18:54:21,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-08 18:54:21,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-08 18:54:21,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-08 18:54:21,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-08 18:54:21,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-08 18:54:21,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-08 18:54:21,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-08 18:54:21,163 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.8307982614094552E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-08 18:54:21,163 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1227622242:83 , srv1815320582:83 , srv2118571088:83 , srv264958943:83 , srv748811689:83 , srv791292305:83 , srv80068901:83 , srv1015032081:84 , srv1029436307:84 , srv1093025275:84 , srv1165227440:84 , srv1263547578:84 , srv1335167883:84 , srv1371320942:84 , srv1446895722:84 , srv150299660:84 , srv1545763870:84 , srv171615464:84 , srv1780763216:84 , srv184720967:84 , srv1883531129:84 , srv1903998857:84 , srv1953952627:84 , srv196552656:84 , srv1984715017:84 , srv2055367488:84 , srv2125112280:84 , srv254946460:84 , srv258310192:84 , srv295923873:84 , srv369611767:84 , srv383918136:84 , srv406797463:84 , srv438230874:84 , srv472208133:84 , srv556630341:84 , srv600557937:84 , srv611218482:84 , srv63718912:84 , srv732320460:84 , srv880591357:84 , srv905409948:84 , srv928817944:84 , srv1091543353:85 , srv1099148575:85 , srv1165283720:85 , srv1227038938:85 , srv1251047279:85 , srv1358303938:85 , srv1385515533:85 , srv1487286882:85 , srv1510389266:85 , srv1531890363:85 , srv1546890970:85 , srv1601784604:85 , srv1645195934:85 , srv1654251301:85 , srv166002060:85 , srv1664204365:85 , srv1690085143:85 , srv1730530190:85 , srv1784716098:85 , srv1790423519:85 , srv1799210384:85 , srv1815384190:85 , srv1820738364:85 , srv1934871472:85 , srv1993544607:85 , srv1995640818:85 , srv2026955242:85 , srv2060836616:85 , srv2082975282:85 , srv2111674424:85 , srv291842323:85 , srv42189848:85 , srv437805336:85 , srv454899917:85 , srv473492188:85 , srv491920910:85 , srv534320921:85 , srv571515858:85 , srv582105459:85 , srv616159782:85 , srv6812572:85 , srv70340170:85 , srv704662528:85 , srv705561700:85 , srv747857555:85 , srv78775113:85 , srv800954771:85 , srv861560231:85 , srv899904709:85 , srv12717803:86 , srv1352199971:86 , srv1433342324:86 , srv1627566024:86 , srv1845335775:86 , srv536317065:86 , srv918237231:86 , srv1325392829:628 }
2024-02-08 18:54:21,190 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=308 (was 231) - SystemLoadAverage LEAK? -, ProcessCount=176 (was 176), AvailableMemoryMB=1625 (was 2255)
2024-02-08 18:54:21,199 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=308, ProcessCount=176, AvailableMemoryMB=1625
2024-02-08 18:54:21,200 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-02-08 18:54:21,200 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-02-08 18:54:21,206 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1184202769:84 , srv1208798643:84 , srv1220326219:84 , srv1238472704:84 , srv1367415727:84 , srv1394019725:84 , srv1401983316:84 , srv1503805413:84 , srv1513467274:84 , srv1524790105:84 , srv1741478081:84 , srv1902403575:84 , srv2020929830:84 , srv2024863461:84 , srv2068577110:84 , srv2074300074:84 , srv2099736974:84 , srv2121644264:84 , srv272047239:84 , srv272088035:84 , srv33365309:84 , srv384948612:84 , srv496786768:84 , srv552032506:84 , srv636915718:84 , srv810438794:84 , srv845142609:84 , srv964731956:84 , srv996065204:84 , srv1782595143:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-02-08 18:54:21,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-08 18:54:21,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1184202769=0, srv1208798643=1, srv1902403575=12, srv552032506=24, srv636915718=25, srv1220326219=2, srv996065204=29, srv272088035=20, srv2121644264=18, srv1513467274=8, srv1782595143=11, srv2020929830=13, srv1741478081=10, srv845142609=27, srv2099736974=17, srv1238472704=3, srv1503805413=7, srv1524790105=9, srv1401983316=6, srv1367415727=4, srv2068577110=15, srv272047239=19, srv33365309=21, srv384948612=22, srv1394019725=5, srv2074300074=16, srv810438794=26, srv964731956=28, srv2024863461=14, srv496786768=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-08 18:54:21,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-08 18:54:21,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-08 18:54:21,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-08 18:54:21,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-08 18:54:21,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-08 18:54:21,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-08 18:54:21,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-08 18:54:21,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-08 18:54:21,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-08 18:54:21,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-08 18:54:21,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-08 18:54:21,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-08 18:54:21,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-08 18:54:21,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-08 18:54:21,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-08 18:54:21,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-08 18:54:21,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-08 18:54:21,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-08 18:54:21,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-08 18:54:21,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-08 18:54:21,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-08 18:54:21,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-08 18:54:21,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-08 18:54:21,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-08 18:54:21,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-08 18:54:21,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-08 18:54:21,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-08 18:54:21,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-08 18:54:21,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-08 18:54:21,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-08 18:54:21,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-02-08 18:54:21,232 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0922279409692202, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-08 18:54:21,232 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912483650494754, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0922279409692202, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-02-08 18:56:21,221 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9720967 different iterations.  Found a solution that moves 1841 regions; Going from a computed imbalance of 0.9912483650494754 to a new imbalance of 4.296985841319951E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6818518518518518, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-08 18:56:21,226 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1184202769:90 , srv1208798643:90 , srv1220326219:90 , srv1238472704:90 , srv1367415727:90 , srv1394019725:90 , srv1401983316:90 , srv1503805413:90 , srv1513467274:90 , srv1524790105:90 , srv1741478081:90 , srv1782595143:90 , srv1902403575:90 , srv2020929830:90 , srv2024863461:90 , srv2068577110:90 , srv2074300074:90 , srv2099736974:90 , srv2121644264:90 , srv272047239:90 , srv272088035:90 , srv33365309:90 , srv384948612:90 , srv496786768:90 , srv552032506:90 , srv636915718:90 , srv810438794:90 , srv845142609:90 , srv964731956:90 , srv996065204:90 }
2024-02-08 18:56:21,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-08 18:56:21,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1184202769=0, srv1208798643=1, srv1902403575=12, srv552032506=24, srv636915718=25, srv1220326219=2, srv996065204=29, srv272088035=20, srv2121644264=18, srv1513467274=8, srv1782595143=11, srv2020929830=13, srv1741478081=10, srv845142609=27, srv2099736974=17, srv1238472704=3, srv1503805413=7, srv1524790105=9, srv1401983316=6, srv1367415727=4, srv2068577110=15, srv272047239=19, srv33365309=21, srv384948612=22, srv1394019725=5, srv2074300074=16, srv810438794=26, srv964731956=28, srv2024863461=14, srv496786768=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-08 18:56:21,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-08 18:56:21,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-08 18:56:21,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-08 18:56:21,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-08 18:56:21,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-08 18:56:21,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-08 18:56:21,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-08 18:56:21,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-08 18:56:21,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-08 18:56:21,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-08 18:56:21,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-08 18:56:21,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-08 18:56:21,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-08 18:56:21,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-08 18:56:21,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-08 18:56:21,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-08 18:56:21,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-08 18:56:21,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-08 18:56:21,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-08 18:56:21,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-08 18:56:21,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-08 18:56:21,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-08 18:56:21,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-08 18:56:21,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-08 18:56:21,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-08 18:56:21,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-08 18:56:21,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-08 18:56:21,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-08 18:56:21,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-08 18:56:21,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-08 18:56:21,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-02-08 18:56:21,242 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-08 18:56:21,242 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1184202769:90 , srv1208798643:90 , srv1220326219:90 , srv1238472704:90 , srv1367415727:90 , srv1394019725:90 , srv1401983316:90 , srv1503805413:90 , srv1513467274:90 , srv1524790105:90 , srv1741478081:90 , srv1782595143:90 , srv1902403575:90 , srv2020929830:90 , srv2024863461:90 , srv2068577110:90 , srv2074300074:90 , srv2099736974:90 , srv2121644264:90 , srv272047239:90 , srv272088035:90 , srv33365309:90 , srv384948612:90 , srv496786768:90 , srv552032506:90 , srv636915718:90 , srv810438794:90 , srv845142609:90 , srv964731956:90 , srv996065204:90 }
2024-02-08 18:56:21,254 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=137 (was 308), ProcessCount=174 (was 176), AvailableMemoryMB=2827 (was 1625) - AvailableMemoryMB LEAK? -