2024-03-07 04:54:20,809 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-03-07 04:54:21,071 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-03-07 04:54:21,166 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-03-07 04:54:21,222 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-07 04:54:21,231 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-07 04:54:21,245 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=189, ProcessCount=170, AvailableMemoryMB=6510
2024-03-07 04:54:21,247 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-07 04:54:21,247 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-07 04:54:21,372 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1036970899:84 , srv1051846249:84 , srv1068987408:84 , srv1123495896:84 , srv1124559248:84 , srv1135598707:84 , srv1137104830:84 , srv1162560054:84 , srv119822529:84 , srv1230228323:84 , srv1230989033:84 , srv1233656980:84 , srv1285069689:84 , srv1288652821:84 , srv1326205158:84 , srv1346721142:84 , srv1367633437:84 , srv13734721:84 , srv1374847270:84 , srv1471517137:84 , srv1475572274:84 , srv1477999159:84 , srv1505450355:84 , srv1527249116:84 , srv1533958817:84 , srv1534632846:84 , srv1537401451:84 , srv1539018993:84 , srv1544858105:84 , srv1555612692:84 , srv1574603706:84 , srv1602091423:84 , srv165447195:84 , srv167226055:84 , srv1672914050:84 , srv1689979024:84 , srv1691757429:84 , srv1745107032:84 , srv177306947:84 , srv1802656065:84 , srv1809456180:84 , srv1823413518:84 , srv1842854006:84 , srv1847088714:84 , srv1858170078:84 , srv1903350203:84 , srv1905924867:84 , srv1920061116:84 , srv1925552451:84 , srv1930660733:84 , srv1986679677:84 , srv202605596:84 , srv2036426874:84 , srv2040254101:84 , srv2072064604:84 , srv2098416321:84 , srv211275071:84 , srv2121496804:84 , srv2132312333:84 , srv213514393:84 , srv2135913787:84 , srv2145658593:84 , srv222350470:84 , srv223267640:84 , srv2312308:84 , srv256087933:84 , srv265557199:84 , srv26885299:84 , srv287835896:84 , srv338220873:84 , srv342359351:84 , srv389496868:84 , srv409249040:84 , srv435922076:84 , srv44388921:84 , srv496744104:84 , srv525341559:84 , srv530668357:84 , srv559613944:84 , srv574601601:84 , srv594431942:84 , srv64885281:84 , srv680846639:84 , srv693769448:84 , srv6937753:84 , srv696472892:84 , srv730305388:84 , srv790968422:84 , srv803445452:84 , srv821140557:84 , srv827120472:84 , srv88030243:84 , srv888742903:84 , srv895351569:84 , srv896940522:84 , srv897779767:84 , srv947799610:84 , srv982912553:84 , srv98805908:84 , srv682604331:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-03-07 04:54:21,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 04:54:21,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1230989033=10, srv1471517137=19, srv790968422=88, srv1288652821=13, srv1367633437=16, srv1527249116=23, srv1137104830=6, srv88030243=92, srv2121496804=57, srv559613944=78, srv1986679677=50, srv1823413518=41, srv696472892=86, srv2132312333=58, srv2145658593=61, srv693769448=84, srv1905924867=46, srv1602091423=31, srv213514393=59, srv435922076=73, srv1574603706=30, srv2040254101=53, srv821140557=90, srv1809456180=40, srv1233656980=11, srv1555612692=29, srv1346721142=15, srv730305388=87, srv2312308=64, srv1068987408=2, srv947799610=97, srv222350470=62, srv342359351=70, srv409249040=72, srv1162560054=7, srv1745107032=37, srv1051846249=1, srv287835896=68, srv119822529=8, srv1537401451=26, srv1672914050=34, srv223267640=63, srv1533958817=24, srv338220873=69, srv177306947=38, srv574601601=79, srv64885281=81, srv682604331=83, srv827120472=91, srv896940522=95, srv525341559=76, srv1475572274=20, srv1123495896=3, srv6937753=85, srv1930660733=49, srv167226055=33, srv2098416321=55, srv1135598707=5, srv1689979024=35, srv1920061116=47, srv2036426874=52, srv1925552451=48, srv1477999159=21, srv1544858105=28, srv1124559248=4, srv1036970899=0, srv594431942=80, srv530668357=77, srv98805908=99, srv1230228323=9, srv1326205158=14, srv2135913787=60, srv1505450355=22, srv1847088714=43, srv13734721=17, srv1691757429=36, srv1903350203=45, srv1534632846=25, srv895351569=94, srv211275071=56, srv1285069689=12, srv265557199=66, srv982912553=98, srv1858170078=44, srv680846639=82, srv1802656065=39, srv1374847270=18, srv803445452=89, srv256087933=65, srv1842854006=42, srv888742903=93, srv1539018993=27, srv44388921=74, srv496744104=75, srv897779767=96, srv389496868=71, srv165447195=32, srv202605596=51, srv26885299=67, srv2072064604=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 04:54:21,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 04:54:21,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 04:54:21,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 04:54:21,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 04:54:21,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 04:54:21,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 04:54:21,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 04:54:21,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 04:54:21,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 04:54:21,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 04:54:21,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 04:54:21,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 04:54:21,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 04:54:21,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 04:54:21,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 04:54:21,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 04:54:21,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 04:54:21,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 04:54:21,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 04:54:21,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 04:54:21,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 04:54:21,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 04:54:21,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 04:54:21,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 04:54:21,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 04:54:21,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 04:54:21,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 04:54:21,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 04:54:21,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 04:54:21,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 04:54:21,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-07 04:54:21,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-07 04:54:21,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-07 04:54:21,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-07 04:54:21,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-07 04:54:21,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-07 04:54:21,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-07 04:54:21,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-07 04:54:21,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-07 04:54:21,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-07 04:54:21,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-07 04:54:21,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-07 04:54:21,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-07 04:54:21,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-07 04:54:21,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-07 04:54:21,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-07 04:54:21,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-07 04:54:21,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-07 04:54:21,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-07 04:54:21,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-07 04:54:21,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-07 04:54:21,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-07 04:54:21,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-07 04:54:21,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-07 04:54:21,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-07 04:54:21,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-07 04:54:21,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-07 04:54:21,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-07 04:54:21,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-07 04:54:21,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-07 04:54:21,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-07 04:54:21,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-07 04:54:21,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-07 04:54:21,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-07 04:54:21,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-07 04:54:21,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-07 04:54:21,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-07 04:54:21,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-07 04:54:21,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-07 04:54:21,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-07 04:54:21,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-07 04:54:21,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-07 04:54:21,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-07 04:54:21,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-07 04:54:21,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-07 04:54:21,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-07 04:54:21,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-07 04:54:21,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-07 04:54:21,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-07 04:54:21,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-07 04:54:21,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-07 04:54:21,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-07 04:54:21,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-07 04:54:21,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-07 04:54:21,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-07 04:54:21,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-07 04:54:21,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-07 04:54:21,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-07 04:54:21,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-07 04:54:21,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-07 04:54:21,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-07 04:54:21,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-07 04:54:21,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-07 04:54:21,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-07 04:54:21,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-07 04:54:21,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-07 04:54:21,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-07 04:54:21,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-07 04:54:21,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-07 04:54:21,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-07 04:54:21,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-07 04:54:21,639 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 04:54:21,639 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-07 04:54:21,639 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-07 04:54:21,896 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 390 ms to try 5000 different iterations.  Found a solution that moves 4656 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18884982559805782. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04007568597117966, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.058620505255103664, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5173333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04007568597117966, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18433333333333332, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24891666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 04:54:21,910 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1477999159:81 , srv1555612692:81 , srv2098416321:81 , srv2132312333:81 , srv790968422:81 , srv1124559248:82 , srv435922076:82 , srv947799610:82 , srv119822529:83 , srv1505450355:83 , srv202605596:83 , srv211275071:83 , srv2121496804:83 , srv338220873:83 , srv693769448:83 , srv1123495896:84 , srv1135598707:84 , srv165447195:84 , srv167226055:84 , srv1809456180:84 , srv1905924867:84 , srv1925552451:84 , srv1930660733:84 , srv1346721142:85 , srv1533958817:85 , srv1689979024:85 , srv1847088714:85 , srv222350470:85 , srv44388921:85 , srv525341559:85 , srv895351569:85 , srv896940522:85 , srv98805908:85 , srv1051846249:86 , srv1230228323:86 , srv1288652821:86 , srv1534632846:86 , srv1544858105:86 , srv1691757429:86 , srv1823413518:86 , srv1842854006:86 , srv1986679677:86 , srv265557199:86 , srv287835896:86 , srv342359351:86 , srv496744104:86 , srv559613944:86 , srv64885281:86 , srv680846639:86 , srv803445452:86 , srv88030243:86 , srv897779767:86 , srv1068987408:87 , srv1230989033:87 , srv1367633437:87 , srv13734721:87 , srv1471517137:87 , srv1574603706:87 , srv1672914050:87 , srv1858170078:87 , srv2036426874:87 , srv2040254101:87 , srv2135913787:87 , srv2312308:87 , srv26885299:87 , srv389496868:87 , srv409249040:87 , srv530668357:87 , srv594431942:87 , srv730305388:87 , srv982912553:87 , srv1036970899:88 , srv1233656980:88 , srv1374847270:88 , srv1475572274:88 , srv1527249116:88 , srv1537401451:88 , srv1539018993:88 , srv1602091423:88 , srv256087933:88 , srv6937753:88 , srv821140557:88 , srv1162560054:89 , srv1745107032:89 , srv1903350203:89 , srv1920061116:89 , srv2072064604:89 , srv696472892:89 , srv1137104830:90 , srv827120472:90 , srv1285069689:92 , srv1326205158:92 , srv177306947:92 , srv1802656065:92 , srv213514393:92 , srv223267640:92 , srv2145658593:93 , srv574601601:93 , srv888742903:93 , srv682604331:446 }
2024-03-07 04:54:21,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 04:54:21,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1230989033=10, srv1471517137=19, srv790968422=88, srv1288652821=13, srv1367633437=16, srv1527249116=23, srv1137104830=6, srv88030243=92, srv2121496804=57, srv559613944=78, srv1986679677=50, srv1823413518=41, srv696472892=86, srv2132312333=58, srv2145658593=61, srv693769448=84, srv1905924867=46, srv1602091423=31, srv213514393=59, srv435922076=73, srv1574603706=30, srv2040254101=53, srv821140557=90, srv1809456180=40, srv1233656980=11, srv1555612692=29, srv1346721142=15, srv730305388=87, srv2312308=64, srv1068987408=2, srv947799610=97, srv222350470=62, srv342359351=70, srv409249040=72, srv1162560054=7, srv1745107032=37, srv1051846249=1, srv287835896=68, srv119822529=8, srv1537401451=26, srv1672914050=34, srv223267640=63, srv1533958817=24, srv338220873=69, srv177306947=38, srv574601601=79, srv64885281=81, srv682604331=83, srv827120472=91, srv896940522=95, srv525341559=76, srv1475572274=20, srv1123495896=3, srv6937753=85, srv1930660733=49, srv167226055=33, srv2098416321=55, srv1135598707=5, srv1689979024=35, srv1920061116=47, srv2036426874=52, srv1925552451=48, srv1477999159=21, srv1544858105=28, srv1124559248=4, srv1036970899=0, srv594431942=80, srv530668357=77, srv98805908=99, srv1230228323=9, srv1326205158=14, srv2135913787=60, srv1505450355=22, srv1847088714=43, srv13734721=17, srv1691757429=36, srv1903350203=45, srv1534632846=25, srv895351569=94, srv211275071=56, srv1285069689=12, srv265557199=66, srv982912553=98, srv1858170078=44, srv680846639=82, srv1802656065=39, srv1374847270=18, srv803445452=89, srv256087933=65, srv1842854006=42, srv888742903=93, srv1539018993=27, srv44388921=74, srv496744104=75, srv897779767=96, srv389496868=71, srv165447195=32, srv202605596=51, srv26885299=67, srv2072064604=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 04:54:21,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 04:54:21,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 04:54:21,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 04:54:21,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 04:54:21,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 04:54:21,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 04:54:21,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 04:54:21,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 04:54:21,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 04:54:21,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 04:54:21,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 04:54:21,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 04:54:21,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 04:54:21,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 04:54:21,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 04:54:21,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 04:54:21,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 04:54:21,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 04:54:21,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 04:54:21,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 04:54:21,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 04:54:21,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 04:54:21,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 04:54:21,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 04:54:21,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 04:54:21,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 04:54:21,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 04:54:21,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 04:54:21,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 04:54:21,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 04:54:21,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-07 04:54:21,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-07 04:54:21,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-07 04:54:21,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-07 04:54:21,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-07 04:54:21,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-07 04:54:21,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-07 04:54:21,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-07 04:54:21,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-07 04:54:21,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-07 04:54:21,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-07 04:54:21,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-07 04:54:21,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-07 04:54:21,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-07 04:54:21,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-07 04:54:21,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-07 04:54:21,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-07 04:54:21,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-07 04:54:21,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-07 04:54:21,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-07 04:54:21,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-07 04:54:21,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-07 04:54:21,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-07 04:54:21,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-07 04:54:21,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-07 04:54:21,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-07 04:54:21,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-07 04:54:21,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-07 04:54:21,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-07 04:54:21,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-07 04:54:21,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-07 04:54:21,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-07 04:54:21,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-07 04:54:21,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-07 04:54:21,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-07 04:54:21,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-07 04:54:21,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-07 04:54:21,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-07 04:54:21,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-07 04:54:21,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-07 04:54:21,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-07 04:54:21,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-07 04:54:21,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-07 04:54:21,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-07 04:54:21,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-07 04:54:21,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-07 04:54:21,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-07 04:54:21,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-07 04:54:21,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-07 04:54:21,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-07 04:54:21,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-07 04:54:21,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-07 04:54:21,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-07 04:54:21,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-07 04:54:21,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-07 04:54:21,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-07 04:54:21,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-07 04:54:21,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-07 04:54:21,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-07 04:54:21,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-07 04:54:21,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-07 04:54:21,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-07 04:54:21,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-07 04:54:21,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-07 04:54:21,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-07 04:54:21,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-07 04:54:21,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-07 04:54:21,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-07 04:54:21,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-07 04:54:21,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-07 04:54:21,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-07 04:54:22,043 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04007568597117966, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.058620505255103664, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04007568597117966, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18433333333333332, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24891666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 04:54:22,044 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-07 04:54:22,044 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18881722358924108, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04007568597117966, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.058620505255103664, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04007568597117966, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18433333333333332, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24891666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-07 04:54:22,215 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 257 ms to try 5000 different iterations.  Found a solution that moves 2640 regions; Going from a computed imbalance of 0.18881722358924108 to a new imbalance of 0.01595443319282247. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02237824019262836); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036289767845504105, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29333333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02237824019262836); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 04:54:22,223 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1285069689:78 , srv1802656065:78 , srv888742903:78 , srv177306947:79 , srv223267640:79 , srv213514393:80 , srv2145658593:80 , srv1903350203:81 , srv1920061116:81 , srv574601601:81 , srv827120472:81 , srv2072064604:82 , srv409249040:82 , srv1036970899:83 , srv1137104830:83 , srv1326205158:83 , srv1374847270:83 , srv1539018993:83 , srv1602091423:83 , srv342359351:83 , srv1162560054:84 , srv1230228323:84 , srv1230989033:84 , srv1367633437:84 , srv1858170078:84 , srv256087933:84 , srv530668357:84 , srv1068987408:85 , srv13734721:85 , srv1471517137:85 , srv1527249116:85 , srv1544858105:85 , srv1672914050:85 , srv1691757429:85 , srv1745107032:85 , srv2036426874:85 , srv2040254101:85 , srv2312308:85 , srv26885299:85 , srv389496868:85 , srv496744104:85 , srv594431942:85 , srv64885281:85 , srv696472892:85 , srv88030243:85 , srv98805908:85 , srv1233656980:86 , srv1288652821:86 , srv1475572274:86 , srv1534632846:86 , srv1537401451:86 , srv1574603706:86 , srv1689979024:86 , srv1842854006:86 , srv6937753:86 , srv730305388:86 , srv803445452:86 , srv821140557:86 , srv1051846249:87 , srv1346721142:87 , srv1533958817:87 , srv1823413518:87 , srv1847088714:87 , srv1925552451:87 , srv1930660733:87 , srv2135913787:87 , srv265557199:87 , srv44388921:87 , srv525341559:87 , srv559613944:87 , srv680846639:87 , srv982912553:87 , srv1135598707:88 , srv165447195:88 , srv167226055:88 , srv1809456180:88 , srv1905924867:88 , srv1986679677:88 , srv202605596:88 , srv2121496804:88 , srv222350470:88 , srv895351569:88 , srv896940522:88 , srv897779767:88 , srv947799610:88 , srv1123495896:89 , srv1124559248:89 , srv119822529:89 , srv1505450355:89 , srv287835896:89 , srv338220873:89 , srv693769448:89 , srv1477999159:90 , srv2098416321:90 , srv211275071:90 , srv2132312333:90 , srv435922076:90 , srv790968422:90 , srv1555612692:92 , srv682604331:527 }
2024-03-07 04:54:22,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 04:54:22,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1230989033=10, srv1471517137=19, srv790968422=88, srv1288652821=13, srv1367633437=16, srv1527249116=23, srv1137104830=6, srv88030243=92, srv2121496804=57, srv559613944=78, srv1986679677=50, srv1823413518=41, srv696472892=86, srv2132312333=58, srv2145658593=61, srv693769448=84, srv1905924867=46, srv1602091423=31, srv213514393=59, srv435922076=73, srv1574603706=30, srv2040254101=53, srv821140557=90, srv1809456180=40, srv1233656980=11, srv1555612692=29, srv1346721142=15, srv730305388=87, srv2312308=64, srv1068987408=2, srv947799610=97, srv222350470=62, srv342359351=70, srv409249040=72, srv1162560054=7, srv1745107032=37, srv1051846249=1, srv287835896=68, srv119822529=8, srv1537401451=26, srv1672914050=34, srv223267640=63, srv1533958817=24, srv338220873=69, srv177306947=38, srv574601601=79, srv64885281=81, srv682604331=83, srv827120472=91, srv896940522=95, srv525341559=76, srv1475572274=20, srv1123495896=3, srv6937753=85, srv1930660733=49, srv167226055=33, srv2098416321=55, srv1135598707=5, srv1689979024=35, srv1920061116=47, srv2036426874=52, srv1925552451=48, srv1477999159=21, srv1544858105=28, srv1124559248=4, srv1036970899=0, srv594431942=80, srv530668357=77, srv98805908=99, srv1230228323=9, srv1326205158=14, srv2135913787=60, srv1505450355=22, srv1847088714=43, srv13734721=17, srv1691757429=36, srv1903350203=45, srv1534632846=25, srv895351569=94, srv211275071=56, srv1285069689=12, srv265557199=66, srv982912553=98, srv1858170078=44, srv680846639=82, srv1802656065=39, srv1374847270=18, srv803445452=89, srv256087933=65, srv1842854006=42, srv888742903=93, srv1539018993=27, srv44388921=74, srv496744104=75, srv897779767=96, srv389496868=71, srv165447195=32, srv202605596=51, srv26885299=67, srv2072064604=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 04:54:22,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 04:54:22,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 04:54:22,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 04:54:22,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 04:54:22,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 04:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 04:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 04:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 04:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 04:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 04:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 04:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 04:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 04:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 04:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 04:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 04:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 04:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 04:54:22,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 04:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 04:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 04:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 04:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 04:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 04:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 04:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 04:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 04:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 04:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 04:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 04:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-07 04:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-07 04:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-07 04:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-07 04:54:22,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-07 04:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-07 04:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-07 04:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-07 04:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-07 04:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-07 04:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-07 04:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-07 04:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-07 04:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-07 04:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-07 04:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-07 04:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-07 04:54:22,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-07 04:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-07 04:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-07 04:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-07 04:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-07 04:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-07 04:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-07 04:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-07 04:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-07 04:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-07 04:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-07 04:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-07 04:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-07 04:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-07 04:54:22,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-07 04:54:22,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-07 04:54:22,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-07 04:54:22,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-07 04:54:22,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-07 04:54:22,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-07 04:54:22,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-07 04:54:22,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-07 04:54:22,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-07 04:54:22,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-07 04:54:22,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-07 04:54:22,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-07 04:54:22,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-07 04:54:22,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-07 04:54:22,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-07 04:54:22,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-07 04:54:22,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-07 04:54:22,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-07 04:54:22,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-07 04:54:22,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-07 04:54:22,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-07 04:54:22,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-07 04:54:22,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-07 04:54:22,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-07 04:54:22,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-07 04:54:22,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-07 04:54:22,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-07 04:54:22,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-07 04:54:22,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-07 04:54:22,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-07 04:54:22,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-07 04:54:22,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-07 04:54:22,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-07 04:54:22,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-07 04:54:22,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-07 04:54:22,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-07 04:54:22,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-07 04:54:22,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-07 04:54:22,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-07 04:54:22,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-07 04:54:22,348 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02237824019262836); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036289767845504105, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02237824019262836); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 04:54:22,348 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-07 04:54:22,348 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015935947517720213, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02237824019262836); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036289767845504105, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02237824019262836); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-07 04:54:22,496 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 230 ms to try 5000 different iterations.  Found a solution that moves 985 regions; Going from a computed imbalance of 0.015935947517720213 to a new imbalance of 7.452998220130734E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00610013298189536); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010845305479975394); MoveCostFunction : (multiplier=7.0, imbalance=0.10944444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00610013298189536); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 04:54:22,498 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1326205158:82 , srv1537401451:83 , srv1986679677:83 , srv2135913787:83 , srv265557199:83 , srv287835896:83 , srv696472892:83 , srv1123495896:84 , srv1162560054:84 , srv1233656980:84 , srv1475572274:84 , srv1527249116:84 , srv1555612692:84 , srv1574603706:84 , srv1745107032:84 , srv1823413518:84 , srv211275071:84 , srv680846639:84 , srv6937753:84 , srv821140557:84 , srv896940522:84 , srv897779767:84 , srv982912553:84 , srv1051846249:85 , srv1135598707:85 , srv1137104830:85 , srv119822529:85 , srv1288652821:85 , srv1471517137:85 , srv1505450355:85 , srv1534632846:85 , srv165447195:85 , srv167226055:85 , srv1842854006:85 , srv1847088714:85 , srv2036426874:85 , srv2040254101:85 , srv213514393:85 , srv2145658593:85 , srv222350470:85 , srv2312308:85 , srv256087933:85 , srv338220873:85 , srv389496868:85 , srv44388921:85 , srv525341559:85 , srv559613944:85 , srv574601601:85 , srv730305388:85 , srv895351569:85 , srv1036970899:86 , srv1068987408:86 , srv1124559248:86 , srv1346721142:86 , srv1367633437:86 , srv1374847270:86 , srv1477999159:86 , srv1533958817:86 , srv1544858105:86 , srv1602091423:86 , srv1672914050:86 , srv1689979024:86 , srv177306947:86 , srv1905924867:86 , srv202605596:86 , srv2072064604:86 , srv2132312333:86 , srv223267640:86 , srv26885299:86 , srv435922076:86 , srv496744104:86 , srv594431942:86 , srv64885281:86 , srv803445452:86 , srv888742903:86 , srv1230228323:87 , srv1230989033:87 , srv1285069689:87 , srv13734721:87 , srv1539018993:87 , srv1691757429:87 , srv1802656065:87 , srv1809456180:87 , srv1858170078:87 , srv1920061116:87 , srv1925552451:87 , srv1930660733:87 , srv2098416321:87 , srv342359351:87 , srv530668357:87 , srv693769448:87 , srv790968422:87 , srv827120472:87 , srv88030243:87 , srv947799610:87 , srv98805908:87 , srv1903350203:88 , srv2121496804:88 , srv409249040:89 , srv682604331:539 }
2024-03-07 04:54:22,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 04:54:22,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1230989033=10, srv1471517137=19, srv790968422=88, srv1288652821=13, srv1367633437=16, srv1527249116=23, srv1137104830=6, srv88030243=92, srv2121496804=57, srv559613944=78, srv1986679677=50, srv1823413518=41, srv696472892=86, srv2132312333=58, srv2145658593=61, srv693769448=84, srv1905924867=46, srv1602091423=31, srv213514393=59, srv435922076=73, srv1574603706=30, srv2040254101=53, srv821140557=90, srv1809456180=40, srv1233656980=11, srv1555612692=29, srv1346721142=15, srv730305388=87, srv2312308=64, srv1068987408=2, srv947799610=97, srv222350470=62, srv342359351=70, srv409249040=72, srv1162560054=7, srv1745107032=37, srv1051846249=1, srv287835896=68, srv119822529=8, srv1537401451=26, srv1672914050=34, srv223267640=63, srv1533958817=24, srv338220873=69, srv177306947=38, srv574601601=79, srv64885281=81, srv682604331=83, srv827120472=91, srv896940522=95, srv525341559=76, srv1475572274=20, srv1123495896=3, srv6937753=85, srv1930660733=49, srv167226055=33, srv2098416321=55, srv1135598707=5, srv1689979024=35, srv1920061116=47, srv2036426874=52, srv1925552451=48, srv1477999159=21, srv1544858105=28, srv1124559248=4, srv1036970899=0, srv594431942=80, srv530668357=77, srv98805908=99, srv1230228323=9, srv1326205158=14, srv2135913787=60, srv1505450355=22, srv1847088714=43, srv13734721=17, srv1691757429=36, srv1903350203=45, srv1534632846=25, srv895351569=94, srv211275071=56, srv1285069689=12, srv265557199=66, srv982912553=98, srv1858170078=44, srv680846639=82, srv1802656065=39, srv1374847270=18, srv803445452=89, srv256087933=65, srv1842854006=42, srv888742903=93, srv1539018993=27, srv44388921=74, srv496744104=75, srv897779767=96, srv389496868=71, srv165447195=32, srv202605596=51, srv26885299=67, srv2072064604=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 04:54:22,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 04:54:22,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 04:54:22,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 04:54:22,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 04:54:22,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 04:54:22,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 04:54:22,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 04:54:22,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 04:54:22,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 04:54:22,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 04:54:22,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 04:54:22,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 04:54:22,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 04:54:22,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 04:54:22,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 04:54:22,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 04:54:22,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 04:54:22,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 04:54:22,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 04:54:22,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 04:54:22,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 04:54:22,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 04:54:22,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 04:54:22,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 04:54:22,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 04:54:22,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 04:54:22,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 04:54:22,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 04:54:22,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 04:54:22,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 04:54:22,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-07 04:54:22,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-07 04:54:22,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-07 04:54:22,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-07 04:54:22,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-07 04:54:22,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-07 04:54:22,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-07 04:54:22,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-07 04:54:22,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-07 04:54:22,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-07 04:54:22,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-07 04:54:22,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-07 04:54:22,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-07 04:54:22,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-07 04:54:22,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-07 04:54:22,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-07 04:54:22,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-07 04:54:22,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-07 04:54:22,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-07 04:54:22,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-07 04:54:22,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-07 04:54:22,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-07 04:54:22,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-07 04:54:22,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-07 04:54:22,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-07 04:54:22,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-07 04:54:22,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-07 04:54:22,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-07 04:54:22,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-07 04:54:22,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-07 04:54:22,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-07 04:54:22,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-07 04:54:22,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-07 04:54:22,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-07 04:54:22,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-07 04:54:22,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-07 04:54:22,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-07 04:54:22,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-07 04:54:22,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-07 04:54:22,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-07 04:54:22,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-07 04:54:22,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-07 04:54:22,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-07 04:54:22,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-07 04:54:22,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-07 04:54:22,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-07 04:54:22,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-07 04:54:22,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-07 04:54:22,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-07 04:54:22,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-07 04:54:22,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-07 04:54:22,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-07 04:54:22,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-07 04:54:22,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-07 04:54:22,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-07 04:54:22,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-07 04:54:22,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-07 04:54:22,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-07 04:54:22,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-07 04:54:22,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-07 04:54:22,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-07 04:54:22,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-07 04:54:22,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-07 04:54:22,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-07 04:54:22,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-07 04:54:22,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-07 04:54:22,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-07 04:54:22,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-07 04:54:22,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-07 04:54:22,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-07 04:54:22,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-07 04:54:22,603 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00610013298189536); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010845305479975394); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00610013298189536); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 04:54:22,603 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-07 04:54:22,604 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.384027045980269E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00610013298189536); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010845305479975394); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00610013298189536); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-07 04:54:22,825 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 293 ms to try 5000 different iterations.  Found a solution that moves 259 regions; Going from a computed imbalance of 7.384027045980269E-4 to a new imbalance of 3.122081442860018E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002721655269759087); MoveCostFunction : (multiplier=7.0, imbalance=0.028777777777777777, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 04:54:22,826 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1233656980:83 , srv13734721:83 , srv2098416321:83 , srv2121496804:83 , srv574601601:83 , srv693769448:83 , srv730305388:83 , srv1051846249:84 , srv1068987408:84 , srv1137104830:84 , srv1230989033:84 , srv1346721142:84 , srv1475572274:84 , srv1533958817:84 , srv1539018993:84 , srv1602091423:84 , srv167226055:84 , srv1672914050:84 , srv1691757429:84 , srv1745107032:84 , srv1802656065:84 , srv1809456180:84 , srv1847088714:84 , srv1858170078:84 , srv1903350203:84 , srv1905924867:84 , srv1925552451:84 , srv1930660733:84 , srv2145658593:84 , srv26885299:84 , srv287835896:84 , srv409249040:84 , srv530668357:84 , srv594431942:84 , srv6937753:84 , srv790968422:84 , srv803445452:84 , srv821140557:84 , srv827120472:84 , srv895351569:84 , srv896940522:84 , srv897779767:84 , srv982912553:84 , srv1036970899:85 , srv1123495896:85 , srv1124559248:85 , srv1135598707:85 , srv1162560054:85 , srv1230228323:85 , srv1285069689:85 , srv1288652821:85 , srv1326205158:85 , srv1367633437:85 , srv1374847270:85 , srv1471517137:85 , srv1477999159:85 , srv1505450355:85 , srv1527249116:85 , srv1534632846:85 , srv1537401451:85 , srv1544858105:85 , srv1555612692:85 , srv1574603706:85 , srv165447195:85 , srv1689979024:85 , srv177306947:85 , srv1823413518:85 , srv1842854006:85 , srv1920061116:85 , srv1986679677:85 , srv202605596:85 , srv2036426874:85 , srv2040254101:85 , srv211275071:85 , srv2132312333:85 , srv213514393:85 , srv2135913787:85 , srv222350470:85 , srv223267640:85 , srv2312308:85 , srv256087933:85 , srv389496868:85 , srv435922076:85 , srv44388921:85 , srv496744104:85 , srv525341559:85 , srv559613944:85 , srv64885281:85 , srv680846639:85 , srv696472892:85 , srv88030243:85 , srv888742903:85 , srv947799610:85 , srv98805908:85 , srv119822529:86 , srv2072064604:86 , srv265557199:86 , srv338220873:86 , srv342359351:86 , srv682604331:630 }
2024-03-07 04:54:22,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 04:54:22,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1230989033=10, srv1471517137=19, srv790968422=88, srv1288652821=13, srv1367633437=16, srv1527249116=23, srv1137104830=6, srv88030243=92, srv2121496804=57, srv559613944=78, srv1986679677=50, srv1823413518=41, srv696472892=86, srv2132312333=58, srv2145658593=61, srv693769448=84, srv1905924867=46, srv1602091423=31, srv213514393=59, srv435922076=73, srv1574603706=30, srv2040254101=53, srv821140557=90, srv1809456180=40, srv1233656980=11, srv1555612692=29, srv1346721142=15, srv730305388=87, srv2312308=64, srv1068987408=2, srv947799610=97, srv222350470=62, srv342359351=70, srv409249040=72, srv1162560054=7, srv1745107032=37, srv1051846249=1, srv287835896=68, srv119822529=8, srv1537401451=26, srv1672914050=34, srv223267640=63, srv1533958817=24, srv338220873=69, srv177306947=38, srv574601601=79, srv64885281=81, srv682604331=83, srv827120472=91, srv896940522=95, srv525341559=76, srv1475572274=20, srv1123495896=3, srv6937753=85, srv1930660733=49, srv167226055=33, srv2098416321=55, srv1135598707=5, srv1689979024=35, srv1920061116=47, srv2036426874=52, srv1925552451=48, srv1477999159=21, srv1544858105=28, srv1124559248=4, srv1036970899=0, srv594431942=80, srv530668357=77, srv98805908=99, srv1230228323=9, srv1326205158=14, srv2135913787=60, srv1505450355=22, srv1847088714=43, srv13734721=17, srv1691757429=36, srv1903350203=45, srv1534632846=25, srv895351569=94, srv211275071=56, srv1285069689=12, srv265557199=66, srv982912553=98, srv1858170078=44, srv680846639=82, srv1802656065=39, srv1374847270=18, srv803445452=89, srv256087933=65, srv1842854006=42, srv888742903=93, srv1539018993=27, srv44388921=74, srv496744104=75, srv897779767=96, srv389496868=71, srv165447195=32, srv202605596=51, srv26885299=67, srv2072064604=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 04:54:22,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 04:54:22,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 04:54:22,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 04:54:22,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 04:54:22,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 04:54:22,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 04:54:22,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 04:54:22,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 04:54:22,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 04:54:22,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 04:54:22,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 04:54:22,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 04:54:22,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 04:54:22,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 04:54:22,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 04:54:22,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 04:54:22,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 04:54:22,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 04:54:22,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 04:54:22,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 04:54:22,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 04:54:22,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 04:54:22,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 04:54:22,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 04:54:22,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 04:54:22,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 04:54:22,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 04:54:22,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 04:54:22,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 04:54:22,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 04:54:22,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-07 04:54:22,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-07 04:54:22,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-07 04:54:22,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-07 04:54:22,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-07 04:54:22,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-07 04:54:22,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-07 04:54:22,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-07 04:54:22,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-07 04:54:22,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-07 04:54:22,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-07 04:54:22,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-07 04:54:22,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-07 04:54:22,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-07 04:54:22,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-07 04:54:22,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-07 04:54:22,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-07 04:54:22,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-07 04:54:22,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-07 04:54:22,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-07 04:54:22,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-07 04:54:22,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-07 04:54:22,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-07 04:54:22,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-07 04:54:22,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-07 04:54:22,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-07 04:54:22,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-07 04:54:22,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-07 04:54:22,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-07 04:54:22,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-07 04:54:22,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-07 04:54:22,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-07 04:54:22,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-07 04:54:22,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-07 04:54:22,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-07 04:54:22,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-07 04:54:22,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-07 04:54:22,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-07 04:54:22,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-07 04:54:22,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-07 04:54:22,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-07 04:54:22,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-07 04:54:22,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-07 04:54:22,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-07 04:54:22,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-07 04:54:22,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-07 04:54:22,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-07 04:54:22,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-07 04:54:22,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-07 04:54:22,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-07 04:54:22,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-07 04:54:22,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-07 04:54:22,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-07 04:54:22,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-07 04:54:22,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-07 04:54:22,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-07 04:54:22,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-07 04:54:22,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-07 04:54:22,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-07 04:54:22,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-07 04:54:22,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-07 04:54:22,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-07 04:54:22,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-07 04:54:22,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-07 04:54:22,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-07 04:54:22,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-07 04:54:22,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-07 04:54:22,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-07 04:54:22,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-07 04:54:22,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-07 04:54:22,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-07 04:54:22,921 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002721655269759087); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 04:54:22,921 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-07 04:54:22,922 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.940725766667427E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002721655269759087); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-07 04:54:23,075 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 225 ms to try 5000 different iterations.  Found a solution that moves 51 regions; Going from a computed imbalance of 2.940725766667427E-5 to a new imbalance of 7.101178686489173E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.005666666666666667); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 04:54:23,075 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv119822529:83 , srv1809456180:83 , srv2072064604:83 , srv222350470:83 , srv338220873:83 , srv435922076:83 , srv559613944:83 , srv88030243:83 , srv1036970899:84 , srv1051846249:84 , srv1068987408:84 , srv1123495896:84 , srv1124559248:84 , srv1135598707:84 , srv1137104830:84 , srv1162560054:84 , srv1230228323:84 , srv1230989033:84 , srv1285069689:84 , srv1288652821:84 , srv1326205158:84 , srv1346721142:84 , srv1367633437:84 , srv13734721:84 , srv1374847270:84 , srv1471517137:84 , srv1475572274:84 , srv1477999159:84 , srv1505450355:84 , srv1527249116:84 , srv1533958817:84 , srv1534632846:84 , srv1537401451:84 , srv1539018993:84 , srv1544858105:84 , srv1555612692:84 , srv1574603706:84 , srv165447195:84 , srv1672914050:84 , srv1689979024:84 , srv1691757429:84 , srv1745107032:84 , srv177306947:84 , srv1823413518:84 , srv1842854006:84 , srv1858170078:84 , srv1903350203:84 , srv1905924867:84 , srv1920061116:84 , srv1925552451:84 , srv1930660733:84 , srv1986679677:84 , srv202605596:84 , srv2036426874:84 , srv2040254101:84 , srv211275071:84 , srv2121496804:84 , srv2132312333:84 , srv213514393:84 , srv2135913787:84 , srv2145658593:84 , srv223267640:84 , srv2312308:84 , srv256087933:84 , srv265557199:84 , srv26885299:84 , srv287835896:84 , srv342359351:84 , srv389496868:84 , srv409249040:84 , srv44388921:84 , srv496744104:84 , srv525341559:84 , srv530668357:84 , srv574601601:84 , srv594431942:84 , srv64885281:84 , srv680846639:84 , srv693769448:84 , srv6937753:84 , srv696472892:84 , srv790968422:84 , srv803445452:84 , srv821140557:84 , srv827120472:84 , srv888742903:84 , srv895351569:84 , srv897779767:84 , srv947799610:84 , srv982912553:84 , srv98805908:84 , srv1233656980:85 , srv1602091423:85 , srv167226055:85 , srv1802656065:85 , srv1847088714:85 , srv2098416321:85 , srv730305388:85 , srv896940522:85 , srv682604331:684 }
2024-03-07 04:54:23,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 04:54:23,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1230989033=10, srv1471517137=19, srv790968422=88, srv1288652821=13, srv1367633437=16, srv1527249116=23, srv1137104830=6, srv88030243=92, srv2121496804=57, srv559613944=78, srv1986679677=50, srv1823413518=41, srv696472892=86, srv2132312333=58, srv2145658593=61, srv693769448=84, srv1905924867=46, srv1602091423=31, srv213514393=59, srv435922076=73, srv1574603706=30, srv2040254101=53, srv821140557=90, srv1809456180=40, srv1233656980=11, srv1555612692=29, srv1346721142=15, srv730305388=87, srv2312308=64, srv1068987408=2, srv947799610=97, srv222350470=62, srv342359351=70, srv409249040=72, srv1162560054=7, srv1745107032=37, srv1051846249=1, srv287835896=68, srv119822529=8, srv1537401451=26, srv1672914050=34, srv223267640=63, srv1533958817=24, srv338220873=69, srv177306947=38, srv574601601=79, srv64885281=81, srv682604331=83, srv827120472=91, srv896940522=95, srv525341559=76, srv1475572274=20, srv1123495896=3, srv6937753=85, srv1930660733=49, srv167226055=33, srv2098416321=55, srv1135598707=5, srv1689979024=35, srv1920061116=47, srv2036426874=52, srv1925552451=48, srv1477999159=21, srv1544858105=28, srv1124559248=4, srv1036970899=0, srv594431942=80, srv530668357=77, srv98805908=99, srv1230228323=9, srv1326205158=14, srv2135913787=60, srv1505450355=22, srv1847088714=43, srv13734721=17, srv1691757429=36, srv1903350203=45, srv1534632846=25, srv895351569=94, srv211275071=56, srv1285069689=12, srv265557199=66, srv982912553=98, srv1858170078=44, srv680846639=82, srv1802656065=39, srv1374847270=18, srv803445452=89, srv256087933=65, srv1842854006=42, srv888742903=93, srv1539018993=27, srv44388921=74, srv496744104=75, srv897779767=96, srv389496868=71, srv165447195=32, srv202605596=51, srv26885299=67, srv2072064604=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 04:54:23,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 04:54:23,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 04:54:23,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 04:54:23,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 04:54:23,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 04:54:23,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 04:54:23,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 04:54:23,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 04:54:23,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 04:54:23,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 04:54:23,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 04:54:23,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 04:54:23,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 04:54:23,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 04:54:23,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 04:54:23,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 04:54:23,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 04:54:23,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 04:54:23,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 04:54:23,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 04:54:23,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 04:54:23,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 04:54:23,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 04:54:23,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 04:54:23,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 04:54:23,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 04:54:23,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 04:54:23,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 04:54:23,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 04:54:23,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 04:54:23,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-07 04:54:23,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-07 04:54:23,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-07 04:54:23,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-07 04:54:23,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-07 04:54:23,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-07 04:54:23,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-07 04:54:23,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-07 04:54:23,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-07 04:54:23,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-07 04:54:23,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-07 04:54:23,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-07 04:54:23,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-07 04:54:23,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-07 04:54:23,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-07 04:54:23,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-07 04:54:23,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-07 04:54:23,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-07 04:54:23,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-07 04:54:23,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-07 04:54:23,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-07 04:54:23,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-07 04:54:23,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-07 04:54:23,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-07 04:54:23,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-07 04:54:23,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-07 04:54:23,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-07 04:54:23,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-07 04:54:23,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-07 04:54:23,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-07 04:54:23,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-07 04:54:23,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-07 04:54:23,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-07 04:54:23,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-07 04:54:23,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-07 04:54:23,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-07 04:54:23,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-07 04:54:23,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-07 04:54:23,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-07 04:54:23,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-07 04:54:23,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-07 04:54:23,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-07 04:54:23,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-07 04:54:23,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-07 04:54:23,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-07 04:54:23,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-07 04:54:23,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-07 04:54:23,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-07 04:54:23,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-07 04:54:23,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-07 04:54:23,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-07 04:54:23,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-07 04:54:23,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-07 04:54:23,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-07 04:54:23,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-07 04:54:23,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-07 04:54:23,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-07 04:54:23,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-07 04:54:23,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-07 04:54:23,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-07 04:54:23,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-07 04:54:23,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-07 04:54:23,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-07 04:54:23,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-07 04:54:23,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-07 04:54:23,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-07 04:54:23,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-07 04:54:23,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-07 04:54:23,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-07 04:54:23,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-07 04:54:23,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-07 04:54:23,176 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 04:54:23,176 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv119822529:83 , srv1809456180:83 , srv2072064604:83 , srv222350470:83 , srv338220873:83 , srv435922076:83 , srv559613944:83 , srv88030243:83 , srv1036970899:84 , srv1051846249:84 , srv1068987408:84 , srv1123495896:84 , srv1124559248:84 , srv1135598707:84 , srv1137104830:84 , srv1162560054:84 , srv1230228323:84 , srv1230989033:84 , srv1285069689:84 , srv1288652821:84 , srv1326205158:84 , srv1346721142:84 , srv1367633437:84 , srv13734721:84 , srv1374847270:84 , srv1471517137:84 , srv1475572274:84 , srv1477999159:84 , srv1505450355:84 , srv1527249116:84 , srv1533958817:84 , srv1534632846:84 , srv1537401451:84 , srv1539018993:84 , srv1544858105:84 , srv1555612692:84 , srv1574603706:84 , srv165447195:84 , srv1672914050:84 , srv1689979024:84 , srv1691757429:84 , srv1745107032:84 , srv177306947:84 , srv1823413518:84 , srv1842854006:84 , srv1858170078:84 , srv1903350203:84 , srv1905924867:84 , srv1920061116:84 , srv1925552451:84 , srv1930660733:84 , srv1986679677:84 , srv202605596:84 , srv2036426874:84 , srv2040254101:84 , srv211275071:84 , srv2121496804:84 , srv2132312333:84 , srv213514393:84 , srv2135913787:84 , srv2145658593:84 , srv223267640:84 , srv2312308:84 , srv256087933:84 , srv265557199:84 , srv26885299:84 , srv287835896:84 , srv342359351:84 , srv389496868:84 , srv409249040:84 , srv44388921:84 , srv496744104:84 , srv525341559:84 , srv530668357:84 , srv574601601:84 , srv594431942:84 , srv64885281:84 , srv680846639:84 , srv693769448:84 , srv6937753:84 , srv696472892:84 , srv790968422:84 , srv803445452:84 , srv821140557:84 , srv827120472:84 , srv888742903:84 , srv895351569:84 , srv897779767:84 , srv947799610:84 , srv982912553:84 , srv98805908:84 , srv1233656980:85 , srv1602091423:85 , srv167226055:85 , srv1802656065:85 , srv1847088714:85 , srv2098416321:85 , srv730305388:85 , srv896940522:85 , srv682604331:684 }
2024-03-07 04:54:23,198 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=189 (was 189), ProcessCount=170 (was 170), AvailableMemoryMB=6021 (was 6510)
2024-03-07 04:54:23,205 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=190, ProcessCount=170, AvailableMemoryMB=6020
2024-03-07 04:54:23,206 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-07 04:54:23,206 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-03-07 04:54:23,213 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1005830201:84 , srv1057560017:84 , srv1112500671:84 , srv1467439518:84 , srv1569572969:84 , srv1609973057:84 , srv1675388148:84 , srv1724082023:84 , srv1744097141:84 , srv1829522002:84 , srv1855074598:84 , srv1857487569:84 , srv1888546492:84 , srv1892281441:84 , srv1896583091:84 , srv2014217786:84 , srv22393552:84 , srv243357887:84 , srv286221052:84 , srv407382994:84 , srv447371676:84 , srv493398986:84 , srv538845767:84 , srv559705309:84 , srv575951082:84 , srv611698918:84 , srv729764493:84 , srv770031278:84 , srv786255269:84 , srv1538270042:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-03-07 04:54:23,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 04:54:23,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1005830201=0, srv538845767=23, srv1538270042=4, srv575951082=25, srv1857487569=12, srv286221052=19, srv611698918=26, srv786255269=29, srv1855074598=11, srv1892281441=14, srv1724082023=8, srv243357887=18, srv559705309=24, srv407382994=20, srv1057560017=1, srv1569572969=5, srv1896583091=15, srv22393552=17, srv1829522002=10, srv1467439518=3, srv770031278=28, srv1112500671=2, srv1744097141=9, srv447371676=21, srv493398986=22, srv1888546492=13, srv1609973057=6, srv729764493=27, srv1675388148=7, srv2014217786=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 04:54:23,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 04:54:23,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 04:54:23,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 04:54:23,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 04:54:23,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 04:54:23,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 04:54:23,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 04:54:23,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 04:54:23,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 04:54:23,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 04:54:23,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 04:54:23,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 04:54:23,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 04:54:23,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 04:54:23,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 04:54:23,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 04:54:23,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 04:54:23,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 04:54:23,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 04:54:23,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 04:54:23,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 04:54:23,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 04:54:23,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 04:54:23,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 04:54:23,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 04:54:23,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 04:54:23,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 04:54:23,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 04:54:23,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 04:54:23,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 04:54:23,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-07 04:54:23,235 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1088516973003766, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 04:54:23,235 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912536031408138, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1088516973003766, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-03-07 04:56:23,225 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9773146 different iterations.  Found a solution that moves 1843 regions; Going from a computed imbalance of 0.9912536031408138 to a new imbalance of 4.3016539410932495E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6825925925925926, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 04:56:23,232 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1005830201:90 , srv1057560017:90 , srv1112500671:90 , srv1467439518:90 , srv1538270042:90 , srv1569572969:90 , srv1609973057:90 , srv1675388148:90 , srv1724082023:90 , srv1744097141:90 , srv1829522002:90 , srv1855074598:90 , srv1857487569:90 , srv1888546492:90 , srv1892281441:90 , srv1896583091:90 , srv2014217786:90 , srv22393552:90 , srv243357887:90 , srv286221052:90 , srv407382994:90 , srv447371676:90 , srv493398986:90 , srv538845767:90 , srv559705309:90 , srv575951082:90 , srv611698918:90 , srv729764493:90 , srv770031278:90 , srv786255269:90 }
2024-03-07 04:56:23,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 04:56:23,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1005830201=0, srv538845767=23, srv1538270042=4, srv575951082=25, srv1857487569=12, srv286221052=19, srv611698918=26, srv786255269=29, srv1855074598=11, srv1892281441=14, srv1724082023=8, srv243357887=18, srv559705309=24, srv407382994=20, srv1057560017=1, srv1569572969=5, srv1896583091=15, srv22393552=17, srv1829522002=10, srv1467439518=3, srv770031278=28, srv1112500671=2, srv1744097141=9, srv447371676=21, srv493398986=22, srv1888546492=13, srv1609973057=6, srv729764493=27, srv1675388148=7, srv2014217786=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 04:56:23,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 04:56:23,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 04:56:23,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 04:56:23,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 04:56:23,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 04:56:23,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 04:56:23,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 04:56:23,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 04:56:23,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 04:56:23,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 04:56:23,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 04:56:23,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 04:56:23,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 04:56:23,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 04:56:23,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 04:56:23,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 04:56:23,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 04:56:23,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 04:56:23,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 04:56:23,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 04:56:23,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 04:56:23,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 04:56:23,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 04:56:23,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 04:56:23,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 04:56:23,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 04:56:23,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 04:56:23,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 04:56:23,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 04:56:23,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 04:56:23,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-07 04:56:23,248 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 04:56:23,249 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1005830201:90 , srv1057560017:90 , srv1112500671:90 , srv1467439518:90 , srv1538270042:90 , srv1569572969:90 , srv1609973057:90 , srv1675388148:90 , srv1724082023:90 , srv1744097141:90 , srv1829522002:90 , srv1855074598:90 , srv1857487569:90 , srv1888546492:90 , srv1892281441:90 , srv1896583091:90 , srv2014217786:90 , srv22393552:90 , srv243357887:90 , srv286221052:90 , srv407382994:90 , srv447371676:90 , srv493398986:90 , srv538845767:90 , srv559705309:90 , srv575951082:90 , srv611698918:90 , srv729764493:90 , srv770031278:90 , srv786255269:90 }
2024-03-07 04:56:23,262 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=127 (was 190), ProcessCount=170 (was 170), AvailableMemoryMB=5719 (was 6020)