2023-11-03 14:01:48,255 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-11-03 14:01:48,629 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-11-03 14:01:48,796 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-11-03 14:01:48,896 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-03 14:01:48,906 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-03 14:01:48,925 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=1499, ProcessCount=193, AvailableMemoryMB=2429
2023-11-03 14:01:48,928 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-03 14:01:48,928 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-03 14:01:49,162 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1023534676:84 , srv1025939375:84 , srv1054456372:84 , srv1109319440:84 , srv1121056958:84 , srv1166986985:84 , srv1167614837:84 , srv118741910:84 , srv1218177577:84 , srv1238298895:84 , srv1279933151:84 , srv1280583961:84 , srv1280662000:84 , srv1285460609:84 , srv1297996249:84 , srv1315570428:84 , srv1379519089:84 , srv1394155604:84 , srv1401315485:84 , srv1423283032:84 , srv1427733667:84 , srv1427826411:84 , srv1449372604:84 , srv1474816269:84 , srv1491969239:84 , srv1506558133:84 , srv1509126417:84 , srv1542097218:84 , srv154960820:84 , srv155714062:84 , srv1585458731:84 , srv1594206863:84 , srv1638952164:84 , srv1669444571:84 , srv168565924:84 , srv1688477327:84 , srv1693040820:84 , srv1696683148:84 , srv1719704058:84 , srv177484989:84 , srv1776237604:84 , srv179170772:84 , srv1792904793:84 , srv1806563617:84 , srv1847913975:84 , srv1848523159:84 , srv1853411703:84 , srv1863780035:84 , srv1864012626:84 , srv1890434768:84 , srv1917877923:84 , srv1921496559:84 , srv1927121277:84 , srv1959356520:84 , srv1965653320:84 , srv1971647945:84 , srv1978893205:84 , srv1993081754:84 , srv2016179797:84 , srv20265230:84 , srv2027972692:84 , srv2039917334:84 , srv2055858243:84 , srv2096989598:84 , srv2099817682:84 , srv2113479422:84 , srv2135075671:84 , srv2140995209:84 , srv222229793:84 , srv26006234:84 , srv266733426:84 , srv276456945:84 , srv282367158:84 , srv369652339:84 , srv386588024:84 , srv388315658:84 , srv408258915:84 , srv420246132:84 , srv431680612:84 , srv454942781:84 , srv473164421:84 , srv482798301:84 , srv483068177:84 , srv488697031:84 , srv515992751:84 , srv548975175:84 , srv616650042:84 , srv670846259:84 , srv73076968:84 , srv735360477:84 , srv741621990:84 , srv757257109:84 , srv791578706:84 , srv822323401:84 , srv822546561:84 , srv826661011:84 , srv854442919:84 , srv881932831:84 , srv897638803:84 , srv1200292988:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-11-03 14:01:49,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-03 14:01:49,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1121056958=4, srv482798301=82, srv1638952164=33, srv1427826411=22, srv735360477=90, srv1167614837=6, srv1109319440=3, srv1315570428=16, srv2039917334=62, srv473164421=81, srv154960820=29, srv548975175=86, srv454942781=80, srv2027972692=61, srv1200292988=8, srv2096989598=64, srv2113479422=66, srv1023534676=0, srv1285460609=14, srv854442919=97, srv741621990=91, srv1394155604=18, srv1509126417=27, srv1669444571=34, srv822323401=94, srv488697031=84, srv1423283032=20, srv179170772=42, srv20265230=60, srv1959356520=54, srv1696683148=38, srv1506558133=26, srv2140995209=68, srv1965653320=55, srv2016179797=59, srv1853411703=47, srv155714062=30, srv1848523159=46, srv369652339=74, srv1542097218=28, srv282367158=73, srv1379519089=17, srv73076968=89, srv1971647945=56, srv386588024=75, srv1054456372=2, srv1792904793=43, srv670846259=88, srv881932831=98, srv26006234=70, srv1238298895=10, srv2099817682=65, srv118741910=7, srv2135075671=67, srv1806563617=44, srv1585458731=31, srv1993081754=58, srv1978893205=57, srv168565924=35, srv1427733667=21, srv826661011=96, srv177484989=40, srv408258915=77, srv1218177577=9, srv420246132=78, srv1401315485=19, srv1491969239=25, srv222229793=69, srv1776237604=41, srv266733426=71, srv276456945=72, srv1921496559=52, srv1449372604=23, srv1474816269=24, srv1166986985=5, srv1280662000=13, srv1297996249=15, srv431680612=79, srv1917877923=51, srv1280583961=12, srv1279933151=11, srv1863780035=48, srv1927121277=53, srv1847913975=45, srv1688477327=36, srv2055858243=63, srv897638803=99, srv1594206863=32, srv1890434768=50, srv1719704058=39, srv791578706=93, srv822546561=95, srv388315658=76, srv515992751=85, srv1025939375=1, srv757257109=92, srv483068177=83, srv1864012626=49, srv1693040820=37, srv616650042=87} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-03 14:01:49,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-03 14:01:49,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-03 14:01:49,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-03 14:01:49,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-03 14:01:49,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-03 14:01:49,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-03 14:01:49,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-03 14:01:49,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-03 14:01:49,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-03 14:01:49,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-03 14:01:49,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-03 14:01:49,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-03 14:01:49,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-03 14:01:49,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-03 14:01:49,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-03 14:01:49,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-03 14:01:49,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-03 14:01:49,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-03 14:01:49,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-03 14:01:49,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-03 14:01:49,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-03 14:01:49,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-03 14:01:49,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-03 14:01:49,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-03 14:01:49,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-03 14:01:49,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-03 14:01:49,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-03 14:01:49,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-03 14:01:49,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-03 14:01:49,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-03 14:01:49,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-03 14:01:49,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-03 14:01:49,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-03 14:01:49,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-03 14:01:49,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-03 14:01:49,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-03 14:01:49,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-03 14:01:49,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-03 14:01:49,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-03 14:01:49,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-03 14:01:49,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-03 14:01:49,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-03 14:01:49,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-03 14:01:49,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-03 14:01:49,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-03 14:01:49,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-03 14:01:49,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-03 14:01:49,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-03 14:01:49,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-03 14:01:49,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-03 14:01:49,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-03 14:01:49,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-03 14:01:49,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-03 14:01:49,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-03 14:01:49,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-03 14:01:49,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-03 14:01:49,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-03 14:01:49,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-03 14:01:49,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-03 14:01:49,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-03 14:01:49,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-03 14:01:49,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-03 14:01:49,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-03 14:01:49,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-03 14:01:49,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-03 14:01:49,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-03 14:01:49,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-03 14:01:49,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-03 14:01:49,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-03 14:01:49,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-03 14:01:49,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-03 14:01:49,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-03 14:01:49,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-03 14:01:49,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-03 14:01:49,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-03 14:01:49,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-03 14:01:49,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-03 14:01:49,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-03 14:01:49,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-03 14:01:49,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-03 14:01:49,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-03 14:01:49,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-03 14:01:49,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-03 14:01:49,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-03 14:01:49,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-03 14:01:49,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-03 14:01:49,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-03 14:01:49,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-03 14:01:49,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-03 14:01:49,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-03 14:01:49,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-03 14:01:49,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-03 14:01:49,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-03 14:01:49,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-03 14:01:49,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-03 14:01:49,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-03 14:01:49,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-03 14:01:49,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-03 14:01:49,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-03 14:01:49,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-03 14:01:49,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-03 14:01:49,697 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 14:01:49,698 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-03 14:01:49,699 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-03 14:01:50,276 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 817 ms to try 5000 different iterations.  Found a solution that moves 4779 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17018487794005216. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039460130296640035, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05426170114361396, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.531, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039460130296640035, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16516666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 14:01:50,321 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1693040820:82 , srv515992751:82 , srv548975175:82 , srv1297996249:83 , srv1509126417:83 , srv386588024:83 , srv73076968:83 , srv1023534676:84 , srv1401315485:84 , srv1776237604:84 , srv1927121277:84 , srv2140995209:84 , srv266733426:84 , srv408258915:84 , srv420246132:84 , srv826661011:84 , srv1054456372:85 , srv1238298895:85 , srv1315570428:85 , srv1379519089:85 , srv1394155604:85 , srv1427826411:85 , srv1491969239:85 , srv1506558133:85 , srv1542097218:85 , srv1585458731:85 , srv1638952164:85 , srv1921496559:85 , srv1959356520:85 , srv2016179797:85 , srv2055858243:85 , srv26006234:85 , srv276456945:85 , srv881932831:85 , srv1025939375:86 , srv1121056958:86 , srv1167614837:86 , srv1218177577:86 , srv1279933151:86 , srv1280583961:86 , srv1285460609:86 , srv155714062:86 , srv1594206863:86 , srv1863780035:86 , srv1890434768:86 , srv282367158:86 , srv482798301:86 , srv483068177:86 , srv822323401:86 , srv1449372604:87 , srv154960820:87 , srv177484989:87 , srv1792904793:87 , srv1806563617:87 , srv1847913975:87 , srv1848523159:87 , srv1853411703:87 , srv1864012626:87 , srv1978893205:87 , srv1993081754:87 , srv2135075671:87 , srv222229793:87 , srv388315658:87 , srv454942781:87 , srv473164421:87 , srv488697031:87 , srv670846259:87 , srv1109319440:88 , srv1166986985:88 , srv1423283032:88 , srv1427733667:88 , srv1474816269:88 , srv1669444571:88 , srv1688477327:88 , srv1719704058:88 , srv1917877923:88 , srv1965653320:88 , srv2027972692:88 , srv2039917334:88 , srv2099817682:88 , srv2113479422:88 , srv735360477:88 , srv757257109:88 , srv822546561:88 , srv897638803:88 , srv118741910:89 , srv1696683148:89 , srv179170772:89 , srv1971647945:89 , srv20265230:89 , srv2096989598:89 , srv369652339:89 , srv431680612:89 , srv741621990:89 , srv791578706:89 , srv1280662000:90 , srv168565924:90 , srv616650042:91 , srv854442919:94 , srv1200292988:441 }
2023-11-03 14:01:50,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-03 14:01:50,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1121056958=4, srv482798301=82, srv1638952164=33, srv1427826411=22, srv735360477=90, srv1167614837=6, srv1109319440=3, srv1315570428=16, srv2039917334=62, srv473164421=81, srv154960820=29, srv548975175=86, srv454942781=80, srv2027972692=61, srv1200292988=8, srv2096989598=64, srv2113479422=66, srv1023534676=0, srv1285460609=14, srv854442919=97, srv741621990=91, srv1394155604=18, srv1509126417=27, srv1669444571=34, srv822323401=94, srv488697031=84, srv1423283032=20, srv179170772=42, srv20265230=60, srv1959356520=54, srv1696683148=38, srv1506558133=26, srv2140995209=68, srv1965653320=55, srv2016179797=59, srv1853411703=47, srv155714062=30, srv1848523159=46, srv369652339=74, srv1542097218=28, srv282367158=73, srv1379519089=17, srv73076968=89, srv1971647945=56, srv386588024=75, srv1054456372=2, srv1792904793=43, srv670846259=88, srv881932831=98, srv26006234=70, srv1238298895=10, srv2099817682=65, srv118741910=7, srv2135075671=67, srv1806563617=44, srv1585458731=31, srv1993081754=58, srv1978893205=57, srv168565924=35, srv1427733667=21, srv826661011=96, srv177484989=40, srv408258915=77, srv1218177577=9, srv420246132=78, srv1401315485=19, srv1491969239=25, srv222229793=69, srv1776237604=41, srv266733426=71, srv276456945=72, srv1921496559=52, srv1449372604=23, srv1474816269=24, srv1166986985=5, srv1280662000=13, srv1297996249=15, srv431680612=79, srv1917877923=51, srv1280583961=12, srv1279933151=11, srv1863780035=48, srv1927121277=53, srv1847913975=45, srv1688477327=36, srv2055858243=63, srv897638803=99, srv1594206863=32, srv1890434768=50, srv1719704058=39, srv791578706=93, srv822546561=95, srv388315658=76, srv515992751=85, srv1025939375=1, srv757257109=92, srv483068177=83, srv1864012626=49, srv1693040820=37, srv616650042=87} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-03 14:01:50,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-03 14:01:50,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-03 14:01:50,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-03 14:01:50,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-03 14:01:50,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-03 14:01:50,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-03 14:01:50,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-03 14:01:50,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-03 14:01:50,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-03 14:01:50,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-03 14:01:50,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-03 14:01:50,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-03 14:01:50,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-03 14:01:50,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-03 14:01:50,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-03 14:01:50,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-03 14:01:50,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-03 14:01:50,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-03 14:01:50,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-03 14:01:50,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-03 14:01:50,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-03 14:01:50,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-03 14:01:50,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-03 14:01:50,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-03 14:01:50,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-03 14:01:50,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-03 14:01:50,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-03 14:01:50,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-03 14:01:50,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-03 14:01:50,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-03 14:01:50,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-03 14:01:50,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-03 14:01:50,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-03 14:01:50,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-03 14:01:50,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-03 14:01:50,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-03 14:01:50,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-03 14:01:50,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-03 14:01:50,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-03 14:01:50,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-03 14:01:50,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-03 14:01:50,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-03 14:01:50,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-03 14:01:50,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-03 14:01:50,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-03 14:01:50,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-03 14:01:50,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-03 14:01:50,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-03 14:01:50,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-03 14:01:50,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-03 14:01:50,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-03 14:01:50,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-03 14:01:50,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-03 14:01:50,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-03 14:01:50,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-03 14:01:50,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-03 14:01:50,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-03 14:01:50,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-03 14:01:50,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-03 14:01:50,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-03 14:01:50,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-03 14:01:50,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-03 14:01:50,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-03 14:01:50,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-03 14:01:50,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-03 14:01:50,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-03 14:01:50,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-03 14:01:50,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-03 14:01:50,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-03 14:01:50,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-03 14:01:50,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-03 14:01:50,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-03 14:01:50,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-03 14:01:50,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-03 14:01:50,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-03 14:01:50,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-03 14:01:50,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-03 14:01:50,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-03 14:01:50,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-03 14:01:50,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-03 14:01:50,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-03 14:01:50,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-03 14:01:50,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-03 14:01:50,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-03 14:01:50,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-03 14:01:50,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-03 14:01:50,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-03 14:01:50,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-03 14:01:50,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-03 14:01:50,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-03 14:01:50,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-03 14:01:50,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-03 14:01:50,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-03 14:01:50,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-03 14:01:50,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-03 14:01:50,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-03 14:01:50,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-03 14:01:50,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-03 14:01:50,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-03 14:01:50,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-03 14:01:50,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-03 14:01:50,568 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039460130296640035, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05426170114361396, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039460130296640035, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16516666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 14:01:50,569 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-03 14:01:50,569 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17015141466682726, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039460130296640035, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05426170114361396, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039460130296640035, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16516666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-03 14:01:50,863 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 431 ms to try 5000 different iterations.  Found a solution that moves 2547 regions; Going from a computed imbalance of 0.17015141466682726 to a new imbalance of 0.013222946031948252. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022404415861744714); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03459562780936531, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.283, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022404415861744714); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042916666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 14:01:50,888 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv854442919:78 , srv168565924:80 , srv616650042:80 , srv1280662000:82 , srv1474816269:82 , srv2113479422:82 , srv222229793:82 , srv369652339:82 , srv1427733667:83 , srv1449372604:83 , srv1594206863:83 , srv1669444571:83 , srv1688477327:83 , srv1965653320:83 , srv2027972692:83 , srv741621990:83 , srv1166986985:84 , srv118741910:84 , srv1279933151:84 , srv1696683148:84 , srv1792904793:84 , srv1848523159:84 , srv1890434768:84 , srv2055858243:84 , srv431680612:84 , srv454942781:84 , srv473164421:84 , srv488697031:84 , srv670846259:84 , srv1109319440:85 , srv1280583961:85 , srv1542097218:85 , srv154960820:85 , srv155714062:85 , srv1719704058:85 , srv177484989:85 , srv1806563617:85 , srv1853411703:85 , srv1863780035:85 , srv20265230:85 , srv2096989598:85 , srv26006234:85 , srv420246132:85 , srv735360477:85 , srv791578706:85 , srv822323401:85 , srv822546561:85 , srv1025939375:86 , srv1054456372:86 , srv1167614837:86 , srv1238298895:86 , srv1379519089:86 , srv1423283032:86 , srv1427826411:86 , srv1491969239:86 , srv1506558133:86 , srv1847913975:86 , srv1959356520:86 , srv1971647945:86 , srv1978893205:86 , srv1993081754:86 , srv2039917334:86 , srv2099817682:86 , srv2135075671:86 , srv2140995209:86 , srv482798301:86 , srv483068177:86 , srv897638803:86 , srv1023534676:87 , srv1121056958:87 , srv1218177577:87 , srv1394155604:87 , srv1401315485:87 , srv1638952164:87 , srv1864012626:87 , srv1917877923:87 , srv1921496559:87 , srv282367158:87 , srv388315658:87 , srv1285460609:88 , srv1297996249:88 , srv1509126417:88 , srv1585458731:88 , srv1776237604:88 , srv1927121277:88 , srv266733426:88 , srv276456945:88 , srv408258915:88 , srv548975175:88 , srv73076968:88 , srv757257109:88 , srv881932831:88 , srv1315570428:89 , srv1693040820:89 , srv179170772:89 , srv2016179797:89 , srv386588024:89 , srv515992751:91 , srv826661011:91 , srv1200292988:532 }
2023-11-03 14:01:50,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-03 14:01:50,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1121056958=4, srv482798301=82, srv1638952164=33, srv1427826411=22, srv735360477=90, srv1167614837=6, srv1109319440=3, srv1315570428=16, srv2039917334=62, srv473164421=81, srv154960820=29, srv548975175=86, srv454942781=80, srv2027972692=61, srv1200292988=8, srv2096989598=64, srv2113479422=66, srv1023534676=0, srv1285460609=14, srv854442919=97, srv741621990=91, srv1394155604=18, srv1509126417=27, srv1669444571=34, srv822323401=94, srv488697031=84, srv1423283032=20, srv179170772=42, srv20265230=60, srv1959356520=54, srv1696683148=38, srv1506558133=26, srv2140995209=68, srv1965653320=55, srv2016179797=59, srv1853411703=47, srv155714062=30, srv1848523159=46, srv369652339=74, srv1542097218=28, srv282367158=73, srv1379519089=17, srv73076968=89, srv1971647945=56, srv386588024=75, srv1054456372=2, srv1792904793=43, srv670846259=88, srv881932831=98, srv26006234=70, srv1238298895=10, srv2099817682=65, srv118741910=7, srv2135075671=67, srv1806563617=44, srv1585458731=31, srv1993081754=58, srv1978893205=57, srv168565924=35, srv1427733667=21, srv826661011=96, srv177484989=40, srv408258915=77, srv1218177577=9, srv420246132=78, srv1401315485=19, srv1491969239=25, srv222229793=69, srv1776237604=41, srv266733426=71, srv276456945=72, srv1921496559=52, srv1449372604=23, srv1474816269=24, srv1166986985=5, srv1280662000=13, srv1297996249=15, srv431680612=79, srv1917877923=51, srv1280583961=12, srv1279933151=11, srv1863780035=48, srv1927121277=53, srv1847913975=45, srv1688477327=36, srv2055858243=63, srv897638803=99, srv1594206863=32, srv1890434768=50, srv1719704058=39, srv791578706=93, srv822546561=95, srv388315658=76, srv515992751=85, srv1025939375=1, srv757257109=92, srv483068177=83, srv1864012626=49, srv1693040820=37, srv616650042=87} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-03 14:01:50,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-03 14:01:50,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-03 14:01:50,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-03 14:01:50,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-03 14:01:50,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-03 14:01:50,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-03 14:01:50,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-03 14:01:50,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-03 14:01:50,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-03 14:01:50,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-03 14:01:50,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-03 14:01:50,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-03 14:01:50,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-03 14:01:50,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-03 14:01:50,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-03 14:01:50,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-03 14:01:50,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-03 14:01:50,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-03 14:01:50,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-03 14:01:50,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-03 14:01:50,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-03 14:01:50,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-03 14:01:50,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-03 14:01:50,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-03 14:01:50,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-03 14:01:50,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-03 14:01:50,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-03 14:01:50,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-03 14:01:50,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-03 14:01:50,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-03 14:01:50,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-03 14:01:50,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-03 14:01:50,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-03 14:01:50,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-03 14:01:50,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-03 14:01:50,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-03 14:01:50,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-03 14:01:50,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-03 14:01:50,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-03 14:01:50,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-03 14:01:50,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-03 14:01:50,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-03 14:01:50,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-03 14:01:50,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-03 14:01:50,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-03 14:01:50,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-03 14:01:50,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-03 14:01:50,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-03 14:01:50,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-03 14:01:50,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-03 14:01:50,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-03 14:01:50,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-03 14:01:50,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-03 14:01:50,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-03 14:01:50,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-03 14:01:50,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-03 14:01:50,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-03 14:01:50,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-03 14:01:50,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-03 14:01:50,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-03 14:01:50,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-03 14:01:50,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-03 14:01:50,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-03 14:01:50,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-03 14:01:50,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-03 14:01:50,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-03 14:01:50,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-03 14:01:50,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-03 14:01:50,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-03 14:01:50,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-03 14:01:50,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-03 14:01:50,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-03 14:01:50,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-03 14:01:50,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-03 14:01:50,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-03 14:01:50,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-03 14:01:50,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-03 14:01:50,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-03 14:01:50,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-03 14:01:50,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-03 14:01:50,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-03 14:01:50,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-03 14:01:50,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-03 14:01:50,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-03 14:01:50,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-03 14:01:50,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-03 14:01:50,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-03 14:01:50,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-03 14:01:50,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-03 14:01:50,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-03 14:01:50,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-03 14:01:50,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-03 14:01:50,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-03 14:01:50,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-03 14:01:50,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-03 14:01:50,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-03 14:01:50,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-03 14:01:50,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-03 14:01:50,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-03 14:01:50,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-03 14:01:50,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-03 14:01:51,079 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022404415861744714); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03459562780936531, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022404415861744714); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042916666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 14:01:51,079 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-03 14:01:51,079 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01320511155676437, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022404415861744714); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03459562780936531, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022404415861744714); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042916666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-03 14:01:51,467 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 515 ms to try 5000 different iterations.  Found a solution that moves 955 regions; Going from a computed imbalance of 0.01320511155676437 to a new imbalance of 4.692607726143114E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008009222767087609); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.012526067096353783); MoveCostFunction : (multiplier=7.0, imbalance=0.10611111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008009222767087609); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 14:01:51,469 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv179170772:79 , srv1917877923:82 , srv1971647945:82 , srv2039917334:82 , srv791578706:82 , srv826661011:82 , srv1285460609:83 , srv1315570428:83 , srv1423283032:83 , srv1696683148:83 , srv2016179797:83 , srv2096989598:83 , srv388315658:83 , srv757257109:83 , srv897638803:83 , srv1121056958:84 , srv154960820:84 , srv1585458731:84 , srv1719704058:84 , srv177484989:84 , srv1847913975:84 , srv1864012626:84 , srv1927121277:84 , srv20265230:84 , srv2135075671:84 , srv431680612:84 , srv515992751:84 , srv735360477:84 , srv822546561:84 , srv1109319440:85 , srv1167614837:85 , srv118741910:85 , srv1218177577:85 , srv1280662000:85 , srv1297996249:85 , srv1394155604:85 , srv1638952164:85 , srv1669444571:85 , srv1921496559:85 , srv1978893205:85 , srv1993081754:85 , srv2027972692:85 , srv2099817682:85 , srv266733426:85 , srv276456945:85 , srv282367158:85 , srv386588024:85 , srv408258915:85 , srv483068177:85 , srv741621990:85 , srv881932831:85 , srv1025939375:86 , srv1054456372:86 , srv1166986985:86 , srv1238298895:86 , srv1279933151:86 , srv1280583961:86 , srv1427826411:86 , srv1474816269:86 , srv1491969239:86 , srv1506558133:86 , srv1509126417:86 , srv1776237604:86 , srv1806563617:86 , srv1853411703:86 , srv2140995209:86 , srv369652339:86 , srv482798301:86 , srv616650042:86 , srv73076968:86 , srv822323401:86 , srv854442919:86 , srv1023534676:87 , srv1379519089:87 , srv1401315485:87 , srv1427733667:87 , srv168565924:87 , srv1688477327:87 , srv1693040820:87 , srv1792904793:87 , srv1848523159:87 , srv1863780035:87 , srv1890434768:87 , srv1959356520:87 , srv1965653320:87 , srv2055858243:87 , srv26006234:87 , srv454942781:87 , srv473164421:87 , srv488697031:87 , srv548975175:87 , srv670846259:87 , srv1449372604:88 , srv1542097218:88 , srv155714062:88 , srv1594206863:88 , srv2113479422:88 , srv222229793:88 , srv420246132:88 , srv1200292988:556 }
2023-11-03 14:01:51,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-03 14:01:51,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1121056958=4, srv482798301=82, srv1638952164=33, srv1427826411=22, srv735360477=90, srv1167614837=6, srv1109319440=3, srv1315570428=16, srv2039917334=62, srv473164421=81, srv154960820=29, srv548975175=86, srv454942781=80, srv2027972692=61, srv1200292988=8, srv2096989598=64, srv2113479422=66, srv1023534676=0, srv1285460609=14, srv854442919=97, srv741621990=91, srv1394155604=18, srv1509126417=27, srv1669444571=34, srv822323401=94, srv488697031=84, srv1423283032=20, srv179170772=42, srv20265230=60, srv1959356520=54, srv1696683148=38, srv1506558133=26, srv2140995209=68, srv1965653320=55, srv2016179797=59, srv1853411703=47, srv155714062=30, srv1848523159=46, srv369652339=74, srv1542097218=28, srv282367158=73, srv1379519089=17, srv73076968=89, srv1971647945=56, srv386588024=75, srv1054456372=2, srv1792904793=43, srv670846259=88, srv881932831=98, srv26006234=70, srv1238298895=10, srv2099817682=65, srv118741910=7, srv2135075671=67, srv1806563617=44, srv1585458731=31, srv1993081754=58, srv1978893205=57, srv168565924=35, srv1427733667=21, srv826661011=96, srv177484989=40, srv408258915=77, srv1218177577=9, srv420246132=78, srv1401315485=19, srv1491969239=25, srv222229793=69, srv1776237604=41, srv266733426=71, srv276456945=72, srv1921496559=52, srv1449372604=23, srv1474816269=24, srv1166986985=5, srv1280662000=13, srv1297996249=15, srv431680612=79, srv1917877923=51, srv1280583961=12, srv1279933151=11, srv1863780035=48, srv1927121277=53, srv1847913975=45, srv1688477327=36, srv2055858243=63, srv897638803=99, srv1594206863=32, srv1890434768=50, srv1719704058=39, srv791578706=93, srv822546561=95, srv388315658=76, srv515992751=85, srv1025939375=1, srv757257109=92, srv483068177=83, srv1864012626=49, srv1693040820=37, srv616650042=87} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-03 14:01:51,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-03 14:01:51,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-03 14:01:51,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-03 14:01:51,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-03 14:01:51,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-03 14:01:51,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-03 14:01:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-03 14:01:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-03 14:01:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-03 14:01:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-03 14:01:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-03 14:01:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-03 14:01:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-03 14:01:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-03 14:01:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-03 14:01:51,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-03 14:01:51,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-03 14:01:51,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-03 14:01:51,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-03 14:01:51,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-03 14:01:51,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-03 14:01:51,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-03 14:01:51,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-03 14:01:51,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-03 14:01:51,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-03 14:01:51,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-03 14:01:51,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-03 14:01:51,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-03 14:01:51,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-03 14:01:51,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-03 14:01:51,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-03 14:01:51,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-03 14:01:51,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-03 14:01:51,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-03 14:01:51,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-03 14:01:51,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-03 14:01:51,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-03 14:01:51,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-03 14:01:51,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-03 14:01:51,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-03 14:01:51,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-03 14:01:51,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-03 14:01:51,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-03 14:01:51,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-03 14:01:51,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-03 14:01:51,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-03 14:01:51,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-03 14:01:51,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-03 14:01:51,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-03 14:01:51,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-03 14:01:51,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-03 14:01:51,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-03 14:01:51,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-03 14:01:51,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-03 14:01:51,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-03 14:01:51,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-03 14:01:51,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-03 14:01:51,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-03 14:01:51,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-03 14:01:51,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-03 14:01:51,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-03 14:01:51,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-03 14:01:51,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-03 14:01:51,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-03 14:01:51,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-03 14:01:51,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-03 14:01:51,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-03 14:01:51,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-03 14:01:51,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-03 14:01:51,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-03 14:01:51,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-03 14:01:51,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-03 14:01:51,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-03 14:01:51,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-03 14:01:51,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-03 14:01:51,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-03 14:01:51,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-03 14:01:51,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-03 14:01:51,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-03 14:01:51,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-03 14:01:51,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-03 14:01:51,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-03 14:01:51,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-03 14:01:51,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-03 14:01:51,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-03 14:01:51,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-03 14:01:51,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-03 14:01:51,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-03 14:01:51,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-03 14:01:51,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-03 14:01:51,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-03 14:01:51,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-03 14:01:51,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-03 14:01:51,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-03 14:01:51,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-03 14:01:51,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-03 14:01:51,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-03 14:01:51,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-03 14:01:51,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-03 14:01:51,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-03 14:01:51,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-03 14:01:51,639 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008009222767087609); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.012526067096353783); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008009222767087609); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 14:01:51,639 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-03 14:01:51,640 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.6257371968906337E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008009222767087609); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.012526067096353783); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008009222767087609); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-03 14:01:51,911 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 391 ms to try 5000 different iterations.  Found a solution that moves 295 regions; Going from a computed imbalance of 4.6257371968906337E-4 to a new imbalance of 1.4220146509633174E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.03277777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 14:01:51,912 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv155714062:83 , srv1688477327:83 , srv1806563617:83 , srv2099817682:83 , srv454942781:83 , srv757257109:83 , srv1023534676:84 , srv1025939375:84 , srv1054456372:84 , srv1166986985:84 , srv1167614837:84 , srv118741910:84 , srv1218177577:84 , srv1285460609:84 , srv1379519089:84 , srv1401315485:84 , srv1427733667:84 , srv1449372604:84 , srv1693040820:84 , srv1776237604:84 , srv1847913975:84 , srv1848523159:84 , srv1863780035:84 , srv1864012626:84 , srv1959356520:84 , srv1965653320:84 , srv1978893205:84 , srv1993081754:84 , srv20265230:84 , srv2113479422:84 , srv276456945:84 , srv282367158:84 , srv473164421:84 , srv482798301:84 , srv670846259:84 , srv854442919:84 , srv881932831:84 , srv1109319440:85 , srv1238298895:85 , srv1280583961:85 , srv1280662000:85 , srv1394155604:85 , srv1423283032:85 , srv1427826411:85 , srv1491969239:85 , srv1506558133:85 , srv1509126417:85 , srv1542097218:85 , srv1585458731:85 , srv1594206863:85 , srv1638952164:85 , srv168565924:85 , srv1719704058:85 , srv179170772:85 , srv1792904793:85 , srv1853411703:85 , srv1890434768:85 , srv1917877923:85 , srv1921496559:85 , srv1971647945:85 , srv2016179797:85 , srv2096989598:85 , srv2135075671:85 , srv222229793:85 , srv26006234:85 , srv266733426:85 , srv369652339:85 , srv386588024:85 , srv388315658:85 , srv408258915:85 , srv420246132:85 , srv431680612:85 , srv483068177:85 , srv488697031:85 , srv515992751:85 , srv548975175:85 , srv616650042:85 , srv73076968:85 , srv741621990:85 , srv791578706:85 , srv822323401:85 , srv822546561:85 , srv826661011:85 , srv897638803:85 , srv1121056958:86 , srv1279933151:86 , srv1297996249:86 , srv1315570428:86 , srv1474816269:86 , srv154960820:86 , srv1669444571:86 , srv1696683148:86 , srv177484989:86 , srv1927121277:86 , srv2027972692:86 , srv2039917334:86 , srv2055858243:86 , srv2140995209:86 , srv735360477:86 , srv1200292988:613 }
2023-11-03 14:01:51,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-03 14:01:51,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1121056958=4, srv482798301=82, srv1638952164=33, srv1427826411=22, srv735360477=90, srv1167614837=6, srv1109319440=3, srv1315570428=16, srv2039917334=62, srv473164421=81, srv154960820=29, srv548975175=86, srv454942781=80, srv2027972692=61, srv1200292988=8, srv2096989598=64, srv2113479422=66, srv1023534676=0, srv1285460609=14, srv854442919=97, srv741621990=91, srv1394155604=18, srv1509126417=27, srv1669444571=34, srv822323401=94, srv488697031=84, srv1423283032=20, srv179170772=42, srv20265230=60, srv1959356520=54, srv1696683148=38, srv1506558133=26, srv2140995209=68, srv1965653320=55, srv2016179797=59, srv1853411703=47, srv155714062=30, srv1848523159=46, srv369652339=74, srv1542097218=28, srv282367158=73, srv1379519089=17, srv73076968=89, srv1971647945=56, srv386588024=75, srv1054456372=2, srv1792904793=43, srv670846259=88, srv881932831=98, srv26006234=70, srv1238298895=10, srv2099817682=65, srv118741910=7, srv2135075671=67, srv1806563617=44, srv1585458731=31, srv1993081754=58, srv1978893205=57, srv168565924=35, srv1427733667=21, srv826661011=96, srv177484989=40, srv408258915=77, srv1218177577=9, srv420246132=78, srv1401315485=19, srv1491969239=25, srv222229793=69, srv1776237604=41, srv266733426=71, srv276456945=72, srv1921496559=52, srv1449372604=23, srv1474816269=24, srv1166986985=5, srv1280662000=13, srv1297996249=15, srv431680612=79, srv1917877923=51, srv1280583961=12, srv1279933151=11, srv1863780035=48, srv1927121277=53, srv1847913975=45, srv1688477327=36, srv2055858243=63, srv897638803=99, srv1594206863=32, srv1890434768=50, srv1719704058=39, srv791578706=93, srv822546561=95, srv388315658=76, srv515992751=85, srv1025939375=1, srv757257109=92, srv483068177=83, srv1864012626=49, srv1693040820=37, srv616650042=87} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-03 14:01:51,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-03 14:01:51,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-03 14:01:51,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-03 14:01:51,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-03 14:01:51,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-03 14:01:51,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-03 14:01:51,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-03 14:01:51,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-03 14:01:51,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-03 14:01:51,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-03 14:01:51,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-03 14:01:51,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-03 14:01:51,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-03 14:01:51,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-03 14:01:51,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-03 14:01:51,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-03 14:01:51,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-03 14:01:51,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-03 14:01:51,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-03 14:01:51,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-03 14:01:51,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-03 14:01:51,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-03 14:01:51,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-03 14:01:51,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-03 14:01:51,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-03 14:01:51,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-03 14:01:51,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-03 14:01:51,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-03 14:01:51,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-03 14:01:51,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-03 14:01:51,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-03 14:01:51,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-03 14:01:51,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-03 14:01:51,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-03 14:01:51,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-03 14:01:51,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-03 14:01:51,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-03 14:01:51,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-03 14:01:51,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-03 14:01:51,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-03 14:01:51,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-03 14:01:51,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-03 14:01:51,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-03 14:01:51,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-03 14:01:51,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-03 14:01:51,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-03 14:01:51,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-03 14:01:51,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-03 14:01:51,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-03 14:01:51,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-03 14:01:51,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-03 14:01:51,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-03 14:01:51,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-03 14:01:51,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-03 14:01:51,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-03 14:01:51,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-03 14:01:51,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-03 14:01:51,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-03 14:01:51,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-03 14:01:51,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-03 14:01:51,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-03 14:01:51,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-03 14:01:51,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-03 14:01:51,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-03 14:01:51,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-03 14:01:51,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-03 14:01:51,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-03 14:01:51,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-03 14:01:51,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-03 14:01:51,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-03 14:01:51,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-03 14:01:51,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-03 14:01:51,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-03 14:01:51,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-03 14:01:51,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-03 14:01:51,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-03 14:01:51,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-03 14:01:51,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-03 14:01:51,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-03 14:01:51,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-03 14:01:51,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-03 14:01:51,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-03 14:01:51,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-03 14:01:51,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-03 14:01:51,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-03 14:01:51,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-03 14:01:51,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-03 14:01:51,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-03 14:01:51,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-03 14:01:51,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-03 14:01:51,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-03 14:01:51,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-03 14:01:51,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-03 14:01:51,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-03 14:01:51,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-03 14:01:51,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-03 14:01:51,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-03 14:01:51,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-03 14:01:51,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-03 14:01:51,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-03 14:01:51,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-03 14:01:52,095 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.2154512359949219E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 14:01:52,095 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv155714062:83 , srv1688477327:83 , srv1806563617:83 , srv2099817682:83 , srv454942781:83 , srv757257109:83 , srv1023534676:84 , srv1025939375:84 , srv1054456372:84 , srv1166986985:84 , srv1167614837:84 , srv118741910:84 , srv1218177577:84 , srv1285460609:84 , srv1379519089:84 , srv1401315485:84 , srv1427733667:84 , srv1449372604:84 , srv1693040820:84 , srv1776237604:84 , srv1847913975:84 , srv1848523159:84 , srv1863780035:84 , srv1864012626:84 , srv1959356520:84 , srv1965653320:84 , srv1978893205:84 , srv1993081754:84 , srv20265230:84 , srv2113479422:84 , srv276456945:84 , srv282367158:84 , srv473164421:84 , srv482798301:84 , srv670846259:84 , srv854442919:84 , srv881932831:84 , srv1109319440:85 , srv1238298895:85 , srv1280583961:85 , srv1280662000:85 , srv1394155604:85 , srv1423283032:85 , srv1427826411:85 , srv1491969239:85 , srv1506558133:85 , srv1509126417:85 , srv1542097218:85 , srv1585458731:85 , srv1594206863:85 , srv1638952164:85 , srv168565924:85 , srv1719704058:85 , srv179170772:85 , srv1792904793:85 , srv1853411703:85 , srv1890434768:85 , srv1917877923:85 , srv1921496559:85 , srv1971647945:85 , srv2016179797:85 , srv2096989598:85 , srv2135075671:85 , srv222229793:85 , srv26006234:85 , srv266733426:85 , srv369652339:85 , srv386588024:85 , srv388315658:85 , srv408258915:85 , srv420246132:85 , srv431680612:85 , srv483068177:85 , srv488697031:85 , srv515992751:85 , srv548975175:85 , srv616650042:85 , srv73076968:85 , srv741621990:85 , srv791578706:85 , srv822323401:85 , srv822546561:85 , srv826661011:85 , srv897638803:85 , srv1121056958:86 , srv1279933151:86 , srv1297996249:86 , srv1315570428:86 , srv1474816269:86 , srv154960820:86 , srv1669444571:86 , srv1696683148:86 , srv177484989:86 , srv1927121277:86 , srv2027972692:86 , srv2039917334:86 , srv2055858243:86 , srv2140995209:86 , srv735360477:86 , srv1200292988:613 }
2023-11-03 14:01:52,137 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=1507 (was 1499) - SystemLoadAverage LEAK? -, ProcessCount=190 (was 193), AvailableMemoryMB=1626 (was 2429)
2023-11-03 14:01:52,156 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=292, MaxFileDescriptor=60000, SystemLoadAverage=1507, ProcessCount=190, AvailableMemoryMB=1625
2023-11-03 14:01:52,157 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-03 14:01:52,157 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-11-03 14:01:52,170 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1012446518:84 , srv1036625786:84 , srv1202850509:84 , srv1331544386:84 , srv1392931337:84 , srv1437348829:84 , srv1454796506:84 , srv148002013:84 , srv154378138:84 , srv1577475562:84 , srv1603437822:84 , srv1626977931:84 , srv1735936444:84 , srv1857387218:84 , srv1940059214:84 , srv2038820558:84 , srv2097374454:84 , srv2142533553:84 , srv271863093:84 , srv289179433:84 , srv302966687:84 , srv357060969:84 , srv391087691:84 , srv531929746:84 , srv562882216:84 , srv6279087:84 , srv628893255:84 , srv839083425:84 , srv911776372:84 , srv1221690022:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-11-03 14:01:52,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-03 14:01:52,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1735936444=13, srv531929746=24, srv271863093=19, srv1392931337=5, srv2142533553=18, srv1437348829=6, srv2038820558=16, srv1577475562=10, srv1857387218=14, srv1012446518=0, srv154378138=9, srv2097374454=17, srv911776372=29, srv289179433=20, srv302966687=21, srv391087691=23, srv1940059214=15, srv839083425=28, srv562882216=25, srv1221690022=3, srv1036625786=1, srv1603437822=11, srv148002013=8, srv1202850509=2, srv1626977931=12, srv1454796506=7, srv1331544386=4, srv628893255=27, srv6279087=26, srv357060969=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-03 14:01:52,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-03 14:01:52,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-03 14:01:52,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-03 14:01:52,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-03 14:01:52,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-03 14:01:52,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-03 14:01:52,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-03 14:01:52,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-03 14:01:52,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-03 14:01:52,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-03 14:01:52,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-03 14:01:52,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-03 14:01:52,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-03 14:01:52,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-03 14:01:52,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-03 14:01:52,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-03 14:01:52,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-03 14:01:52,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-03 14:01:52,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-03 14:01:52,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-03 14:01:52,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-03 14:01:52,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-03 14:01:52,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-03 14:01:52,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-03 14:01:52,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-03 14:01:52,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-03 14:01:52,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-03 14:01:52,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-03 14:01:52,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-03 14:01:52,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-03 14:01:52,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-03 14:01:52,289 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.120665869832807, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 14:01:52,289 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.991257325748002, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.120665869832807, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-11-03 14:03:52,245 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6076732 different iterations.  Found a solution that moves 1849 regions; Going from a computed imbalance of 0.991257325748002 to a new imbalance of 4.3156582404131396E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6848148148148148, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 14:03:52,253 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1012446518:90 , srv1036625786:90 , srv1202850509:90 , srv1221690022:90 , srv1331544386:90 , srv1392931337:90 , srv1437348829:90 , srv1454796506:90 , srv148002013:90 , srv154378138:90 , srv1577475562:90 , srv1603437822:90 , srv1626977931:90 , srv1735936444:90 , srv1857387218:90 , srv1940059214:90 , srv2038820558:90 , srv2097374454:90 , srv2142533553:90 , srv271863093:90 , srv289179433:90 , srv302966687:90 , srv357060969:90 , srv391087691:90 , srv531929746:90 , srv562882216:90 , srv6279087:90 , srv628893255:90 , srv839083425:90 , srv911776372:90 }
2023-11-03 14:03:52,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-03 14:03:52,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1735936444=13, srv531929746=24, srv271863093=19, srv1392931337=5, srv2142533553=18, srv1437348829=6, srv2038820558=16, srv1577475562=10, srv1857387218=14, srv1012446518=0, srv154378138=9, srv2097374454=17, srv911776372=29, srv289179433=20, srv302966687=21, srv391087691=23, srv1940059214=15, srv839083425=28, srv562882216=25, srv1221690022=3, srv1036625786=1, srv1603437822=11, srv148002013=8, srv1202850509=2, srv1626977931=12, srv1454796506=7, srv1331544386=4, srv628893255=27, srv6279087=26, srv357060969=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-03 14:03:52,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-03 14:03:52,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-03 14:03:52,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-03 14:03:52,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-03 14:03:52,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-03 14:03:52,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-03 14:03:52,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-03 14:03:52,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-03 14:03:52,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-03 14:03:52,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-03 14:03:52,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-03 14:03:52,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-03 14:03:52,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-03 14:03:52,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-03 14:03:52,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-03 14:03:52,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-03 14:03:52,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-03 14:03:52,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-03 14:03:52,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-03 14:03:52,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-03 14:03:52,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-03 14:03:52,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-03 14:03:52,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-03 14:03:52,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-03 14:03:52,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-03 14:03:52,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-03 14:03:52,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-03 14:03:52,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-03 14:03:52,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-03 14:03:52,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-03 14:03:52,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-03 14:03:52,278 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 14:03:52,278 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1012446518:90 , srv1036625786:90 , srv1202850509:90 , srv1221690022:90 , srv1331544386:90 , srv1392931337:90 , srv1437348829:90 , srv1454796506:90 , srv148002013:90 , srv154378138:90 , srv1577475562:90 , srv1603437822:90 , srv1626977931:90 , srv1735936444:90 , srv1857387218:90 , srv1940059214:90 , srv2038820558:90 , srv2097374454:90 , srv2142533553:90 , srv271863093:90 , srv289179433:90 , srv302966687:90 , srv357060969:90 , srv391087691:90 , srv531929746:90 , srv562882216:90 , srv6279087:90 , srv628893255:90 , srv839083425:90 , srv911776372:90 }
2023-11-03 14:03:52,295 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 292), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=1236 (was 1507), ProcessCount=188 (was 190), AvailableMemoryMB=1776 (was 1625) - AvailableMemoryMB LEAK? -