2024-03-04 15:53:16,915 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-03-04 15:53:17,281 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-03-04 15:53:17,392 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-03-04 15:53:17,464 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-04 15:53:17,471 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-04 15:53:17,486 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=272, ProcessCount=171, AvailableMemoryMB=7400
2024-03-04 15:53:17,489 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-04 15:53:17,490 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-04 15:53:17,621 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1000551755:84 , srv1015964345:84 , srv102575050:84 , srv1026341256:84 , srv1036839579:84 , srv1047465214:84 , srv1060653929:84 , srv1123323211:84 , srv1158623377:84 , srv1185724046:84 , srv1200054538:84 , srv1212659790:84 , srv1215526370:84 , srv1233572515:84 , srv1304527888:84 , srv1339719841:84 , srv1369737746:84 , srv1383848619:84 , srv1457347326:84 , srv1459860266:84 , srv1509403764:84 , srv1610730038:84 , srv1627284945:84 , srv163925158:84 , srv1670321590:84 , srv1673088105:84 , srv169101699:84 , srv1721211607:84 , srv1722522321:84 , srv1734437485:84 , srv1740823751:84 , srv1754925412:84 , srv1759442264:84 , srv1840446700:84 , srv1866138429:84 , srv1866349396:84 , srv1866781180:84 , srv189752035:84 , srv1909064112:84 , srv1914532191:84 , srv1926241637:84 , srv1928833588:84 , srv1958804378:84 , srv1967278805:84 , srv1996629718:84 , srv2014761222:84 , srv2027596397:84 , srv2046608026:84 , srv2065721178:84 , srv2069867151:84 , srv2090022030:84 , srv2096256557:84 , srv2128457898:84 , srv245681872:84 , srv246520594:84 , srv26097267:84 , srv272199544:84 , srv273331617:84 , srv288406703:84 , srv293564775:84 , srv318598421:84 , srv333647942:84 , srv336960318:84 , srv33857861:84 , srv339870641:84 , srv343722921:84 , srv363064955:84 , srv36578996:84 , srv367259247:84 , srv405385196:84 , srv421114357:84 , srv450433295:84 , srv473918185:84 , srv477978123:84 , srv521565876:84 , srv559502189:84 , srv560559254:84 , srv565453473:84 , srv598113637:84 , srv61313177:84 , srv649254805:84 , srv653875360:84 , srv671084513:84 , srv684493372:84 , srv700586633:84 , srv705783928:84 , srv726793991:84 , srv737979975:84 , srv762655395:84 , srv764911011:84 , srv794760323:84 , srv832065481:84 , srv837859479:84 , srv881131282:84 , srv896368575:84 , srv921372211:84 , srv923824175:84 , srv952227798:84 , srv975020717:84 , srv322390054:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-03-04 15:53:17,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-04 15:53:17,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv921372211=96, srv1015964345=1, srv565453473=78, srv26097267=55, srv1383848619=17, srv343722921=66, srv1457347326=18, srv1233572515=13, srv2128457898=52, srv881131282=94, srv2069867151=49, srv1000551755=0, srv293564775=59, srv1339719841=15, srv1909064112=38, srv1215526370=12, srv1123323211=7, srv1158623377=8, srv322390054=61, srv1026341256=3, srv521565876=75, srv2065721178=48, srv1627284945=22, srv1967278805=43, srv952227798=98, srv1304527888=14, srv923824175=97, srv421114357=71, srv1721211607=27, srv737979975=88, srv1926241637=40, srv367259247=69, srv700586633=85, srv832065481=92, srv1996629718=44, srv36578996=68, srv837859479=93, srv246520594=54, srv339870641=65, srv975020717=99, srv272199544=56, srv273331617=57, srv764911011=90, srv473918185=73, srv649254805=81, srv245681872=53, srv33857861=64, srv1866138429=34, srv363064955=67, srv405385196=70, srv1369737746=16, srv762655395=89, srv653875360=82, srv2014761222=45, srv671084513=83, srv2027596397=46, srv1914532191=39, srv726793991=87, srv1610730038=21, srv1200054538=10, srv189752035=37, srv318598421=60, srv1722522321=28, srv1212659790=11, srv1740823751=30, srv336960318=63, srv2046608026=47, srv1958804378=42, srv2096256557=51, srv684493372=84, srv598113637=79, srv477978123=74, srv896368575=95, srv560559254=77, srv1866349396=35, srv1185724046=9, srv1670321590=24, srv1673088105=25, srv1759442264=32, srv1509403764=20, srv61313177=80, srv1036839579=4, srv1840446700=33, srv1047465214=5, srv794760323=91, srv450433295=72, srv1060653929=6, srv559502189=76, srv1866781180=36, srv1459860266=19, srv102575050=2, srv333647942=62, srv288406703=58, srv705783928=86, srv1754925412=31, srv169101699=26, srv2090022030=50, srv1928833588=41, srv163925158=23, srv1734437485=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-04 15:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-04 15:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-04 15:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-04 15:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-04 15:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-04 15:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-04 15:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-04 15:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-04 15:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-04 15:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-04 15:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-04 15:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-04 15:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-04 15:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-04 15:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-04 15:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-04 15:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-04 15:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-04 15:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-04 15:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-04 15:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-04 15:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-04 15:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-04 15:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-04 15:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-04 15:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-04 15:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-04 15:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-04 15:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-04 15:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-04 15:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-04 15:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-04 15:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-04 15:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-04 15:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-04 15:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-04 15:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-04 15:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-04 15:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-04 15:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-04 15:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-04 15:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-04 15:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-04 15:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-04 15:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-04 15:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-04 15:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-04 15:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-04 15:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-04 15:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-04 15:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-04 15:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-04 15:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-04 15:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-04 15:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-04 15:53:17,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-04 15:53:17,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-04 15:53:17,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-04 15:53:17,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-04 15:53:17,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-04 15:53:17,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-04 15:53:17,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-04 15:53:17,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-04 15:53:17,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-04 15:53:17,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-04 15:53:17,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-04 15:53:17,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-04 15:53:17,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-04 15:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-04 15:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-04 15:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-04 15:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-04 15:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-04 15:53:17,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-04 15:53:17,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-04 15:53:17,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-04 15:53:17,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-04 15:53:17,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-04 15:53:17,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-04 15:53:17,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-04 15:53:17,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-04 15:53:17,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-04 15:53:17,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-04 15:53:17,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-04 15:53:17,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-04 15:53:17,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-04 15:53:17,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-04 15:53:17,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-04 15:53:17,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-04 15:53:17,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-04 15:53:17,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-04 15:53:17,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-04 15:53:17,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-04 15:53:17,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-04 15:53:17,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-04 15:53:17,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-04 15:53:17,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-04 15:53:17,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-04 15:53:17,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-04 15:53:17,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-04 15:53:17,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-04 15:53:18,072 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-04 15:53:18,072 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-04 15:53:18,073 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-04 15:53:18,469 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 695 ms to try 5000 different iterations.  Found a solution that moves 4698 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17938417725745337. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038045201225992024, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05789614914048618, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.522, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038045201225992024, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1745, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24225, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-04 15:53:18,507 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1047465214:82 , srv1123323211:82 , srv1734437485:82 , srv705783928:82 , srv1233572515:83 , srv1339719841:83 , srv1509403764:83 , srv1673088105:83 , srv1740823751:83 , srv1909064112:83 , srv1967278805:83 , srv2069867151:83 , srv246520594:83 , srv26097267:83 , srv273331617:83 , srv288406703:83 , srv653875360:83 , srv764911011:83 , srv923824175:83 , srv1754925412:84 , srv245681872:84 , srv560559254:84 , srv1369737746:85 , srv1627284945:85 , srv163925158:85 , srv189752035:85 , srv1958804378:85 , srv2027596397:85 , srv2065721178:85 , srv421114357:85 , srv1036839579:86 , srv1060653929:86 , srv1158623377:86 , srv1185724046:86 , srv1215526370:86 , srv1457347326:86 , srv1926241637:86 , srv1996629718:86 , srv272199544:86 , srv293564775:86 , srv318598421:86 , srv333647942:86 , srv405385196:86 , srv477978123:86 , srv559502189:86 , srv565453473:86 , srv700586633:86 , srv832065481:86 , srv1000551755:87 , srv102575050:87 , srv1200054538:87 , srv1212659790:87 , srv1383848619:87 , srv1721211607:87 , srv1866349396:87 , srv2046608026:87 , srv2096256557:87 , srv336960318:87 , srv649254805:87 , srv671084513:87 , srv737979975:87 , srv794760323:87 , srv896368575:87 , srv975020717:87 , srv1304527888:88 , srv1610730038:88 , srv169101699:88 , srv1840446700:88 , srv1866781180:88 , srv2014761222:88 , srv2128457898:88 , srv339870641:88 , srv363064955:88 , srv367259247:88 , srv473918185:88 , srv521565876:88 , srv598113637:88 , srv684493372:88 , srv837859479:88 , srv1914532191:89 , srv450433295:89 , srv726793991:89 , srv762655395:89 , srv881131282:89 , srv952227798:89 , srv1026341256:90 , srv1670321590:90 , srv1759442264:90 , srv1928833588:90 , srv2090022030:90 , srv343722921:90 , srv921372211:90 , srv1015964345:91 , srv1722522321:91 , srv1866138429:91 , srv1459860266:92 , srv61313177:92 , srv33857861:93 , srv36578996:93 , srv322390054:428 }
2024-03-04 15:53:18,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-04 15:53:18,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv921372211=96, srv1015964345=1, srv565453473=78, srv26097267=55, srv1383848619=17, srv343722921=66, srv1457347326=18, srv1233572515=13, srv2128457898=52, srv881131282=94, srv2069867151=49, srv1000551755=0, srv293564775=59, srv1339719841=15, srv1909064112=38, srv1215526370=12, srv1123323211=7, srv1158623377=8, srv322390054=61, srv1026341256=3, srv521565876=75, srv2065721178=48, srv1627284945=22, srv1967278805=43, srv952227798=98, srv1304527888=14, srv923824175=97, srv421114357=71, srv1721211607=27, srv737979975=88, srv1926241637=40, srv367259247=69, srv700586633=85, srv832065481=92, srv1996629718=44, srv36578996=68, srv837859479=93, srv246520594=54, srv339870641=65, srv975020717=99, srv272199544=56, srv273331617=57, srv764911011=90, srv473918185=73, srv649254805=81, srv245681872=53, srv33857861=64, srv1866138429=34, srv363064955=67, srv405385196=70, srv1369737746=16, srv762655395=89, srv653875360=82, srv2014761222=45, srv671084513=83, srv2027596397=46, srv1914532191=39, srv726793991=87, srv1610730038=21, srv1200054538=10, srv189752035=37, srv318598421=60, srv1722522321=28, srv1212659790=11, srv1740823751=30, srv336960318=63, srv2046608026=47, srv1958804378=42, srv2096256557=51, srv684493372=84, srv598113637=79, srv477978123=74, srv896368575=95, srv560559254=77, srv1866349396=35, srv1185724046=9, srv1670321590=24, srv1673088105=25, srv1759442264=32, srv1509403764=20, srv61313177=80, srv1036839579=4, srv1840446700=33, srv1047465214=5, srv794760323=91, srv450433295=72, srv1060653929=6, srv559502189=76, srv1866781180=36, srv1459860266=19, srv102575050=2, srv333647942=62, srv288406703=58, srv705783928=86, srv1754925412=31, srv169101699=26, srv2090022030=50, srv1928833588=41, srv163925158=23, srv1734437485=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-04 15:53:18,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-04 15:53:18,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-04 15:53:18,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-04 15:53:18,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-04 15:53:18,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-04 15:53:18,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-04 15:53:18,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-04 15:53:18,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-04 15:53:18,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-04 15:53:18,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-04 15:53:18,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-04 15:53:18,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-04 15:53:18,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-04 15:53:18,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-04 15:53:18,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-04 15:53:18,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-04 15:53:18,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-04 15:53:18,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-04 15:53:18,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-04 15:53:18,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-04 15:53:18,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-04 15:53:18,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-04 15:53:18,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-04 15:53:18,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-04 15:53:18,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-04 15:53:18,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-04 15:53:18,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-04 15:53:18,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-04 15:53:18,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-04 15:53:18,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-04 15:53:18,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-04 15:53:18,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-04 15:53:18,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-04 15:53:18,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-04 15:53:18,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-04 15:53:18,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-04 15:53:18,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-04 15:53:18,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-04 15:53:18,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-04 15:53:18,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-04 15:53:18,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-04 15:53:18,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-04 15:53:18,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-04 15:53:18,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-04 15:53:18,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-04 15:53:18,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-04 15:53:18,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-04 15:53:18,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-04 15:53:18,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-04 15:53:18,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-04 15:53:18,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-04 15:53:18,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-04 15:53:18,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-04 15:53:18,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-04 15:53:18,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-04 15:53:18,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-04 15:53:18,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-04 15:53:18,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-04 15:53:18,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-04 15:53:18,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-04 15:53:18,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-04 15:53:18,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-04 15:53:18,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-04 15:53:18,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-04 15:53:18,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-04 15:53:18,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-04 15:53:18,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-04 15:53:18,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-04 15:53:18,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-04 15:53:18,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-04 15:53:18,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-04 15:53:18,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-04 15:53:18,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-04 15:53:18,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-04 15:53:18,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-04 15:53:18,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-04 15:53:18,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-04 15:53:18,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-04 15:53:18,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-04 15:53:18,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-04 15:53:18,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-04 15:53:18,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-04 15:53:18,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-04 15:53:18,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-04 15:53:18,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-04 15:53:18,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-04 15:53:18,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-04 15:53:18,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-04 15:53:18,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-04 15:53:18,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-04 15:53:18,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-04 15:53:18,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-04 15:53:18,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-04 15:53:18,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-04 15:53:18,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-04 15:53:18,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-04 15:53:18,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-04 15:53:18,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-04 15:53:18,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-04 15:53:18,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-04 15:53:18,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-04 15:53:18,782 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038045201225992024, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05789614914048618, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038045201225992024, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1745, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24225, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-04 15:53:18,783 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-04 15:53:18,783 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17935128115835094, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038045201225992024, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05789614914048618, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038045201225992024, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1745, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24225, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-04 15:53:19,129 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 543 ms to try 5000 different iterations.  Found a solution that moves 2691 regions; Going from a computed imbalance of 0.17935128115835094 to a new imbalance of 0.013072133041994338. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.018602503160351877); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.033269298762967234, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.299, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.018602503160351877); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043166666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-04 15:53:19,143 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1459860266:78 , srv36578996:78 , srv33857861:79 , srv61313177:80 , srv1015964345:81 , srv1866138429:81 , srv726793991:81 , srv1026341256:82 , srv169101699:82 , srv1914532191:82 , srv952227798:82 , srv1212659790:83 , srv1670321590:83 , srv1721211607:83 , srv1722522321:83 , srv1759442264:83 , srv1866349396:83 , srv2014761222:83 , srv2128457898:83 , srv339870641:83 , srv921372211:83 , srv1158623377:84 , srv1304527888:84 , srv1610730038:84 , srv1926241637:84 , srv1928833588:84 , srv2090022030:84 , srv333647942:84 , srv343722921:84 , srv450433295:84 , srv649254805:84 , srv737979975:84 , srv762655395:84 , srv881131282:84 , srv1000551755:85 , srv102575050:85 , srv1369737746:85 , srv1840446700:85 , srv189752035:85 , srv336960318:85 , srv405385196:85 , srv473918185:85 , srv559502189:85 , srv598113637:85 , srv684493372:85 , srv837859479:85 , srv1036839579:86 , srv1060653929:86 , srv1215526370:86 , srv1383848619:86 , srv1457347326:86 , srv1866781180:86 , srv1996629718:86 , srv2065721178:86 , srv2096256557:86 , srv272199544:86 , srv293564775:86 , srv363064955:86 , srv367259247:86 , srv477978123:86 , srv521565876:86 , srv565453473:86 , srv1185724046:87 , srv1339719841:87 , srv163925158:87 , srv246520594:87 , srv318598421:87 , srv896368575:87 , srv975020717:87 , srv1509403764:88 , srv1673088105:88 , srv1740823751:88 , srv26097267:88 , srv273331617:88 , srv288406703:88 , srv421114357:88 , srv671084513:88 , srv794760323:88 , srv832065481:88 , srv1123323211:89 , srv1200054538:89 , srv1627284945:89 , srv1734437485:89 , srv1958804378:89 , srv1967278805:89 , srv2069867151:89 , srv245681872:89 , srv653875360:89 , srv705783928:89 , srv1233572515:90 , srv2027596397:90 , srv2046608026:90 , srv560559254:90 , srv700586633:90 , srv764911011:90 , srv1047465214:91 , srv1754925412:91 , srv923824175:91 , srv1909064112:93 , srv322390054:511 }
2024-03-04 15:53:19,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-04 15:53:19,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv921372211=96, srv1015964345=1, srv565453473=78, srv26097267=55, srv1383848619=17, srv343722921=66, srv1457347326=18, srv1233572515=13, srv2128457898=52, srv881131282=94, srv2069867151=49, srv1000551755=0, srv293564775=59, srv1339719841=15, srv1909064112=38, srv1215526370=12, srv1123323211=7, srv1158623377=8, srv322390054=61, srv1026341256=3, srv521565876=75, srv2065721178=48, srv1627284945=22, srv1967278805=43, srv952227798=98, srv1304527888=14, srv923824175=97, srv421114357=71, srv1721211607=27, srv737979975=88, srv1926241637=40, srv367259247=69, srv700586633=85, srv832065481=92, srv1996629718=44, srv36578996=68, srv837859479=93, srv246520594=54, srv339870641=65, srv975020717=99, srv272199544=56, srv273331617=57, srv764911011=90, srv473918185=73, srv649254805=81, srv245681872=53, srv33857861=64, srv1866138429=34, srv363064955=67, srv405385196=70, srv1369737746=16, srv762655395=89, srv653875360=82, srv2014761222=45, srv671084513=83, srv2027596397=46, srv1914532191=39, srv726793991=87, srv1610730038=21, srv1200054538=10, srv189752035=37, srv318598421=60, srv1722522321=28, srv1212659790=11, srv1740823751=30, srv336960318=63, srv2046608026=47, srv1958804378=42, srv2096256557=51, srv684493372=84, srv598113637=79, srv477978123=74, srv896368575=95, srv560559254=77, srv1866349396=35, srv1185724046=9, srv1670321590=24, srv1673088105=25, srv1759442264=32, srv1509403764=20, srv61313177=80, srv1036839579=4, srv1840446700=33, srv1047465214=5, srv794760323=91, srv450433295=72, srv1060653929=6, srv559502189=76, srv1866781180=36, srv1459860266=19, srv102575050=2, srv333647942=62, srv288406703=58, srv705783928=86, srv1754925412=31, srv169101699=26, srv2090022030=50, srv1928833588=41, srv163925158=23, srv1734437485=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-04 15:53:19,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-04 15:53:19,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-04 15:53:19,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-04 15:53:19,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-04 15:53:19,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-04 15:53:19,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-04 15:53:19,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-04 15:53:19,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-04 15:53:19,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-04 15:53:19,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-04 15:53:19,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-04 15:53:19,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-04 15:53:19,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-04 15:53:19,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-04 15:53:19,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-04 15:53:19,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-04 15:53:19,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-04 15:53:19,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-04 15:53:19,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-04 15:53:19,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-04 15:53:19,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-04 15:53:19,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-04 15:53:19,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-04 15:53:19,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-04 15:53:19,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-04 15:53:19,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-04 15:53:19,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-04 15:53:19,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-04 15:53:19,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-04 15:53:19,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-04 15:53:19,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-04 15:53:19,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-04 15:53:19,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-04 15:53:19,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-04 15:53:19,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-04 15:53:19,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-04 15:53:19,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-04 15:53:19,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-04 15:53:19,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-04 15:53:19,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-04 15:53:19,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-04 15:53:19,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-04 15:53:19,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-04 15:53:19,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-04 15:53:19,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-04 15:53:19,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-04 15:53:19,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-04 15:53:19,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-04 15:53:19,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-04 15:53:19,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-04 15:53:19,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-04 15:53:19,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-04 15:53:19,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-04 15:53:19,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-04 15:53:19,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-04 15:53:19,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-04 15:53:19,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-04 15:53:19,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-04 15:53:19,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-04 15:53:19,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-04 15:53:19,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-04 15:53:19,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-04 15:53:19,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-04 15:53:19,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-04 15:53:19,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-04 15:53:19,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-04 15:53:19,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-04 15:53:19,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-04 15:53:19,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-04 15:53:19,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-04 15:53:19,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-04 15:53:19,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-04 15:53:19,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-04 15:53:19,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-04 15:53:19,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-04 15:53:19,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-04 15:53:19,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-04 15:53:19,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-04 15:53:19,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-04 15:53:19,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-04 15:53:19,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-04 15:53:19,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-04 15:53:19,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-04 15:53:19,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-04 15:53:19,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-04 15:53:19,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-04 15:53:19,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-04 15:53:19,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-04 15:53:19,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-04 15:53:19,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-04 15:53:19,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-04 15:53:19,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-04 15:53:19,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-04 15:53:19,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-04 15:53:19,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-04 15:53:19,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-04 15:53:19,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-04 15:53:19,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-04 15:53:19,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-04 15:53:19,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-04 15:53:19,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-04 15:53:19,339 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.018602503160351877); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.033269298762967234, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.018602503160351877); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043166666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-04 15:53:19,339 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-04 15:53:19,340 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013053290257259426, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.018602503160351877); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.033269298762967234, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.018602503160351877); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043166666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-04 15:53:19,652 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 442 ms to try 5000 different iterations.  Found a solution that moves 936 regions; Going from a computed imbalance of 0.013053290257259426 to a new imbalance of 6.793912380726237E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005531936323856988); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008554309236258637); MoveCostFunction : (multiplier=7.0, imbalance=0.104, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005531936323856988); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-04 15:53:19,654 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1909064112:81 , srv2046608026:81 , srv700586633:81 , srv1200054538:82 , srv2027596397:82 , srv671084513:82 , srv1627284945:83 , srv1754925412:83 , srv2090022030:83 , srv343722921:83 , srv363064955:83 , srv367259247:83 , srv560559254:83 , srv794760323:83 , srv1233572515:84 , srv1670321590:84 , srv1722522321:84 , srv1759442264:84 , srv1840446700:84 , srv1866781180:84 , srv1928833588:84 , srv1958804378:84 , srv2096256557:84 , srv245681872:84 , srv318598421:84 , srv421114357:84 , srv450433295:84 , srv473918185:84 , srv521565876:84 , srv684493372:84 , srv764911011:84 , srv832065481:84 , srv881131282:84 , srv896368575:84 , srv921372211:84 , srv923824175:84 , srv975020717:84 , srv1000551755:85 , srv102575050:85 , srv1026341256:85 , srv1036839579:85 , srv1047465214:85 , srv1185724046:85 , srv1215526370:85 , srv1383848619:85 , srv163925158:85 , srv1866138429:85 , srv1967278805:85 , srv272199544:85 , srv565453473:85 , srv598113637:85 , srv61313177:85 , srv762655395:85 , srv837859479:85 , srv1015964345:86 , srv1060653929:86 , srv1123323211:86 , srv1304527888:86 , srv1457347326:86 , srv1509403764:86 , srv1610730038:86 , srv1673088105:86 , srv1914532191:86 , srv1996629718:86 , srv2014761222:86 , srv2065721178:86 , srv2069867151:86 , srv273331617:86 , srv293564775:86 , srv33857861:86 , srv339870641:86 , srv477978123:86 , srv653875360:86 , srv952227798:86 , srv1158623377:87 , srv1212659790:87 , srv1339719841:87 , srv1459860266:87 , srv1721211607:87 , srv1740823751:87 , srv2128457898:87 , srv246520594:87 , srv26097267:87 , srv288406703:87 , srv336960318:87 , srv36578996:87 , srv405385196:87 , srv559502189:87 , srv649254805:87 , srv705783928:87 , srv726793991:87 , srv737979975:87 , srv1369737746:88 , srv169101699:88 , srv1734437485:88 , srv1866349396:88 , srv189752035:88 , srv1926241637:88 , srv333647942:88 , srv322390054:568 }
2024-03-04 15:53:19,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-04 15:53:19,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv921372211=96, srv1015964345=1, srv565453473=78, srv26097267=55, srv1383848619=17, srv343722921=66, srv1457347326=18, srv1233572515=13, srv2128457898=52, srv881131282=94, srv2069867151=49, srv1000551755=0, srv293564775=59, srv1339719841=15, srv1909064112=38, srv1215526370=12, srv1123323211=7, srv1158623377=8, srv322390054=61, srv1026341256=3, srv521565876=75, srv2065721178=48, srv1627284945=22, srv1967278805=43, srv952227798=98, srv1304527888=14, srv923824175=97, srv421114357=71, srv1721211607=27, srv737979975=88, srv1926241637=40, srv367259247=69, srv700586633=85, srv832065481=92, srv1996629718=44, srv36578996=68, srv837859479=93, srv246520594=54, srv339870641=65, srv975020717=99, srv272199544=56, srv273331617=57, srv764911011=90, srv473918185=73, srv649254805=81, srv245681872=53, srv33857861=64, srv1866138429=34, srv363064955=67, srv405385196=70, srv1369737746=16, srv762655395=89, srv653875360=82, srv2014761222=45, srv671084513=83, srv2027596397=46, srv1914532191=39, srv726793991=87, srv1610730038=21, srv1200054538=10, srv189752035=37, srv318598421=60, srv1722522321=28, srv1212659790=11, srv1740823751=30, srv336960318=63, srv2046608026=47, srv1958804378=42, srv2096256557=51, srv684493372=84, srv598113637=79, srv477978123=74, srv896368575=95, srv560559254=77, srv1866349396=35, srv1185724046=9, srv1670321590=24, srv1673088105=25, srv1759442264=32, srv1509403764=20, srv61313177=80, srv1036839579=4, srv1840446700=33, srv1047465214=5, srv794760323=91, srv450433295=72, srv1060653929=6, srv559502189=76, srv1866781180=36, srv1459860266=19, srv102575050=2, srv333647942=62, srv288406703=58, srv705783928=86, srv1754925412=31, srv169101699=26, srv2090022030=50, srv1928833588=41, srv163925158=23, srv1734437485=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-04 15:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-04 15:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-04 15:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-04 15:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-04 15:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-04 15:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-04 15:53:19,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-04 15:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-04 15:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-04 15:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-04 15:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-04 15:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-04 15:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-04 15:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-04 15:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-04 15:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-04 15:53:19,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-04 15:53:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-04 15:53:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-04 15:53:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-04 15:53:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-04 15:53:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-04 15:53:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-04 15:53:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-04 15:53:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-04 15:53:19,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-04 15:53:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-04 15:53:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-04 15:53:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-04 15:53:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-04 15:53:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-04 15:53:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-04 15:53:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-04 15:53:19,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-04 15:53:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-04 15:53:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-04 15:53:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-04 15:53:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-04 15:53:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-04 15:53:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-04 15:53:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-04 15:53:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-04 15:53:19,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-04 15:53:19,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-04 15:53:19,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-04 15:53:19,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-04 15:53:19,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-04 15:53:19,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-04 15:53:19,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-04 15:53:19,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-04 15:53:19,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-04 15:53:19,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-04 15:53:19,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-04 15:53:19,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-04 15:53:19,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-04 15:53:19,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-04 15:53:19,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-04 15:53:19,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-04 15:53:19,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-04 15:53:19,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-04 15:53:19,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-04 15:53:19,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-04 15:53:19,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-04 15:53:19,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-04 15:53:19,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-04 15:53:19,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-04 15:53:19,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-04 15:53:19,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-04 15:53:19,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-04 15:53:19,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-04 15:53:19,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-04 15:53:19,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-04 15:53:19,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-04 15:53:19,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-04 15:53:19,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-04 15:53:19,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-04 15:53:19,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-04 15:53:19,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-04 15:53:19,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-04 15:53:19,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-04 15:53:19,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-04 15:53:19,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-04 15:53:19,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-04 15:53:19,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-04 15:53:19,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-04 15:53:19,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-04 15:53:19,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-04 15:53:19,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-04 15:53:19,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-04 15:53:19,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-04 15:53:19,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-04 15:53:19,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-04 15:53:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-04 15:53:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-04 15:53:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-04 15:53:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-04 15:53:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-04 15:53:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-04 15:53:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-04 15:53:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-04 15:53:19,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-04 15:53:19,833 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005531936323856988); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008554309236258637); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005531936323856988); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-04 15:53:19,833 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-04 15:53:19,834 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.728372259909145E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005531936323856988); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008554309236258637); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005531936323856988); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-04 15:53:20,226 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 520 ms to try 5000 different iterations.  Found a solution that moves 212 regions; Going from a computed imbalance of 6.728372259909145E-4 to a new imbalance of 2.006189739742136E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.023555555555555555); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-04 15:53:20,227 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1185724046:83 , srv1734437485:83 , srv1928833588:83 , srv2069867151:83 , srv336960318:83 , srv1015964345:84 , srv1047465214:84 , srv1215526370:84 , srv1304527888:84 , srv1369737746:84 , srv1383848619:84 , srv1457347326:84 , srv1610730038:84 , srv1673088105:84 , srv169101699:84 , srv1722522321:84 , srv1740823751:84 , srv1866349396:84 , srv1866781180:84 , srv189752035:84 , srv1926241637:84 , srv1967278805:84 , srv1996629718:84 , srv2014761222:84 , srv2046608026:84 , srv2065721178:84 , srv2128457898:84 , srv26097267:84 , srv288406703:84 , srv293564775:84 , srv333647942:84 , srv33857861:84 , srv36578996:84 , srv367259247:84 , srv405385196:84 , srv477978123:84 , srv521565876:84 , srv559502189:84 , srv598113637:84 , srv649254805:84 , srv653875360:84 , srv705783928:84 , srv762655395:84 , srv794760323:84 , srv832065481:84 , srv837859479:84 , srv896368575:84 , srv923824175:84 , srv975020717:84 , srv1000551755:85 , srv1026341256:85 , srv1036839579:85 , srv1060653929:85 , srv1123323211:85 , srv1158623377:85 , srv1200054538:85 , srv1212659790:85 , srv1233572515:85 , srv1339719841:85 , srv1459860266:85 , srv1509403764:85 , srv1627284945:85 , srv163925158:85 , srv1670321590:85 , srv1721211607:85 , srv1754925412:85 , srv1759442264:85 , srv1840446700:85 , srv1866138429:85 , srv1909064112:85 , srv1914532191:85 , srv1958804378:85 , srv2027596397:85 , srv2090022030:85 , srv2096256557:85 , srv245681872:85 , srv246520594:85 , srv272199544:85 , srv273331617:85 , srv318598421:85 , srv339870641:85 , srv363064955:85 , srv421114357:85 , srv450433295:85 , srv473918185:85 , srv560559254:85 , srv565453473:85 , srv61313177:85 , srv671084513:85 , srv684493372:85 , srv700586633:85 , srv726793991:85 , srv737979975:85 , srv764911011:85 , srv952227798:85 , srv102575050:86 , srv343722921:86 , srv881131282:86 , srv921372211:86 , srv322390054:635 }
2024-03-04 15:53:20,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-04 15:53:20,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv921372211=96, srv1015964345=1, srv565453473=78, srv26097267=55, srv1383848619=17, srv343722921=66, srv1457347326=18, srv1233572515=13, srv2128457898=52, srv881131282=94, srv2069867151=49, srv1000551755=0, srv293564775=59, srv1339719841=15, srv1909064112=38, srv1215526370=12, srv1123323211=7, srv1158623377=8, srv322390054=61, srv1026341256=3, srv521565876=75, srv2065721178=48, srv1627284945=22, srv1967278805=43, srv952227798=98, srv1304527888=14, srv923824175=97, srv421114357=71, srv1721211607=27, srv737979975=88, srv1926241637=40, srv367259247=69, srv700586633=85, srv832065481=92, srv1996629718=44, srv36578996=68, srv837859479=93, srv246520594=54, srv339870641=65, srv975020717=99, srv272199544=56, srv273331617=57, srv764911011=90, srv473918185=73, srv649254805=81, srv245681872=53, srv33857861=64, srv1866138429=34, srv363064955=67, srv405385196=70, srv1369737746=16, srv762655395=89, srv653875360=82, srv2014761222=45, srv671084513=83, srv2027596397=46, srv1914532191=39, srv726793991=87, srv1610730038=21, srv1200054538=10, srv189752035=37, srv318598421=60, srv1722522321=28, srv1212659790=11, srv1740823751=30, srv336960318=63, srv2046608026=47, srv1958804378=42, srv2096256557=51, srv684493372=84, srv598113637=79, srv477978123=74, srv896368575=95, srv560559254=77, srv1866349396=35, srv1185724046=9, srv1670321590=24, srv1673088105=25, srv1759442264=32, srv1509403764=20, srv61313177=80, srv1036839579=4, srv1840446700=33, srv1047465214=5, srv794760323=91, srv450433295=72, srv1060653929=6, srv559502189=76, srv1866781180=36, srv1459860266=19, srv102575050=2, srv333647942=62, srv288406703=58, srv705783928=86, srv1754925412=31, srv169101699=26, srv2090022030=50, srv1928833588=41, srv163925158=23, srv1734437485=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-04 15:53:20,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-04 15:53:20,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-04 15:53:20,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-04 15:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-04 15:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-04 15:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-04 15:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-04 15:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-04 15:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-04 15:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-04 15:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-04 15:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-04 15:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-04 15:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-04 15:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-04 15:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-04 15:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-04 15:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-04 15:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-04 15:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-04 15:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-04 15:53:20,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-04 15:53:20,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-04 15:53:20,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-04 15:53:20,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-04 15:53:20,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-04 15:53:20,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-04 15:53:20,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-04 15:53:20,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-04 15:53:20,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-04 15:53:20,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-04 15:53:20,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-04 15:53:20,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-04 15:53:20,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-04 15:53:20,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-04 15:53:20,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-04 15:53:20,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-04 15:53:20,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-04 15:53:20,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-04 15:53:20,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-04 15:53:20,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-04 15:53:20,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-04 15:53:20,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-04 15:53:20,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-04 15:53:20,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-04 15:53:20,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-04 15:53:20,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-04 15:53:20,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-04 15:53:20,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-04 15:53:20,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-04 15:53:20,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-04 15:53:20,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-04 15:53:20,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-04 15:53:20,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-04 15:53:20,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-04 15:53:20,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-04 15:53:20,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-04 15:53:20,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-04 15:53:20,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-04 15:53:20,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-04 15:53:20,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-04 15:53:20,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-04 15:53:20,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-04 15:53:20,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-04 15:53:20,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-04 15:53:20,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-04 15:53:20,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-04 15:53:20,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-04 15:53:20,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-04 15:53:20,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-04 15:53:20,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-04 15:53:20,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-04 15:53:20,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-04 15:53:20,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-04 15:53:20,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-04 15:53:20,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-04 15:53:20,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-04 15:53:20,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-04 15:53:20,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-04 15:53:20,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-04 15:53:20,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-04 15:53:20,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-04 15:53:20,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-04 15:53:20,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-04 15:53:20,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-04 15:53:20,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-04 15:53:20,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-04 15:53:20,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-04 15:53:20,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-04 15:53:20,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-04 15:53:20,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-04 15:53:20,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-04 15:53:20,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-04 15:53:20,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-04 15:53:20,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-04 15:53:20,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-04 15:53:20,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-04 15:53:20,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-04 15:53:20,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-04 15:53:20,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-04 15:53:20,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-04 15:53:20,357 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.8577441669512895E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-04 15:53:20,358 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1185724046:83 , srv1734437485:83 , srv1928833588:83 , srv2069867151:83 , srv336960318:83 , srv1015964345:84 , srv1047465214:84 , srv1215526370:84 , srv1304527888:84 , srv1369737746:84 , srv1383848619:84 , srv1457347326:84 , srv1610730038:84 , srv1673088105:84 , srv169101699:84 , srv1722522321:84 , srv1740823751:84 , srv1866349396:84 , srv1866781180:84 , srv189752035:84 , srv1926241637:84 , srv1967278805:84 , srv1996629718:84 , srv2014761222:84 , srv2046608026:84 , srv2065721178:84 , srv2128457898:84 , srv26097267:84 , srv288406703:84 , srv293564775:84 , srv333647942:84 , srv33857861:84 , srv36578996:84 , srv367259247:84 , srv405385196:84 , srv477978123:84 , srv521565876:84 , srv559502189:84 , srv598113637:84 , srv649254805:84 , srv653875360:84 , srv705783928:84 , srv762655395:84 , srv794760323:84 , srv832065481:84 , srv837859479:84 , srv896368575:84 , srv923824175:84 , srv975020717:84 , srv1000551755:85 , srv1026341256:85 , srv1036839579:85 , srv1060653929:85 , srv1123323211:85 , srv1158623377:85 , srv1200054538:85 , srv1212659790:85 , srv1233572515:85 , srv1339719841:85 , srv1459860266:85 , srv1509403764:85 , srv1627284945:85 , srv163925158:85 , srv1670321590:85 , srv1721211607:85 , srv1754925412:85 , srv1759442264:85 , srv1840446700:85 , srv1866138429:85 , srv1909064112:85 , srv1914532191:85 , srv1958804378:85 , srv2027596397:85 , srv2090022030:85 , srv2096256557:85 , srv245681872:85 , srv246520594:85 , srv272199544:85 , srv273331617:85 , srv318598421:85 , srv339870641:85 , srv363064955:85 , srv421114357:85 , srv450433295:85 , srv473918185:85 , srv560559254:85 , srv565453473:85 , srv61313177:85 , srv671084513:85 , srv684493372:85 , srv700586633:85 , srv726793991:85 , srv737979975:85 , srv764911011:85 , srv952227798:85 , srv102575050:86 , srv343722921:86 , srv881131282:86 , srv921372211:86 , srv322390054:635 }
2024-03-04 15:53:20,384 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=272 (was 272), ProcessCount=171 (was 171), AvailableMemoryMB=6671 (was 7400)
2024-03-04 15:53:20,393 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=272, ProcessCount=171, AvailableMemoryMB=6670
2024-03-04 15:53:20,394 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-04 15:53:20,394 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-03-04 15:53:20,406 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1024568846:84 , srv1028376192:84 , srv1054128987:84 , srv1102997409:84 , srv1130458669:84 , srv136502988:84 , srv1449630910:84 , srv1450309106:84 , srv1471618668:84 , srv1649546563:84 , srv1706387292:84 , srv1713497095:84 , srv1819199391:84 , srv1822203496:84 , srv1887918467:84 , srv1922465461:84 , srv1985406095:84 , srv2059593149:84 , srv2071312191:84 , srv307859144:84 , srv457062558:84 , srv481684725:84 , srv520894957:84 , srv590232330:84 , srv637882122:84 , srv711915283:84 , srv843793838:84 , srv861546565:84 , srv961180198:84 , srv1439553030:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-03-04 15:53:20,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-04 15:53:20,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1887918467=15, srv1819199391=13, srv1028376192=1, srv481684725=22, srv1471618668=9, srv2071312191=19, srv1450309106=8, srv1922465461=16, srv861546565=28, srv457062558=21, srv307859144=20, srv711915283=26, srv1706387292=11, srv520894957=23, srv1102997409=3, srv1130458669=4, srv590232330=24, srv843793838=27, srv1449630910=7, srv1713497095=12, srv136502988=5, srv961180198=29, srv1024568846=0, srv1439553030=6, srv1822203496=14, srv1054128987=2, srv1649546563=10, srv2059593149=18, srv1985406095=17, srv637882122=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-04 15:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-04 15:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-04 15:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-04 15:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-04 15:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-04 15:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-04 15:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-04 15:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-04 15:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-04 15:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-04 15:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-04 15:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-04 15:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-04 15:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-04 15:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-04 15:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-04 15:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-04 15:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-04 15:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-04 15:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-04 15:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-04 15:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-04 15:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-04 15:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-04 15:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-04 15:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-04 15:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-04 15:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-04 15:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-04 15:53:20,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-04 15:53:20,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-04 15:53:20,439 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0708969272578126, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-04 15:53:20,439 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912416437167073, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0708969272578126, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-03-04 15:55:20,427 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5461387 different iterations.  Found a solution that moves 1854 regions; Going from a computed imbalance of 0.9912416437167073 to a new imbalance of 4.3273284898463826E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6866666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-04 15:55:20,435 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1024568846:90 , srv1028376192:90 , srv1054128987:90 , srv1102997409:90 , srv1130458669:90 , srv136502988:90 , srv1439553030:90 , srv1449630910:90 , srv1450309106:90 , srv1471618668:90 , srv1649546563:90 , srv1706387292:90 , srv1713497095:90 , srv1819199391:90 , srv1822203496:90 , srv1887918467:90 , srv1922465461:90 , srv1985406095:90 , srv2059593149:90 , srv2071312191:90 , srv307859144:90 , srv457062558:90 , srv481684725:90 , srv520894957:90 , srv590232330:90 , srv637882122:90 , srv711915283:90 , srv843793838:90 , srv861546565:90 , srv961180198:90 }
2024-03-04 15:55:20,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-04 15:55:20,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1887918467=15, srv1819199391=13, srv1028376192=1, srv481684725=22, srv1471618668=9, srv2071312191=19, srv1450309106=8, srv1922465461=16, srv861546565=28, srv457062558=21, srv307859144=20, srv711915283=26, srv1706387292=11, srv520894957=23, srv1102997409=3, srv1130458669=4, srv590232330=24, srv843793838=27, srv1449630910=7, srv1713497095=12, srv136502988=5, srv961180198=29, srv1024568846=0, srv1439553030=6, srv1822203496=14, srv1054128987=2, srv1649546563=10, srv2059593149=18, srv1985406095=17, srv637882122=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-04 15:55:20,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-04 15:55:20,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-04 15:55:20,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-04 15:55:20,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-04 15:55:20,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-04 15:55:20,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-04 15:55:20,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-04 15:55:20,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-04 15:55:20,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-04 15:55:20,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-04 15:55:20,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-04 15:55:20,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-04 15:55:20,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-04 15:55:20,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-04 15:55:20,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-04 15:55:20,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-04 15:55:20,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-04 15:55:20,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-04 15:55:20,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-04 15:55:20,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-04 15:55:20,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-04 15:55:20,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-04 15:55:20,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-04 15:55:20,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-04 15:55:20,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-04 15:55:20,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-04 15:55:20,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-04 15:55:20,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-04 15:55:20,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-04 15:55:20,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-04 15:55:20,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-04 15:55:20,458 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-04 15:55:20,458 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1024568846:90 , srv1028376192:90 , srv1054128987:90 , srv1102997409:90 , srv1130458669:90 , srv136502988:90 , srv1439553030:90 , srv1449630910:90 , srv1450309106:90 , srv1471618668:90 , srv1649546563:90 , srv1706387292:90 , srv1713497095:90 , srv1819199391:90 , srv1822203496:90 , srv1887918467:90 , srv1922465461:90 , srv1985406095:90 , srv2059593149:90 , srv2071312191:90 , srv307859144:90 , srv457062558:90 , srv481684725:90 , srv520894957:90 , srv590232330:90 , srv637882122:90 , srv711915283:90 , srv843793838:90 , srv861546565:90 , srv961180198:90 }
2024-03-04 15:55:20,475 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=240 (was 272), ProcessCount=168 (was 171), AvailableMemoryMB=5874 (was 6670)