2024-02-16 22:53:16,918 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-02-16 22:53:17,237 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-02-16 22:53:17,356 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-02-16 22:53:17,427 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-02-16 22:53:17,435 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-02-16 22:53:17,451 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=242, ProcessCount=167, AvailableMemoryMB=2078
2024-02-16 22:53:17,454 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-02-16 22:53:17,454 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-02-16 22:53:17,594 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1002999641:84 , srv1077267980:84 , srv1077318361:84 , srv1087165708:84 , srv1116147865:84 , srv1159984636:84 , srv1229986605:84 , srv1237767695:84 , srv124875127:84 , srv126408440:84 , srv1276332607:84 , srv1276660927:84 , srv1287421600:84 , srv1293717552:84 , srv1326118130:84 , srv1335231443:84 , srv1342891233:84 , srv1352984232:84 , srv1364012279:84 , srv1374309824:84 , srv1383637791:84 , srv1435583734:84 , srv144330083:84 , srv1489846726:84 , srv1490692480:84 , srv151740055:84 , srv1517524848:84 , srv1528711165:84 , srv1530992952:84 , srv1540820154:84 , srv1556631065:84 , srv1579112239:84 , srv1605695334:84 , srv1645672360:84 , srv1648981947:84 , srv1683059030:84 , srv1715839024:84 , srv1715919513:84 , srv1718990548:84 , srv1729786170:84 , srv1752352187:84 , srv1804632528:84 , srv1818296830:84 , srv1825923403:84 , srv1829787210:84 , srv1974506163:84 , srv1974766626:84 , srv2008853511:84 , srv2014694053:84 , srv2029391119:84 , srv2033026185:84 , srv2040965131:84 , srv2055338773:84 , srv205889273:84 , srv2063239586:84 , srv2093630067:84 , srv2111655988:84 , srv322292824:84 , srv326959537:84 , srv358073330:84 , srv374239978:84 , srv381503612:84 , srv405305187:84 , srv424473516:84 , srv427650903:84 , srv437902871:84 , srv494718325:84 , srv499855304:84 , srv517693999:84 , srv519614187:84 , srv531420664:84 , srv55457752:84 , srv565801838:84 , srv572078895:84 , srv616515088:84 , srv627101551:84 , srv652427364:84 , srv657515389:84 , srv663704847:84 , srv723202449:84 , srv730598171:84 , srv74047349:84 , srv748236473:84 , srv748531781:84 , srv777262691:84 , srv779455057:84 , srv818651787:84 , srv823297470:84 , srv830596995:84 , srv836642912:84 , srv838732657:84 , srv845401678:84 , srv846607982:84 , srv862359258:84 , srv867064429:84 , srv884984535:84 , srv909699950:84 , srv945453194:84 , srv989841873:84 , srv316868272:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-02-16 22:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-16 22:53:17,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv437902871=66, srv823297470=88, srv1374309824=19, srv1383637791=20, srv326959537=59, srv1276332607=10, srv1335231443=15, srv652427364=77, srv55457752=72, srv424473516=64, srv1804632528=41, srv730598171=81, srv1530992952=28, srv1159984636=5, srv2029391119=49, srv1579112239=31, srv1352984232=17, srv1490692480=24, srv1715919513=37, srv616515088=75, srv1276660927=11, srv1364012279=18, srv1825923403=43, srv124875127=8, srv945453194=98, srv1489846726=23, srv723202449=80, srv1648981947=34, srv862359258=94, srv1556631065=30, srv748531781=84, srv2040965131=51, srv1605695334=32, srv1087165708=3, srv2008853511=47, srv519614187=70, srv1528711165=27, srv1974506163=45, srv572078895=74, srv1517524848=26, srv427650903=65, srv1077267980=1, srv2055338773=52, srv836642912=90, srv1237767695=7, srv2111655988=56, srv1435583734=21, srv1287421600=12, srv1974766626=46, srv989841873=99, srv2063239586=54, srv205889273=53, srv845401678=92, srv1116147865=4, srv2014694053=48, srv1077318361=2, srv374239978=61, srv565801838=73, srv499855304=68, srv74047349=82, srv1342891233=16, srv867064429=95, srv322292824=58, srv846607982=93, srv1645672360=33, srv531420664=71, srv381503612=62, srv517693999=69, srv779455057=86, srv2093630067=55, srv663704847=79, srv494718325=67, srv1002999641=0, srv151740055=25, srv830596995=89, srv1829787210=44, srv884984535=96, srv1293717552=13, srv1718990548=38, srv316868272=57, srv627101551=76, srv1229986605=6, srv1683059030=35, srv657515389=78, srv2033026185=50, srv1818296830=42, srv126408440=9, srv777262691=85, srv405305187=63, srv1752352187=40, srv1326118130=14, srv1540820154=29, srv358073330=60, srv144330083=22, srv1729786170=39, srv909699950=97, srv1715839024=36, srv748236473=83, srv838732657=91, srv818651787=87} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-16 22:53:17,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-16 22:53:17,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-16 22:53:17,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-16 22:53:17,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-16 22:53:17,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-16 22:53:17,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-16 22:53:17,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-16 22:53:17,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-16 22:53:17,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-16 22:53:17,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-16 22:53:17,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-16 22:53:17,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-16 22:53:17,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-16 22:53:17,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-16 22:53:17,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-16 22:53:17,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-16 22:53:17,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-16 22:53:17,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-16 22:53:17,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-16 22:53:17,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-16 22:53:17,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-16 22:53:17,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-16 22:53:17,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-16 22:53:17,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-16 22:53:17,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-16 22:53:17,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-16 22:53:17,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-16 22:53:17,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-16 22:53:17,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-16 22:53:17,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-16 22:53:17,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-16 22:53:17,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-16 22:53:17,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-16 22:53:17,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-16 22:53:17,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-16 22:53:17,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-16 22:53:17,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-16 22:53:17,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-16 22:53:17,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-16 22:53:17,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-16 22:53:17,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-16 22:53:17,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-16 22:53:17,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-16 22:53:17,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-16 22:53:17,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-16 22:53:17,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-16 22:53:17,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-16 22:53:17,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-16 22:53:17,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-16 22:53:17,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-16 22:53:17,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-16 22:53:17,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-16 22:53:17,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-16 22:53:17,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-16 22:53:17,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-16 22:53:17,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-16 22:53:17,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-16 22:53:17,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-16 22:53:17,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-16 22:53:17,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-16 22:53:17,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-16 22:53:17,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-16 22:53:17,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-16 22:53:17,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-16 22:53:17,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-16 22:53:17,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-16 22:53:17,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-16 22:53:17,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-16 22:53:17,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-16 22:53:17,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-16 22:53:17,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-16 22:53:17,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-16 22:53:17,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-16 22:53:17,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-16 22:53:17,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-16 22:53:17,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-16 22:53:17,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-16 22:53:17,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-16 22:53:17,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-16 22:53:17,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-16 22:53:17,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-16 22:53:17,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-16 22:53:17,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-16 22:53:17,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-16 22:53:17,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-16 22:53:17,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-16 22:53:17,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-16 22:53:17,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-16 22:53:17,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-16 22:53:17,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-16 22:53:17,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-16 22:53:17,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-16 22:53:17,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-16 22:53:17,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-16 22:53:17,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-16 22:53:17,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-16 22:53:17,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-16 22:53:17,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-16 22:53:17,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-16 22:53:17,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-16 22:53:17,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-16 22:53:18,105 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-16 22:53:18,106 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-16 22:53:18,106 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-16 22:53:18,886 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1141 ms to try 5000 different iterations.  Found a solution that moves 4698 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17906964523598481. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039037235095556184, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06196012620272202, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.522, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039037235095556184, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17408333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24266666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-16 22:53:18,931 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv779455057:81 , srv867064429:82 , srv884984535:82 , srv1276660927:83 , srv2040965131:83 , srv205889273:83 , srv358073330:83 , srv1087165708:84 , srv1490692480:84 , srv1804632528:84 , srv1974766626:84 , srv2055338773:84 , srv405305187:84 , srv424473516:84 , srv437902871:84 , srv572078895:84 , srv845401678:84 , srv1159984636:85 , srv1237767695:85 , srv124875127:85 , srv1276332607:85 , srv1287421600:85 , srv1326118130:85 , srv1489846726:85 , srv1517524848:85 , srv1715919513:85 , srv1825923403:85 , srv2029391119:85 , srv2033026185:85 , srv322292824:85 , srv499855304:85 , srv517693999:85 , srv531420664:85 , srv55457752:85 , srv652427364:85 , srv74047349:85 , srv836642912:85 , srv838732657:85 , srv1229986605:86 , srv1335231443:86 , srv1383637791:86 , srv144330083:86 , srv1579112239:86 , srv1645672360:86 , srv1648981947:86 , srv1715839024:86 , srv1729786170:86 , srv2008853511:86 , srv2063239586:86 , srv657515389:86 , srv818651787:86 , srv830596995:86 , srv909699950:86 , srv1077318361:87 , srv1293717552:87 , srv1364012279:87 , srv1374309824:87 , srv1530992952:87 , srv1556631065:87 , srv1752352187:87 , srv2014694053:87 , srv326959537:87 , srv374239978:87 , srv427650903:87 , srv565801838:87 , srv616515088:87 , srv627101551:87 , srv663704847:87 , srv748236473:87 , srv823297470:87 , srv862359258:87 , srv1002999641:88 , srv1116147865:88 , srv151740055:88 , srv1528711165:88 , srv1605695334:88 , srv1683059030:88 , srv1818296830:88 , srv1829787210:88 , srv2111655988:88 , srv381503612:88 , srv519614187:88 , srv748531781:88 , srv1077267980:89 , srv1342891233:89 , srv1352984232:89 , srv494718325:89 , srv846607982:89 , srv945453194:89 , srv989841873:89 , srv126408440:90 , srv1540820154:90 , srv1435583734:91 , srv1974506163:91 , srv1718990548:92 , srv723202449:92 , srv777262691:92 , srv2093630067:94 , srv730598171:94 , srv316868272:437 }
2024-02-16 22:53:18,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-16 22:53:18,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv437902871=66, srv823297470=88, srv1374309824=19, srv1383637791=20, srv326959537=59, srv1276332607=10, srv1335231443=15, srv652427364=77, srv55457752=72, srv424473516=64, srv1804632528=41, srv730598171=81, srv1530992952=28, srv1159984636=5, srv2029391119=49, srv1579112239=31, srv1352984232=17, srv1490692480=24, srv1715919513=37, srv616515088=75, srv1276660927=11, srv1364012279=18, srv1825923403=43, srv124875127=8, srv945453194=98, srv1489846726=23, srv723202449=80, srv1648981947=34, srv862359258=94, srv1556631065=30, srv748531781=84, srv2040965131=51, srv1605695334=32, srv1087165708=3, srv2008853511=47, srv519614187=70, srv1528711165=27, srv1974506163=45, srv572078895=74, srv1517524848=26, srv427650903=65, srv1077267980=1, srv2055338773=52, srv836642912=90, srv1237767695=7, srv2111655988=56, srv1435583734=21, srv1287421600=12, srv1974766626=46, srv989841873=99, srv2063239586=54, srv205889273=53, srv845401678=92, srv1116147865=4, srv2014694053=48, srv1077318361=2, srv374239978=61, srv565801838=73, srv499855304=68, srv74047349=82, srv1342891233=16, srv867064429=95, srv322292824=58, srv846607982=93, srv1645672360=33, srv531420664=71, srv381503612=62, srv517693999=69, srv779455057=86, srv2093630067=55, srv663704847=79, srv494718325=67, srv1002999641=0, srv151740055=25, srv830596995=89, srv1829787210=44, srv884984535=96, srv1293717552=13, srv1718990548=38, srv316868272=57, srv627101551=76, srv1229986605=6, srv1683059030=35, srv657515389=78, srv2033026185=50, srv1818296830=42, srv126408440=9, srv777262691=85, srv405305187=63, srv1752352187=40, srv1326118130=14, srv1540820154=29, srv358073330=60, srv144330083=22, srv1729786170=39, srv909699950=97, srv1715839024=36, srv748236473=83, srv838732657=91, srv818651787=87} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-16 22:53:18,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-16 22:53:18,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-16 22:53:18,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-16 22:53:18,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-16 22:53:18,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-16 22:53:18,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-16 22:53:18,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-16 22:53:18,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-16 22:53:18,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-16 22:53:18,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-16 22:53:18,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-16 22:53:18,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-16 22:53:18,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-16 22:53:18,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-16 22:53:18,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-16 22:53:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-16 22:53:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-16 22:53:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-16 22:53:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-16 22:53:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-16 22:53:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-16 22:53:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-16 22:53:18,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-16 22:53:18,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-16 22:53:18,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-16 22:53:18,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-16 22:53:18,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-16 22:53:18,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-16 22:53:18,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-16 22:53:18,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-16 22:53:18,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-16 22:53:18,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-16 22:53:18,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-16 22:53:18,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-16 22:53:18,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-16 22:53:18,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-16 22:53:18,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-16 22:53:18,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-16 22:53:18,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-16 22:53:18,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-16 22:53:18,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-16 22:53:18,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-16 22:53:18,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-16 22:53:18,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-16 22:53:18,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-16 22:53:18,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-16 22:53:18,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-16 22:53:18,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-16 22:53:18,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-16 22:53:18,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-16 22:53:18,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-16 22:53:18,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-16 22:53:18,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-16 22:53:18,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-16 22:53:18,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-16 22:53:18,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-16 22:53:18,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-16 22:53:18,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-16 22:53:18,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-16 22:53:18,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-16 22:53:18,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-16 22:53:18,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-16 22:53:18,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-16 22:53:18,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-16 22:53:18,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-16 22:53:18,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-16 22:53:18,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-16 22:53:18,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-16 22:53:18,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-16 22:53:18,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-16 22:53:18,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-16 22:53:18,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-16 22:53:18,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-16 22:53:18,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-16 22:53:18,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-16 22:53:18,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-16 22:53:18,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-16 22:53:18,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-16 22:53:18,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-16 22:53:18,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-16 22:53:18,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-16 22:53:18,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-16 22:53:18,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-16 22:53:18,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-16 22:53:18,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-16 22:53:18,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-16 22:53:18,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-16 22:53:18,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-16 22:53:18,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-16 22:53:18,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-16 22:53:18,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-16 22:53:18,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-16 22:53:18,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-16 22:53:18,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-16 22:53:18,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-16 22:53:18,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-16 22:53:18,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-16 22:53:18,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-16 22:53:18,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-16 22:53:18,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-16 22:53:18,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-16 22:53:19,171 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039037235095556184, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06196012620272202, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039037235095556184, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17408333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24266666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-16 22:53:19,172 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-16 22:53:19,172 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17903674913688242, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039037235095556184, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06196012620272202, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039037235095556184, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17408333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24266666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-16 22:53:19,513 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 503 ms to try 5000 different iterations.  Found a solution that moves 2578 regions; Going from a computed imbalance of 0.17903674913688242 to a new imbalance of 0.015407515950182924. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020932256722405847); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038100228148407696, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28644444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020932256722405847); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044583333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-16 22:53:19,524 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv723202449:78 , srv730598171:78 , srv2093630067:79 , srv1435583734:80 , srv1540820154:81 , srv1342891233:82 , srv151740055:82 , srv1718990548:82 , srv1974506163:82 , srv494718325:82 , srv1077267980:83 , srv1116147865:83 , srv1818296830:83 , srv1829787210:83 , srv519614187:83 , srv846607982:83 , srv1002999641:84 , srv1335231443:84 , srv1352984232:84 , srv1364012279:84 , srv1374309824:84 , srv1383637791:84 , srv1683059030:84 , srv2008853511:84 , srv2111655988:84 , srv374239978:84 , srv565801838:84 , srv627101551:84 , srv777262691:84 , srv823297470:84 , srv862359258:84 , srv945453194:84 , srv989841873:84 , srv144330083:85 , srv1528711165:85 , srv1530992952:85 , srv1556631065:85 , srv1715839024:85 , srv2014694053:85 , srv2063239586:85 , srv381503612:85 , srv427650903:85 , srv616515088:85 , srv657515389:85 , srv663704847:85 , srv74047349:85 , srv818651787:85 , srv830596995:85 , srv836642912:85 , srv1077318361:86 , srv126408440:86 , srv1293717552:86 , srv1517524848:86 , srv1648981947:86 , srv1729786170:86 , srv1752352187:86 , srv499855304:86 , srv517693999:86 , srv748236473:86 , srv909699950:86 , srv1229986605:87 , srv124875127:87 , srv1276332607:87 , srv1287421600:87 , srv1489846726:87 , srv1579112239:87 , srv1605695334:87 , srv1645672360:87 , srv1715919513:87 , srv1825923403:87 , srv1974766626:87 , srv2029391119:87 , srv2033026185:87 , srv2055338773:87 , srv405305187:87 , srv424473516:87 , srv437902871:87 , srv652427364:87 , srv1326118130:88 , srv1804632528:88 , srv2040965131:88 , srv322292824:88 , srv326959537:88 , srv531420664:88 , srv572078895:88 , srv838732657:88 , srv1087165708:89 , srv1159984636:89 , srv1276660927:89 , srv205889273:89 , srv55457752:89 , srv748531781:89 , srv1237767695:90 , srv1490692480:90 , srv358073330:90 , srv779455057:91 , srv845401678:91 , srv884984535:91 , srv867064429:92 , srv316868272:523 }
2024-02-16 22:53:19,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-16 22:53:19,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv437902871=66, srv823297470=88, srv1374309824=19, srv1383637791=20, srv326959537=59, srv1276332607=10, srv1335231443=15, srv652427364=77, srv55457752=72, srv424473516=64, srv1804632528=41, srv730598171=81, srv1530992952=28, srv1159984636=5, srv2029391119=49, srv1579112239=31, srv1352984232=17, srv1490692480=24, srv1715919513=37, srv616515088=75, srv1276660927=11, srv1364012279=18, srv1825923403=43, srv124875127=8, srv945453194=98, srv1489846726=23, srv723202449=80, srv1648981947=34, srv862359258=94, srv1556631065=30, srv748531781=84, srv2040965131=51, srv1605695334=32, srv1087165708=3, srv2008853511=47, srv519614187=70, srv1528711165=27, srv1974506163=45, srv572078895=74, srv1517524848=26, srv427650903=65, srv1077267980=1, srv2055338773=52, srv836642912=90, srv1237767695=7, srv2111655988=56, srv1435583734=21, srv1287421600=12, srv1974766626=46, srv989841873=99, srv2063239586=54, srv205889273=53, srv845401678=92, srv1116147865=4, srv2014694053=48, srv1077318361=2, srv374239978=61, srv565801838=73, srv499855304=68, srv74047349=82, srv1342891233=16, srv867064429=95, srv322292824=58, srv846607982=93, srv1645672360=33, srv531420664=71, srv381503612=62, srv517693999=69, srv779455057=86, srv2093630067=55, srv663704847=79, srv494718325=67, srv1002999641=0, srv151740055=25, srv830596995=89, srv1829787210=44, srv884984535=96, srv1293717552=13, srv1718990548=38, srv316868272=57, srv627101551=76, srv1229986605=6, srv1683059030=35, srv657515389=78, srv2033026185=50, srv1818296830=42, srv126408440=9, srv777262691=85, srv405305187=63, srv1752352187=40, srv1326118130=14, srv1540820154=29, srv358073330=60, srv144330083=22, srv1729786170=39, srv909699950=97, srv1715839024=36, srv748236473=83, srv838732657=91, srv818651787=87} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-16 22:53:19,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-16 22:53:19,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-16 22:53:19,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-16 22:53:19,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-16 22:53:19,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-16 22:53:19,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-16 22:53:19,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-16 22:53:19,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-16 22:53:19,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-16 22:53:19,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-16 22:53:19,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-16 22:53:19,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-16 22:53:19,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-16 22:53:19,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-16 22:53:19,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-16 22:53:19,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-16 22:53:19,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-16 22:53:19,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-16 22:53:19,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-16 22:53:19,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-16 22:53:19,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-16 22:53:19,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-16 22:53:19,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-16 22:53:19,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-16 22:53:19,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-16 22:53:19,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-16 22:53:19,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-16 22:53:19,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-16 22:53:19,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-16 22:53:19,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-16 22:53:19,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-16 22:53:19,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-16 22:53:19,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-16 22:53:19,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-16 22:53:19,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-16 22:53:19,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-16 22:53:19,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-16 22:53:19,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-16 22:53:19,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-16 22:53:19,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-16 22:53:19,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-16 22:53:19,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-16 22:53:19,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-16 22:53:19,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-16 22:53:19,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-16 22:53:19,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-16 22:53:19,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-16 22:53:19,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-16 22:53:19,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-16 22:53:19,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-16 22:53:19,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-16 22:53:19,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-16 22:53:19,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-16 22:53:19,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-16 22:53:19,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-16 22:53:19,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-16 22:53:19,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-16 22:53:19,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-16 22:53:19,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-16 22:53:19,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-16 22:53:19,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-16 22:53:19,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-16 22:53:19,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-16 22:53:19,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-16 22:53:19,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-16 22:53:19,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-16 22:53:19,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-16 22:53:19,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-16 22:53:19,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-16 22:53:19,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-16 22:53:19,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-16 22:53:19,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-16 22:53:19,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-16 22:53:19,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-16 22:53:19,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-16 22:53:19,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-16 22:53:19,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-16 22:53:19,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-16 22:53:19,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-16 22:53:19,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-16 22:53:19,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-16 22:53:19,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-16 22:53:19,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-16 22:53:19,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-16 22:53:19,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-16 22:53:19,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-16 22:53:19,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-16 22:53:19,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-16 22:53:19,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-16 22:53:19,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-16 22:53:19,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-16 22:53:19,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-16 22:53:19,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-16 22:53:19,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-16 22:53:19,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-16 22:53:19,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-16 22:53:19,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-16 22:53:19,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-16 22:53:19,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-16 22:53:19,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-16 22:53:19,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-16 22:53:19,723 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020932256722405847); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038100228148407696, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020932256722405847); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044583333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-16 22:53:19,723 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-16 22:53:19,724 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015389464408359584, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020932256722405847); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038100228148407696, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020932256722405847); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044583333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-16 22:53:20,128 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 542 ms to try 5000 different iterations.  Found a solution that moves 979 regions; Going from a computed imbalance of 0.015389464408359584 to a new imbalance of 0.0012375559527211528. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005534190117523058); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00915019947284304); MoveCostFunction : (multiplier=7.0, imbalance=0.10877777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005534190117523058); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-16 22:53:20,130 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv748531781:80 , srv126408440:82 , srv326959537:82 , srv777262691:82 , srv845401678:82 , srv1159984636:83 , srv1237767695:83 , srv1605695334:83 , srv1718990548:83 , srv867064429:83 , srv1326118130:84 , srv1352984232:84 , srv1490692480:84 , srv1528711165:84 , srv1752352187:84 , srv1974506163:84 , srv358073330:84 , srv55457752:84 , srv838732657:84 , srv945453194:84 , srv1002999641:85 , srv1077267980:85 , srv1077318361:85 , srv1087165708:85 , srv124875127:85 , srv1287421600:85 , srv1293717552:85 , srv1579112239:85 , srv1648981947:85 , srv1715919513:85 , srv1804632528:85 , srv1825923403:85 , srv2029391119:85 , srv205889273:85 , srv2093630067:85 , srv2111655988:85 , srv322292824:85 , srv381503612:85 , srv572078895:85 , srv652427364:85 , srv663704847:85 , srv730598171:85 , srv748236473:85 , srv846607982:85 , srv884984535:85 , srv989841873:85 , srv1229986605:86 , srv1276332607:86 , srv1276660927:86 , srv1342891233:86 , srv144330083:86 , srv1489846726:86 , srv1517524848:86 , srv1530992952:86 , srv1556631065:86 , srv1645672360:86 , srv1683059030:86 , srv1729786170:86 , srv1818296830:86 , srv1974766626:86 , srv2014694053:86 , srv2033026185:86 , srv2040965131:86 , srv2055338773:86 , srv2063239586:86 , srv374239978:86 , srv405305187:86 , srv424473516:86 , srv427650903:86 , srv494718325:86 , srv499855304:86 , srv517693999:86 , srv531420664:86 , srv565801838:86 , srv616515088:86 , srv779455057:86 , srv818651787:86 , srv823297470:86 , srv830596995:86 , srv909699950:86 , srv1116147865:87 , srv1335231443:87 , srv1364012279:87 , srv1374309824:87 , srv1435583734:87 , srv151740055:87 , srv1540820154:87 , srv1715839024:87 , srv1829787210:87 , srv437902871:87 , srv519614187:87 , srv627101551:87 , srv657515389:87 , srv723202449:87 , srv836642912:87 , srv862359258:87 , srv1383637791:88 , srv2008853511:88 , srv74047349:88 , srv316868272:547 }
2024-02-16 22:53:20,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-16 22:53:20,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv437902871=66, srv823297470=88, srv1374309824=19, srv1383637791=20, srv326959537=59, srv1276332607=10, srv1335231443=15, srv652427364=77, srv55457752=72, srv424473516=64, srv1804632528=41, srv730598171=81, srv1530992952=28, srv1159984636=5, srv2029391119=49, srv1579112239=31, srv1352984232=17, srv1490692480=24, srv1715919513=37, srv616515088=75, srv1276660927=11, srv1364012279=18, srv1825923403=43, srv124875127=8, srv945453194=98, srv1489846726=23, srv723202449=80, srv1648981947=34, srv862359258=94, srv1556631065=30, srv748531781=84, srv2040965131=51, srv1605695334=32, srv1087165708=3, srv2008853511=47, srv519614187=70, srv1528711165=27, srv1974506163=45, srv572078895=74, srv1517524848=26, srv427650903=65, srv1077267980=1, srv2055338773=52, srv836642912=90, srv1237767695=7, srv2111655988=56, srv1435583734=21, srv1287421600=12, srv1974766626=46, srv989841873=99, srv2063239586=54, srv205889273=53, srv845401678=92, srv1116147865=4, srv2014694053=48, srv1077318361=2, srv374239978=61, srv565801838=73, srv499855304=68, srv74047349=82, srv1342891233=16, srv867064429=95, srv322292824=58, srv846607982=93, srv1645672360=33, srv531420664=71, srv381503612=62, srv517693999=69, srv779455057=86, srv2093630067=55, srv663704847=79, srv494718325=67, srv1002999641=0, srv151740055=25, srv830596995=89, srv1829787210=44, srv884984535=96, srv1293717552=13, srv1718990548=38, srv316868272=57, srv627101551=76, srv1229986605=6, srv1683059030=35, srv657515389=78, srv2033026185=50, srv1818296830=42, srv126408440=9, srv777262691=85, srv405305187=63, srv1752352187=40, srv1326118130=14, srv1540820154=29, srv358073330=60, srv144330083=22, srv1729786170=39, srv909699950=97, srv1715839024=36, srv748236473=83, srv838732657=91, srv818651787=87} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-16 22:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-16 22:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-16 22:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-16 22:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-16 22:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-16 22:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-16 22:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-16 22:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-16 22:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-16 22:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-16 22:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-16 22:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-16 22:53:20,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-16 22:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-16 22:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-16 22:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-16 22:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-16 22:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-16 22:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-16 22:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-16 22:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-16 22:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-16 22:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-16 22:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-16 22:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-16 22:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-16 22:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-16 22:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-16 22:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-16 22:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-16 22:53:20,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-16 22:53:20,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-16 22:53:20,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-16 22:53:20,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-16 22:53:20,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-16 22:53:20,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-16 22:53:20,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-16 22:53:20,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-16 22:53:20,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-16 22:53:20,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-16 22:53:20,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-16 22:53:20,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-16 22:53:20,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-16 22:53:20,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-16 22:53:20,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-16 22:53:20,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-16 22:53:20,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-16 22:53:20,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-16 22:53:20,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-16 22:53:20,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-16 22:53:20,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-16 22:53:20,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-16 22:53:20,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-16 22:53:20,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-16 22:53:20,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-16 22:53:20,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-16 22:53:20,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-16 22:53:20,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-16 22:53:20,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-16 22:53:20,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-16 22:53:20,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-16 22:53:20,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-16 22:53:20,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-16 22:53:20,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-16 22:53:20,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-16 22:53:20,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-16 22:53:20,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-16 22:53:20,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-16 22:53:20,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-16 22:53:20,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-16 22:53:20,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-16 22:53:20,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-16 22:53:20,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-16 22:53:20,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-16 22:53:20,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-16 22:53:20,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-16 22:53:20,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-16 22:53:20,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-16 22:53:20,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-16 22:53:20,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-16 22:53:20,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-16 22:53:20,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-16 22:53:20,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-16 22:53:20,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-16 22:53:20,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-16 22:53:20,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-16 22:53:20,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-16 22:53:20,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-16 22:53:20,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-16 22:53:20,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-16 22:53:20,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-16 22:53:20,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-16 22:53:20,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-16 22:53:20,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-16 22:53:20,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-16 22:53:20,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-16 22:53:20,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-16 22:53:20,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-16 22:53:20,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-16 22:53:20,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-16 22:53:20,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-16 22:53:20,265 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005534190117523058); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00915019947284304); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005534190117523058); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-16 22:53:20,265 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-16 22:53:20,265 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.001230700848204066, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005534190117523058); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00915019947284304); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005534190117523058); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-16 22:53:20,478 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 316 ms to try 5000 different iterations.  Found a solution that moves 261 regions; Going from a computed imbalance of 0.001230700848204066 to a new imbalance of 1.886906318581272E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.029, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-16 22:53:20,479 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1229986605:83 , srv1364012279:83 , srv1645672360:83 , srv531420664:83 , srv777262691:83 , srv1077267980:84 , srv1077318361:84 , srv1087165708:84 , srv1116147865:84 , srv1237767695:84 , srv126408440:84 , srv1276332607:84 , srv1276660927:84 , srv1293717552:84 , srv1374309824:84 , srv1383637791:84 , srv1435583734:84 , srv1489846726:84 , srv1490692480:84 , srv1530992952:84 , srv1540820154:84 , srv1556631065:84 , srv1579112239:84 , srv1683059030:84 , srv1715839024:84 , srv1715919513:84 , srv1729786170:84 , srv1829787210:84 , srv2014694053:84 , srv2029391119:84 , srv2033026185:84 , srv2093630067:84 , srv322292824:84 , srv381503612:84 , srv427650903:84 , srv437902871:84 , srv519614187:84 , srv55457752:84 , srv616515088:84 , srv652427364:84 , srv657515389:84 , srv74047349:84 , srv748236473:84 , srv779455057:84 , srv836642912:84 , srv884984535:84 , srv909699950:84 , srv1002999641:85 , srv1159984636:85 , srv124875127:85 , srv1287421600:85 , srv1326118130:85 , srv1335231443:85 , srv1352984232:85 , srv144330083:85 , srv151740055:85 , srv1517524848:85 , srv1605695334:85 , srv1648981947:85 , srv1718990548:85 , srv1752352187:85 , srv1804632528:85 , srv1818296830:85 , srv1825923403:85 , srv1974506163:85 , srv1974766626:85 , srv2008853511:85 , srv2040965131:85 , srv2055338773:85 , srv205889273:85 , srv2063239586:85 , srv2111655988:85 , srv326959537:85 , srv374239978:85 , srv405305187:85 , srv424473516:85 , srv494718325:85 , srv499855304:85 , srv517693999:85 , srv565801838:85 , srv572078895:85 , srv627101551:85 , srv663704847:85 , srv723202449:85 , srv730598171:85 , srv748531781:85 , srv818651787:85 , srv823297470:85 , srv830596995:85 , srv838732657:85 , srv845401678:85 , srv846607982:85 , srv862359258:85 , srv867064429:85 , srv945453194:85 , srv989841873:85 , srv1342891233:86 , srv1528711165:86 , srv358073330:86 , srv316868272:634 }
2024-02-16 22:53:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-16 22:53:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv437902871=66, srv823297470=88, srv1374309824=19, srv1383637791=20, srv326959537=59, srv1276332607=10, srv1335231443=15, srv652427364=77, srv55457752=72, srv424473516=64, srv1804632528=41, srv730598171=81, srv1530992952=28, srv1159984636=5, srv2029391119=49, srv1579112239=31, srv1352984232=17, srv1490692480=24, srv1715919513=37, srv616515088=75, srv1276660927=11, srv1364012279=18, srv1825923403=43, srv124875127=8, srv945453194=98, srv1489846726=23, srv723202449=80, srv1648981947=34, srv862359258=94, srv1556631065=30, srv748531781=84, srv2040965131=51, srv1605695334=32, srv1087165708=3, srv2008853511=47, srv519614187=70, srv1528711165=27, srv1974506163=45, srv572078895=74, srv1517524848=26, srv427650903=65, srv1077267980=1, srv2055338773=52, srv836642912=90, srv1237767695=7, srv2111655988=56, srv1435583734=21, srv1287421600=12, srv1974766626=46, srv989841873=99, srv2063239586=54, srv205889273=53, srv845401678=92, srv1116147865=4, srv2014694053=48, srv1077318361=2, srv374239978=61, srv565801838=73, srv499855304=68, srv74047349=82, srv1342891233=16, srv867064429=95, srv322292824=58, srv846607982=93, srv1645672360=33, srv531420664=71, srv381503612=62, srv517693999=69, srv779455057=86, srv2093630067=55, srv663704847=79, srv494718325=67, srv1002999641=0, srv151740055=25, srv830596995=89, srv1829787210=44, srv884984535=96, srv1293717552=13, srv1718990548=38, srv316868272=57, srv627101551=76, srv1229986605=6, srv1683059030=35, srv657515389=78, srv2033026185=50, srv1818296830=42, srv126408440=9, srv777262691=85, srv405305187=63, srv1752352187=40, srv1326118130=14, srv1540820154=29, srv358073330=60, srv144330083=22, srv1729786170=39, srv909699950=97, srv1715839024=36, srv748236473=83, srv838732657=91, srv818651787=87} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-16 22:53:20,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-16 22:53:20,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-16 22:53:20,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-16 22:53:20,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-16 22:53:20,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-16 22:53:20,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-16 22:53:20,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-16 22:53:20,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-16 22:53:20,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-16 22:53:20,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-16 22:53:20,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-16 22:53:20,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-16 22:53:20,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-16 22:53:20,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-16 22:53:20,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-16 22:53:20,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-16 22:53:20,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-16 22:53:20,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-16 22:53:20,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-16 22:53:20,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-16 22:53:20,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-16 22:53:20,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-16 22:53:20,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-16 22:53:20,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-16 22:53:20,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-16 22:53:20,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-16 22:53:20,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-16 22:53:20,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-16 22:53:20,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-16 22:53:20,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-16 22:53:20,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-16 22:53:20,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-16 22:53:20,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-16 22:53:20,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-16 22:53:20,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-16 22:53:20,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-16 22:53:20,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-16 22:53:20,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-16 22:53:20,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-16 22:53:20,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-16 22:53:20,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-16 22:53:20,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-16 22:53:20,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-16 22:53:20,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-16 22:53:20,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-16 22:53:20,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-16 22:53:20,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-16 22:53:20,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-16 22:53:20,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-16 22:53:20,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-16 22:53:20,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-16 22:53:20,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-16 22:53:20,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-16 22:53:20,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-16 22:53:20,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-16 22:53:20,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-16 22:53:20,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-16 22:53:20,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-16 22:53:20,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-16 22:53:20,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-16 22:53:20,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-16 22:53:20,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-16 22:53:20,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-16 22:53:20,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-16 22:53:20,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-16 22:53:20,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-16 22:53:20,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-16 22:53:20,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-16 22:53:20,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-16 22:53:20,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-16 22:53:20,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-16 22:53:20,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-16 22:53:20,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-16 22:53:20,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-16 22:53:20,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-16 22:53:20,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-16 22:53:20,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-16 22:53:20,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-16 22:53:20,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-16 22:53:20,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-16 22:53:20,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-16 22:53:20,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-16 22:53:20,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-16 22:53:20,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-16 22:53:20,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-16 22:53:20,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-16 22:53:20,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-16 22:53:20,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-16 22:53:20,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-16 22:53:20,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-16 22:53:20,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-16 22:53:20,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-16 22:53:20,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-16 22:53:20,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-16 22:53:20,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-16 22:53:20,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-16 22:53:20,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-16 22:53:20,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-16 22:53:20,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-16 22:53:20,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-16 22:53:20,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-16 22:53:20,610 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.7041502124566916E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-16 22:53:20,610 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1229986605:83 , srv1364012279:83 , srv1645672360:83 , srv531420664:83 , srv777262691:83 , srv1077267980:84 , srv1077318361:84 , srv1087165708:84 , srv1116147865:84 , srv1237767695:84 , srv126408440:84 , srv1276332607:84 , srv1276660927:84 , srv1293717552:84 , srv1374309824:84 , srv1383637791:84 , srv1435583734:84 , srv1489846726:84 , srv1490692480:84 , srv1530992952:84 , srv1540820154:84 , srv1556631065:84 , srv1579112239:84 , srv1683059030:84 , srv1715839024:84 , srv1715919513:84 , srv1729786170:84 , srv1829787210:84 , srv2014694053:84 , srv2029391119:84 , srv2033026185:84 , srv2093630067:84 , srv322292824:84 , srv381503612:84 , srv427650903:84 , srv437902871:84 , srv519614187:84 , srv55457752:84 , srv616515088:84 , srv652427364:84 , srv657515389:84 , srv74047349:84 , srv748236473:84 , srv779455057:84 , srv836642912:84 , srv884984535:84 , srv909699950:84 , srv1002999641:85 , srv1159984636:85 , srv124875127:85 , srv1287421600:85 , srv1326118130:85 , srv1335231443:85 , srv1352984232:85 , srv144330083:85 , srv151740055:85 , srv1517524848:85 , srv1605695334:85 , srv1648981947:85 , srv1718990548:85 , srv1752352187:85 , srv1804632528:85 , srv1818296830:85 , srv1825923403:85 , srv1974506163:85 , srv1974766626:85 , srv2008853511:85 , srv2040965131:85 , srv2055338773:85 , srv205889273:85 , srv2063239586:85 , srv2111655988:85 , srv326959537:85 , srv374239978:85 , srv405305187:85 , srv424473516:85 , srv494718325:85 , srv499855304:85 , srv517693999:85 , srv565801838:85 , srv572078895:85 , srv627101551:85 , srv663704847:85 , srv723202449:85 , srv730598171:85 , srv748531781:85 , srv818651787:85 , srv823297470:85 , srv830596995:85 , srv838732657:85 , srv845401678:85 , srv846607982:85 , srv862359258:85 , srv867064429:85 , srv945453194:85 , srv989841873:85 , srv1342891233:86 , srv1528711165:86 , srv358073330:86 , srv316868272:634 }
2024-02-16 22:53:20,638 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=271 (was 242) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=1325 (was 2078)
2024-02-16 22:53:20,646 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=271, ProcessCount=167, AvailableMemoryMB=1324
2024-02-16 22:53:20,646 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-02-16 22:53:20,647 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-02-16 22:53:20,654 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1011322104:84 , srv1141993953:84 , srv1191188740:84 , srv121317519:84 , srv1282671854:84 , srv1336040224:84 , srv1399977768:84 , srv1429857587:84 , srv1597511048:84 , srv1640461161:84 , srv169888357:84 , srv1854356902:84 , srv2022649293:84 , srv2061324522:84 , srv212500565:84 , srv216258425:84 , srv253961459:84 , srv265676843:84 , srv272133628:84 , srv43038436:84 , srv59368786:84 , srv599617946:84 , srv610174545:84 , srv639285115:84 , srv729114188:84 , srv748436250:84 , srv766309425:84 , srv767831273:84 , srv830331970:84 , srv792588329:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-02-16 22:53:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-16 22:53:20,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv212500565=14, srv639285115=23, srv767831273=27, srv121317519=3, srv748436250=25, srv59368786=20, srv265676843=17, srv1282671854=4, srv1429857587=7, srv766309425=26, srv610174545=22, srv1011322104=0, srv1597511048=8, srv1191188740=2, srv216258425=15, srv729114188=24, srv1336040224=5, srv599617946=21, srv1640461161=9, srv169888357=10, srv1399977768=6, srv1141993953=1, srv2061324522=13, srv253961459=16, srv43038436=19, srv1854356902=11, srv792588329=28, srv830331970=29, srv2022649293=12, srv272133628=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-16 22:53:20,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-16 22:53:20,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-16 22:53:20,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-16 22:53:20,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-16 22:53:20,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-16 22:53:20,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-16 22:53:20,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-16 22:53:20,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-16 22:53:20,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-16 22:53:20,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-16 22:53:20,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-16 22:53:20,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-16 22:53:20,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-16 22:53:20,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-16 22:53:20,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-16 22:53:20,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-16 22:53:20,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-16 22:53:20,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-16 22:53:20,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-16 22:53:20,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-16 22:53:20,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-16 22:53:20,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-16 22:53:20,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-16 22:53:20,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-16 22:53:20,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-16 22:53:20,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-16 22:53:20,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-16 22:53:20,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-16 22:53:20,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-16 22:53:20,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-16 22:53:20,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-02-16 22:53:20,678 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0886629559511773, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-16 22:53:20,678 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912472417343371, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0886629559511773, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-02-16 22:55:20,669 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6321092 different iterations.  Found a solution that moves 1850 regions; Going from a computed imbalance of 0.9912472417343371 to a new imbalance of 4.317992290299789E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6851851851851852, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-16 22:55:20,675 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1011322104:90 , srv1141993953:90 , srv1191188740:90 , srv121317519:90 , srv1282671854:90 , srv1336040224:90 , srv1399977768:90 , srv1429857587:90 , srv1597511048:90 , srv1640461161:90 , srv169888357:90 , srv1854356902:90 , srv2022649293:90 , srv2061324522:90 , srv212500565:90 , srv216258425:90 , srv253961459:90 , srv265676843:90 , srv272133628:90 , srv43038436:90 , srv59368786:90 , srv599617946:90 , srv610174545:90 , srv639285115:90 , srv729114188:90 , srv748436250:90 , srv766309425:90 , srv767831273:90 , srv792588329:90 , srv830331970:90 }
2024-02-16 22:55:20,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-16 22:55:20,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv212500565=14, srv639285115=23, srv767831273=27, srv121317519=3, srv748436250=25, srv59368786=20, srv265676843=17, srv1282671854=4, srv1429857587=7, srv766309425=26, srv610174545=22, srv1011322104=0, srv1597511048=8, srv1191188740=2, srv216258425=15, srv729114188=24, srv1336040224=5, srv599617946=21, srv1640461161=9, srv169888357=10, srv1399977768=6, srv1141993953=1, srv2061324522=13, srv253961459=16, srv43038436=19, srv1854356902=11, srv792588329=28, srv830331970=29, srv2022649293=12, srv272133628=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-16 22:55:20,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-16 22:55:20,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-16 22:55:20,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-16 22:55:20,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-16 22:55:20,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-16 22:55:20,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-16 22:55:20,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-16 22:55:20,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-16 22:55:20,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-16 22:55:20,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-16 22:55:20,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-16 22:55:20,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-16 22:55:20,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-16 22:55:20,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-16 22:55:20,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-16 22:55:20,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-16 22:55:20,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-16 22:55:20,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-16 22:55:20,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-16 22:55:20,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-16 22:55:20,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-16 22:55:20,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-16 22:55:20,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-16 22:55:20,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-16 22:55:20,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-16 22:55:20,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-16 22:55:20,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-16 22:55:20,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-16 22:55:20,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-16 22:55:20,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-16 22:55:20,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-02-16 22:55:20,694 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-16 22:55:20,694 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1011322104:90 , srv1141993953:90 , srv1191188740:90 , srv121317519:90 , srv1282671854:90 , srv1336040224:90 , srv1399977768:90 , srv1429857587:90 , srv1597511048:90 , srv1640461161:90 , srv169888357:90 , srv1854356902:90 , srv2022649293:90 , srv2061324522:90 , srv212500565:90 , srv216258425:90 , srv253961459:90 , srv265676843:90 , srv272133628:90 , srv43038436:90 , srv59368786:90 , srv599617946:90 , srv610174545:90 , srv639285115:90 , srv729114188:90 , srv748436250:90 , srv766309425:90 , srv767831273:90 , srv792588329:90 , srv830331970:90 }
2024-02-16 22:55:20,706 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=216 (was 271), ProcessCount=167 (was 167), AvailableMemoryMB=825 (was 1324)