2023-02-09 20:56:02,356 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-02-09 20:56:02,667 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-09 20:56:02,772 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-09 20:56:02,829 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-09 20:56:02,835 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-09 20:56:02,848 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=109, ProcessCount=168, AvailableMemoryMB=11314 2023-02-09 20:56:02,850 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-09 20:56:02,850 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-09 20:56:02,974 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1011754410:84 , srv1032901515:84 , srv108754540:84 , srv1133532161:84 , srv1135369458:84 , srv1139470149:84 , srv1198130962:84 , srv1238978027:84 , srv1248901967:84 , srv1273132569:84 , srv1275574106:84 , srv12800004:84 , srv1331931098:84 , srv1348485565:84 , srv1369659203:84 , srv1386651195:84 , srv1396083602:84 , srv140240816:84 , srv1432392741:84 , srv1437146033:84 , srv143958476:84 , srv1443466124:84 , srv1449534603:84 , srv1458703862:84 , srv1558705672:84 , srv1560378556:84 , srv1563947690:84 , srv1582277429:84 , srv1591809167:84 , srv159421071:84 , srv1638860738:84 , srv1670745778:84 , srv1688525345:84 , srv1696672136:84 , srv1706069689:84 , srv1716821226:84 , srv1724151337:84 , srv1746578783:84 , srv1764435724:84 , srv1779133902:84 , srv1781128148:84 , srv1806534266:84 , srv1822901157:84 , srv1833632253:84 , srv1837280168:84 , srv1926896261:84 , srv1944983744:84 , srv1945497026:84 , srv1948447898:84 , srv1977039332:84 , srv1991649509:84 , srv2040627750:84 , srv2050379766:84 , srv2063398274:84 , srv2088204433:84 , srv2106357305:84 , srv2107537787:84 , srv2115580922:84 , srv2137182738:84 , srv306436941:84 , srv342211078:84 , srv359379696:84 , srv384064707:84 , srv43148850:84 , srv437802099:84 , srv508151738:84 , srv540733152:84 , srv542732284:84 , srv562065845:84 , srv565336939:84 , srv574813628:84 , srv576148991:84 , srv583589246:84 , srv584165283:84 , srv587795278:84 , srv62409602:84 , srv676750122:84 , srv6909476:84 , srv697161722:84 , srv699689301:84 , srv759158939:84 , srv768164256:84 , srv791727720:84 , srv792743265:84 , srv807137969:84 , srv807761083:84 , srv807884969:84 , srv813862102:84 , srv815551203:84 , srv818466845:84 , srv826642930:84 , srv83481427:84 , srv837629247:84 , srv839662882:84 , srv873301559:84 , srv904591864:84 , srv918530928:84 , srv97826298:84 , srv995786506:84 , srv1417622643:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-02-09 20:56:02,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-09 20:56:02,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv12800004=11, srv1139470149=5, srv1696672136=34, srv1273132569=9, srv1432392741=19, srv1945497026=48, srv2050379766=53, srv807884969=87, srv1133532161=3, srv576148991=72, srv768164256=82, srv1563947690=27, srv2137182738=59, srv826642930=91, srv1638860738=31, srv1458703862=24, srv2063398274=54, srv676750122=77, srv792743265=84, srv542732284=68, srv143958476=21, srv1706069689=35, srv1135369458=4, srv1238978027=7, srv359379696=62, srv1582277429=28, srv699689301=80, srv995786506=99, srv1560378556=26, srv583589246=73, srv918530928=97, srv1746578783=38, srv584165283=74, srv839662882=94, srv1032901515=1, srv813862102=88, srv1198130962=6, srv437802099=65, srv837629247=93, srv1948447898=49, srv1591809167=29, srv1386651195=15, srv2040627750=52, srv43148850=64, srv562065845=69, srv2107537787=57, srv815551203=89, srv6909476=78, srv1449534603=23, srv159421071=30, srv1558705672=25, srv697161722=79, srv807137969=85, srv587795278=75, srv1779133902=40, srv508151738=66, srv904591864=96, srv97826298=98, srv2106357305=56, srv1443466124=22, srv1417622643=18, srv1781128148=41, srv1926896261=46, srv540733152=67, srv1991649509=51, srv565336939=70, srv1822901157=43, srv791727720=83, srv2088204433=55, srv306436941=60, srv807761083=86, srv1837280168=45, srv1248901967=8, srv574813628=71, srv108754540=2, srv1396083602=16, srv759158939=81, srv1011754410=0, srv140240816=17, srv1437146033=20, srv1806534266=42, srv384064707=63, srv1724151337=37, srv1688525345=33, srv1716821226=36, srv1348485565=13, srv873301559=95, srv83481427=92, srv62409602=76, srv1331931098=12, srv342211078=61, srv1670745778=32, srv1977039332=50, srv818466845=90, srv1275574106=10, srv1764435724=39, srv2115580922=58, srv1369659203=14, srv1833632253=44, srv1944983744=47} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-09 20:56:03,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 20:56:03,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-09 20:56:03,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-09 20:56:03,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-09 20:56:03,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-09 20:56:03,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-09 20:56:03,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-09 20:56:03,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-09 20:56:03,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-09 20:56:03,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-09 20:56:03,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-09 20:56:03,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-09 20:56:03,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-09 20:56:03,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-09 20:56:03,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-09 20:56:03,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-09 20:56:03,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-09 20:56:03,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-09 20:56:03,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-09 20:56:03,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-09 20:56:03,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-09 20:56:03,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-09 20:56:03,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-09 20:56:03,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-09 20:56:03,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-09 20:56:03,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-09 20:56:03,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-09 20:56:03,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-09 20:56:03,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-09 20:56:03,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-09 20:56:03,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-09 20:56:03,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-09 20:56:03,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-09 20:56:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-09 20:56:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-09 20:56:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-09 20:56:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-09 20:56:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-09 20:56:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-09 20:56:03,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-09 20:56:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-09 20:56:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-09 20:56:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-09 20:56:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-09 20:56:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-09 20:56:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-09 20:56:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-09 20:56:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-09 20:56:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-09 20:56:03,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-09 20:56:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-09 20:56:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-09 20:56:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-09 20:56:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-09 20:56:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-09 20:56:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-09 20:56:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-09 20:56:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-09 20:56:03,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-09 20:56:03,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-09 20:56:03,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-09 20:56:03,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-09 20:56:03,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-09 20:56:03,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-09 20:56:03,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-09 20:56:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-09 20:56:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-09 20:56:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-09 20:56:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-09 20:56:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-09 20:56:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-09 20:56:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-09 20:56:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-09 20:56:03,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-09 20:56:03,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-09 20:56:03,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-09 20:56:03,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-09 20:56:03,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-09 20:56:03,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-09 20:56:03,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-09 20:56:03,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-09 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-09 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-09 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-09 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-09 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-09 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-09 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-09 20:56:03,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-09 20:56:03,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-09 20:56:03,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-09 20:56:03,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-09 20:56:03,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-09 20:56:03,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-09 20:56:03,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-09 20:56:03,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-09 20:56:03,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-09 20:56:03,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-09 20:56:03,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-09 20:56:03,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-09 20:56:03,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-09 20:56:03,402 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-09 20:56:03,402 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-09 20:56:03,403 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-09 20:56:04,026 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 900 ms to try 5000 different iterations. Found a solution that moves 4662 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1834726295314606. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04151063189316198, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05417683147307899, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.518, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04151063189316198, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17875, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24516666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-09 20:56:04,046 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1011754410:82 , srv1348485565:82 , srv1449534603:82 , srv1560378556:82 , srv807884969:82 , srv815551203:82 , srv83481427:82 , srv1275574106:83 , srv1779133902:83 , srv1837280168:83 , srv1133532161:84 , srv1139470149:84 , srv1432392741:84 , srv1443466124:84 , srv1582277429:84 , srv1591809167:84 , srv2063398274:84 , srv2088204433:84 , srv359379696:84 , srv43148850:84 , srv697161722:84 , srv699689301:84 , srv873301559:84 , srv904591864:84 , srv97826298:84 , srv108754540:85 , srv1273132569:85 , srv1396083602:85 , srv143958476:85 , srv1558705672:85 , srv159421071:85 , srv1716821226:85 , srv1822901157:85 , srv1991649509:85 , srv542732284:85 , srv562065845:85 , srv62409602:85 , srv676750122:85 , srv792743265:85 , srv837629247:85 , srv1135369458:86 , srv1238978027:86 , srv140240816:86 , srv1638860738:86 , srv1724151337:86 , srv1746578783:86 , srv1945497026:86 , srv2040627750:86 , srv2050379766:86 , srv574813628:86 , srv759158939:86 , srv791727720:86 , srv807761083:86 , srv826642930:86 , srv918530928:86 , srv1458703862:87 , srv1563947690:87 , srv1688525345:87 , srv1764435724:87 , srv1833632253:87 , srv1926896261:87 , srv1944983744:87 , srv2107537787:87 , srv306436941:87 , srv342211078:87 , srv565336939:87 , srv583589246:87 , srv587795278:87 , srv6909476:87 , srv839662882:87 , srv1248901967:88 , srv1386651195:88 , srv1670745778:88 , srv1696672136:88 , srv1781128148:88 , srv1806534266:88 , srv1948447898:88 , srv1977039332:88 , srv2115580922:88 , srv508151738:88 , srv576148991:88 , srv584165283:88 , srv768164256:88 , srv813862102:88 , srv1032901515:89 , srv12800004:89 , srv1437146033:89 , srv437802099:89 , srv807137969:89 , srv995786506:89 , srv1331931098:90 , srv1706069689:90 , srv2106357305:90 , srv2137182738:90 , srv384064707:90 , srv1198130962:91 , srv1369659203:91 , srv540733152:92 , srv818466845:98 , srv1417622643:459 } 2023-02-09 20:56:04,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-09 20:56:04,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv12800004=11, srv1139470149=5, srv1696672136=34, srv1273132569=9, srv1432392741=19, srv1945497026=48, srv2050379766=53, srv807884969=87, srv1133532161=3, srv576148991=72, srv768164256=82, srv1563947690=27, srv2137182738=59, srv826642930=91, srv1638860738=31, srv1458703862=24, srv2063398274=54, srv676750122=77, srv792743265=84, srv542732284=68, srv143958476=21, srv1706069689=35, srv1135369458=4, srv1238978027=7, srv359379696=62, srv1582277429=28, srv699689301=80, srv995786506=99, srv1560378556=26, srv583589246=73, srv918530928=97, srv1746578783=38, srv584165283=74, srv839662882=94, srv1032901515=1, srv813862102=88, srv1198130962=6, srv437802099=65, srv837629247=93, srv1948447898=49, srv1591809167=29, srv1386651195=15, srv2040627750=52, srv43148850=64, srv562065845=69, srv2107537787=57, srv815551203=89, srv6909476=78, srv1449534603=23, srv159421071=30, srv1558705672=25, srv697161722=79, srv807137969=85, srv587795278=75, srv1779133902=40, srv508151738=66, srv904591864=96, srv97826298=98, srv2106357305=56, srv1443466124=22, srv1417622643=18, srv1781128148=41, srv1926896261=46, srv540733152=67, srv1991649509=51, srv565336939=70, srv1822901157=43, srv791727720=83, srv2088204433=55, srv306436941=60, srv807761083=86, srv1837280168=45, srv1248901967=8, srv574813628=71, srv108754540=2, srv1396083602=16, srv759158939=81, srv1011754410=0, srv140240816=17, srv1437146033=20, srv1806534266=42, srv384064707=63, srv1724151337=37, srv1688525345=33, srv1716821226=36, srv1348485565=13, srv873301559=95, srv83481427=92, srv62409602=76, srv1331931098=12, srv342211078=61, srv1670745778=32, srv1977039332=50, srv818466845=90, srv1275574106=10, srv1764435724=39, srv2115580922=58, srv1369659203=14, srv1833632253=44, srv1944983744=47} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-09 20:56:04,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 20:56:04,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-09 20:56:04,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-09 20:56:04,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-09 20:56:04,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-09 20:56:04,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-09 20:56:04,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-09 20:56:04,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-09 20:56:04,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-09 20:56:04,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-09 20:56:04,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-09 20:56:04,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-09 20:56:04,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-09 20:56:04,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-09 20:56:04,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-09 20:56:04,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-09 20:56:04,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-09 20:56:04,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-09 20:56:04,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-09 20:56:04,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-09 20:56:04,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-09 20:56:04,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-09 20:56:04,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-09 20:56:04,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-09 20:56:04,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-09 20:56:04,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-09 20:56:04,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-09 20:56:04,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-09 20:56:04,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-09 20:56:04,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-09 20:56:04,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-09 20:56:04,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-09 20:56:04,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-09 20:56:04,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-09 20:56:04,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-09 20:56:04,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-09 20:56:04,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-09 20:56:04,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-09 20:56:04,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-09 20:56:04,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-09 20:56:04,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-09 20:56:04,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-09 20:56:04,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-09 20:56:04,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-09 20:56:04,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-09 20:56:04,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-09 20:56:04,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-09 20:56:04,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-09 20:56:04,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-09 20:56:04,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-09 20:56:04,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-09 20:56:04,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-09 20:56:04,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-09 20:56:04,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-09 20:56:04,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-09 20:56:04,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-09 20:56:04,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-09 20:56:04,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-09 20:56:04,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-09 20:56:04,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-09 20:56:04,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-09 20:56:04,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-09 20:56:04,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-09 20:56:04,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-09 20:56:04,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-09 20:56:04,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-09 20:56:04,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-09 20:56:04,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-09 20:56:04,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-09 20:56:04,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-09 20:56:04,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-09 20:56:04,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-09 20:56:04,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-09 20:56:04,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-09 20:56:04,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-09 20:56:04,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-09 20:56:04,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-09 20:56:04,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-09 20:56:04,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-09 20:56:04,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-09 20:56:04,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-09 20:56:04,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-09 20:56:04,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-09 20:56:04,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-09 20:56:04,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-09 20:56:04,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-09 20:56:04,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-09 20:56:04,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-09 20:56:04,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-09 20:56:04,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-09 20:56:04,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-09 20:56:04,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-09 20:56:04,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-09 20:56:04,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-09 20:56:04,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-09 20:56:04,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-09 20:56:04,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-09 20:56:04,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-09 20:56:04,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-09 20:56:04,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-09 20:56:04,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-09 20:56:04,227 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04151063189316198, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05417683147307899, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04151063189316198, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17875, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24516666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-09 20:56:04,227 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-09 20:56:04,227 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18343998550974594, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04151063189316198, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05417683147307899, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04151063189316198, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17875, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24516666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-09 20:56:04,508 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 404 ms to try 5000 different iterations. Found a solution that moves 2584 regions; Going from a computed imbalance of 0.18343998550974594 to a new imbalance of 0.01594346501345823. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02353263183734808); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03603838831161523, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2871111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02353263183734808); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04633333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-09 20:56:04,518 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv818466845:75 , srv1198130962:77 , srv1369659203:79 , srv2137182738:80 , srv437802099:80 , srv1706069689:81 , srv584165283:81 , srv143958476:82 , srv2106357305:82 , srv2115580922:82 , srv508151738:82 , srv807137969:82 , srv12800004:83 , srv1670745778:83 , srv1696672136:83 , srv1781128148:83 , srv1806534266:83 , srv2107537787:83 , srv1032901515:84 , srv1331931098:84 , srv1948447898:84 , srv1977039332:84 , srv342211078:84 , srv565336939:84 , srv576148991:84 , srv587795278:84 , srv6909476:84 , srv768164256:84 , srv813862102:84 , srv826642930:84 , srv839662882:84 , srv995786506:84 , srv108754540:85 , srv1133532161:85 , srv1248901967:85 , srv1386651195:85 , srv140240816:85 , srv1437146033:85 , srv1458703862:85 , srv1563947690:85 , srv1764435724:85 , srv1944983744:85 , srv1945497026:85 , srv2050379766:85 , srv43148850:85 , srv540733152:85 , srv792743265:85 , srv837629247:85 , srv1135369458:86 , srv1273132569:86 , srv1432392741:86 , srv1558705672:86 , srv159421071:86 , srv1724151337:86 , srv1746578783:86 , srv2040627750:86 , srv306436941:86 , srv759158939:86 , srv1396083602:87 , srv1582277429:87 , srv1591809167:87 , srv1638860738:87 , srv1822901157:87 , srv1833632253:87 , srv1926896261:87 , srv384064707:87 , srv542732284:87 , srv562065845:87 , srv583589246:87 , srv676750122:87 , srv699689301:87 , srv791727720:87 , srv807761083:87 , srv918530928:87 , srv1139470149:88 , srv1348485565:88 , srv1688525345:88 , srv1716821226:88 , srv1779133902:88 , srv1991649509:88 , srv62409602:88 , srv807884969:88 , srv97826298:88 , srv1238978027:89 , srv1275574106:89 , srv1443466124:89 , srv1449534603:89 , srv1560378556:89 , srv2088204433:89 , srv574813628:89 , srv697161722:89 , srv873301559:89 , srv1011754410:90 , srv359379696:90 , srv83481427:90 , srv904591864:90 , srv1837280168:91 , srv2063398274:91 , srv815551203:92 , srv1417622643:524 } 2023-02-09 20:56:04,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-09 20:56:04,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv12800004=11, srv1139470149=5, srv1696672136=34, srv1273132569=9, srv1432392741=19, srv1945497026=48, srv2050379766=53, srv807884969=87, srv1133532161=3, srv576148991=72, srv768164256=82, srv1563947690=27, srv2137182738=59, srv826642930=91, srv1638860738=31, srv1458703862=24, srv2063398274=54, srv676750122=77, srv792743265=84, srv542732284=68, srv143958476=21, srv1706069689=35, srv1135369458=4, srv1238978027=7, srv359379696=62, srv1582277429=28, srv699689301=80, srv995786506=99, srv1560378556=26, srv583589246=73, srv918530928=97, srv1746578783=38, srv584165283=74, srv839662882=94, srv1032901515=1, srv813862102=88, srv1198130962=6, srv437802099=65, srv837629247=93, srv1948447898=49, srv1591809167=29, srv1386651195=15, srv2040627750=52, srv43148850=64, srv562065845=69, srv2107537787=57, srv815551203=89, srv6909476=78, srv1449534603=23, srv159421071=30, srv1558705672=25, srv697161722=79, srv807137969=85, srv587795278=75, srv1779133902=40, srv508151738=66, srv904591864=96, srv97826298=98, srv2106357305=56, srv1443466124=22, srv1417622643=18, srv1781128148=41, srv1926896261=46, srv540733152=67, srv1991649509=51, srv565336939=70, srv1822901157=43, srv791727720=83, srv2088204433=55, srv306436941=60, srv807761083=86, srv1837280168=45, srv1248901967=8, srv574813628=71, srv108754540=2, srv1396083602=16, srv759158939=81, srv1011754410=0, srv140240816=17, srv1437146033=20, srv1806534266=42, srv384064707=63, srv1724151337=37, srv1688525345=33, srv1716821226=36, srv1348485565=13, srv873301559=95, srv83481427=92, srv62409602=76, srv1331931098=12, srv342211078=61, srv1670745778=32, srv1977039332=50, srv818466845=90, srv1275574106=10, srv1764435724=39, srv2115580922=58, srv1369659203=14, srv1833632253=44, srv1944983744=47} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-09 20:56:04,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 20:56:04,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-09 20:56:04,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-09 20:56:04,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-09 20:56:04,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-09 20:56:04,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-09 20:56:04,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-09 20:56:04,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-09 20:56:04,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-09 20:56:04,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-09 20:56:04,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-09 20:56:04,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-09 20:56:04,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-09 20:56:04,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-09 20:56:04,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-09 20:56:04,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-09 20:56:04,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-09 20:56:04,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-09 20:56:04,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-09 20:56:04,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-09 20:56:04,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-09 20:56:04,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-09 20:56:04,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-09 20:56:04,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-09 20:56:04,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-09 20:56:04,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-09 20:56:04,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-09 20:56:04,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-09 20:56:04,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-09 20:56:04,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-09 20:56:04,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-09 20:56:04,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-09 20:56:04,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-09 20:56:04,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-09 20:56:04,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-09 20:56:04,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-09 20:56:04,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-09 20:56:04,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-09 20:56:04,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-09 20:56:04,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-09 20:56:04,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-09 20:56:04,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-09 20:56:04,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-09 20:56:04,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-09 20:56:04,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-09 20:56:04,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-09 20:56:04,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-09 20:56:04,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-09 20:56:04,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-09 20:56:04,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-09 20:56:04,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-09 20:56:04,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-09 20:56:04,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-09 20:56:04,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-09 20:56:04,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-09 20:56:04,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-09 20:56:04,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-09 20:56:04,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-09 20:56:04,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-09 20:56:04,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-09 20:56:04,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-09 20:56:04,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-09 20:56:04,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-09 20:56:04,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-09 20:56:04,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-09 20:56:04,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-09 20:56:04,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-09 20:56:04,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-09 20:56:04,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-09 20:56:04,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-09 20:56:04,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-09 20:56:04,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-09 20:56:04,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-09 20:56:04,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-09 20:56:04,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-09 20:56:04,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-09 20:56:04,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-09 20:56:04,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-09 20:56:04,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-09 20:56:04,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-09 20:56:04,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-09 20:56:04,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-09 20:56:04,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-09 20:56:04,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-09 20:56:04,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-09 20:56:04,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-09 20:56:04,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-09 20:56:04,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-09 20:56:04,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-09 20:56:04,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-09 20:56:04,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-09 20:56:04,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-09 20:56:04,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-09 20:56:04,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-09 20:56:04,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-09 20:56:04,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-09 20:56:04,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-09 20:56:04,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-09 20:56:04,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-09 20:56:04,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-09 20:56:04,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-09 20:56:04,658 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02353263183734808); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03603838831161523, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02353263183734808); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04633333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-09 20:56:04,658 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-09 20:56:04,658 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01592537145873693, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02353263183734808); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03603838831161523, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02353263183734808); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04633333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-09 20:56:04,949 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 386 ms to try 5000 different iterations. Found a solution that moves 1065 regions; Going from a computed imbalance of 0.01592537145873693 to a new imbalance of 5.411736191887097E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0063288916631981705); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008461970117626566); MoveCostFunction : (multiplier=7.0, imbalance=0.11833333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0063288916631981705); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-09 20:56:04,951 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv384064707:80 , srv540733152:81 , srv1238978027:82 , srv1833632253:82 , srv1688525345:83 , srv2063398274:83 , srv574813628:83 , srv1032901515:84 , srv1331931098:84 , srv1437146033:84 , srv1716821226:84 , srv1837280168:84 , srv1926896261:84 , srv1991649509:84 , srv306436941:84 , srv62409602:84 , srv676750122:84 , srv791727720:84 , srv807761083:84 , srv815551203:84 , srv818466845:84 , srv873301559:84 , srv904591864:84 , srv97826298:84 , srv995786506:84 , srv1011754410:85 , srv1135369458:85 , srv1386651195:85 , srv1443466124:85 , srv1458703862:85 , srv1638860738:85 , srv1696672136:85 , srv1746578783:85 , srv1764435724:85 , srv1944983744:85 , srv2040627750:85 , srv2088204433:85 , srv359379696:85 , srv542732284:85 , srv576148991:85 , srv583589246:85 , srv697161722:85 , srv768164256:85 , srv83481427:85 , srv918530928:85 , srv108754540:86 , srv1139470149:86 , srv1248901967:86 , srv1273132569:86 , srv1275574106:86 , srv12800004:86 , srv1396083602:86 , srv1558705672:86 , srv1560378556:86 , srv1563947690:86 , srv1591809167:86 , srv159421071:86 , srv1706069689:86 , srv1724151337:86 , srv1779133902:86 , srv1822901157:86 , srv1945497026:86 , srv1948447898:86 , srv1977039332:86 , srv2050379766:86 , srv2137182738:86 , srv342211078:86 , srv562065845:86 , srv587795278:86 , srv699689301:86 , srv759158939:86 , srv807137969:86 , srv813862102:86 , srv839662882:86 , srv1348485565:87 , srv1369659203:87 , srv140240816:87 , srv1432392741:87 , srv1449534603:87 , srv1582277429:87 , srv1670745778:87 , srv1781128148:87 , srv1806534266:87 , srv2106357305:87 , srv2115580922:87 , srv508151738:87 , srv565336939:87 , srv6909476:87 , srv792743265:87 , srv807884969:87 , srv826642930:87 , srv1133532161:88 , srv2107537787:88 , srv43148850:88 , srv837629247:88 , srv1198130962:89 , srv437802099:89 , srv584165283:89 , srv143958476:91 , srv1417622643:531 } 2023-02-09 20:56:04,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-09 20:56:04,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv12800004=11, srv1139470149=5, srv1696672136=34, srv1273132569=9, srv1432392741=19, srv1945497026=48, srv2050379766=53, srv807884969=87, srv1133532161=3, srv576148991=72, srv768164256=82, srv1563947690=27, srv2137182738=59, srv826642930=91, srv1638860738=31, srv1458703862=24, srv2063398274=54, srv676750122=77, srv792743265=84, srv542732284=68, srv143958476=21, srv1706069689=35, srv1135369458=4, srv1238978027=7, srv359379696=62, srv1582277429=28, srv699689301=80, srv995786506=99, srv1560378556=26, srv583589246=73, srv918530928=97, srv1746578783=38, srv584165283=74, srv839662882=94, srv1032901515=1, srv813862102=88, srv1198130962=6, srv437802099=65, srv837629247=93, srv1948447898=49, srv1591809167=29, srv1386651195=15, srv2040627750=52, srv43148850=64, srv562065845=69, srv2107537787=57, srv815551203=89, srv6909476=78, srv1449534603=23, srv159421071=30, srv1558705672=25, srv697161722=79, srv807137969=85, srv587795278=75, srv1779133902=40, srv508151738=66, srv904591864=96, srv97826298=98, srv2106357305=56, srv1443466124=22, srv1417622643=18, srv1781128148=41, srv1926896261=46, srv540733152=67, srv1991649509=51, srv565336939=70, srv1822901157=43, srv791727720=83, srv2088204433=55, srv306436941=60, srv807761083=86, srv1837280168=45, srv1248901967=8, srv574813628=71, srv108754540=2, srv1396083602=16, srv759158939=81, srv1011754410=0, srv140240816=17, srv1437146033=20, srv1806534266=42, srv384064707=63, srv1724151337=37, srv1688525345=33, srv1716821226=36, srv1348485565=13, srv873301559=95, srv83481427=92, srv62409602=76, srv1331931098=12, srv342211078=61, srv1670745778=32, srv1977039332=50, srv818466845=90, srv1275574106=10, srv1764435724=39, srv2115580922=58, srv1369659203=14, srv1833632253=44, srv1944983744=47} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-09 20:56:04,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 20:56:04,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-09 20:56:04,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-09 20:56:04,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-09 20:56:04,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-09 20:56:04,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-09 20:56:04,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-09 20:56:04,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-09 20:56:04,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-09 20:56:04,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-09 20:56:04,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-09 20:56:04,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-09 20:56:04,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-09 20:56:04,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-09 20:56:04,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-09 20:56:04,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-09 20:56:04,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-09 20:56:04,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-09 20:56:04,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-09 20:56:04,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-09 20:56:04,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-09 20:56:04,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-09 20:56:04,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-09 20:56:04,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-09 20:56:04,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-09 20:56:04,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-09 20:56:04,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-09 20:56:04,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-09 20:56:04,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-09 20:56:04,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-09 20:56:04,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-09 20:56:04,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-09 20:56:04,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-09 20:56:04,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-09 20:56:04,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-09 20:56:04,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-09 20:56:04,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-09 20:56:04,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-09 20:56:04,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-09 20:56:04,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-09 20:56:04,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-09 20:56:04,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-09 20:56:04,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-09 20:56:04,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-09 20:56:04,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-09 20:56:04,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-09 20:56:04,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-09 20:56:04,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-09 20:56:04,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-09 20:56:04,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-09 20:56:04,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-09 20:56:04,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-09 20:56:04,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-09 20:56:04,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-09 20:56:04,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-09 20:56:04,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-09 20:56:04,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-09 20:56:04,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-09 20:56:04,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-09 20:56:04,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-09 20:56:04,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-09 20:56:04,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-09 20:56:04,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-09 20:56:04,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-09 20:56:04,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-09 20:56:04,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-09 20:56:04,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-09 20:56:04,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-09 20:56:04,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-09 20:56:04,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-09 20:56:04,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-09 20:56:04,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-09 20:56:04,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-09 20:56:04,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-09 20:56:04,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-09 20:56:04,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-09 20:56:04,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-09 20:56:04,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-09 20:56:04,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-09 20:56:04,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-09 20:56:04,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-09 20:56:04,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-09 20:56:04,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-09 20:56:04,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-09 20:56:04,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-09 20:56:04,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-09 20:56:04,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-09 20:56:04,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-09 20:56:04,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-09 20:56:04,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-09 20:56:04,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-09 20:56:04,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-09 20:56:04,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-09 20:56:04,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-09 20:56:04,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-09 20:56:04,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-09 20:56:04,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-09 20:56:04,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-09 20:56:04,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-09 20:56:04,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-09 20:56:04,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-09 20:56:05,088 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0063288916631981705); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008461970117626566); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0063288916631981705); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-09 20:56:05,088 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-09 20:56:05,088 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.337163298008677E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0063288916631981705); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008461970117626566); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0063288916631981705); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-09 20:56:05,347 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 355 ms to try 5000 different iterations. Found a solution that moves 255 regions; Going from a computed imbalance of 5.337163298008677E-4 to a new imbalance of 2.1667179911798534E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.028333333333333332, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-09 20:56:05,348 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv359379696:82 , srv2106357305:83 , srv583589246:83 , srv584165283:83 , srv6909476:83 , srv1139470149:84 , srv1248901967:84 , srv1275574106:84 , srv1331931098:84 , srv1386651195:84 , srv1396083602:84 , srv140240816:84 , srv1437146033:84 , srv143958476:84 , srv1443466124:84 , srv1563947690:84 , srv1582277429:84 , srv1638860738:84 , srv1688525345:84 , srv1724151337:84 , srv1781128148:84 , srv1806534266:84 , srv1822901157:84 , srv1837280168:84 , srv1926896261:84 , srv1948447898:84 , srv1977039332:84 , srv2063398274:84 , srv2088204433:84 , srv2107537787:84 , srv2115580922:84 , srv437802099:84 , srv540733152:84 , srv562065845:84 , srv565336939:84 , srv574813628:84 , srv576148991:84 , srv759158939:84 , srv813862102:84 , srv815551203:84 , srv818466845:84 , srv837629247:84 , srv904591864:84 , srv918530928:84 , srv995786506:84 , srv1011754410:85 , srv1032901515:85 , srv1198130962:85 , srv1238978027:85 , srv1273132569:85 , srv12800004:85 , srv1348485565:85 , srv1369659203:85 , srv1432392741:85 , srv1449534603:85 , srv1458703862:85 , srv1558705672:85 , srv1560378556:85 , srv1591809167:85 , srv159421071:85 , srv1670745778:85 , srv1716821226:85 , srv1746578783:85 , srv1764435724:85 , srv1779133902:85 , srv1944983744:85 , srv1945497026:85 , srv1991649509:85 , srv2040627750:85 , srv2050379766:85 , srv306436941:85 , srv342211078:85 , srv384064707:85 , srv43148850:85 , srv508151738:85 , srv542732284:85 , srv587795278:85 , srv62409602:85 , srv697161722:85 , srv699689301:85 , srv768164256:85 , srv791727720:85 , srv792743265:85 , srv807137969:85 , srv807761083:85 , srv807884969:85 , srv826642930:85 , srv83481427:85 , srv839662882:85 , srv873301559:85 , srv108754540:86 , srv1133532161:86 , srv1135369458:86 , srv1696672136:86 , srv1706069689:86 , srv1833632253:86 , srv2137182738:86 , srv676750122:86 , srv97826298:86 , srv1417622643:627 } 2023-02-09 20:56:05,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-09 20:56:05,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv12800004=11, srv1139470149=5, srv1696672136=34, srv1273132569=9, srv1432392741=19, srv1945497026=48, srv2050379766=53, srv807884969=87, srv1133532161=3, srv576148991=72, srv768164256=82, srv1563947690=27, srv2137182738=59, srv826642930=91, srv1638860738=31, srv1458703862=24, srv2063398274=54, srv676750122=77, srv792743265=84, srv542732284=68, srv143958476=21, srv1706069689=35, srv1135369458=4, srv1238978027=7, srv359379696=62, srv1582277429=28, srv699689301=80, srv995786506=99, srv1560378556=26, srv583589246=73, srv918530928=97, srv1746578783=38, srv584165283=74, srv839662882=94, srv1032901515=1, srv813862102=88, srv1198130962=6, srv437802099=65, srv837629247=93, srv1948447898=49, srv1591809167=29, srv1386651195=15, srv2040627750=52, srv43148850=64, srv562065845=69, srv2107537787=57, srv815551203=89, srv6909476=78, srv1449534603=23, srv159421071=30, srv1558705672=25, srv697161722=79, srv807137969=85, srv587795278=75, srv1779133902=40, srv508151738=66, srv904591864=96, srv97826298=98, srv2106357305=56, srv1443466124=22, srv1417622643=18, srv1781128148=41, srv1926896261=46, srv540733152=67, srv1991649509=51, srv565336939=70, srv1822901157=43, srv791727720=83, srv2088204433=55, srv306436941=60, srv807761083=86, srv1837280168=45, srv1248901967=8, srv574813628=71, srv108754540=2, srv1396083602=16, srv759158939=81, srv1011754410=0, srv140240816=17, srv1437146033=20, srv1806534266=42, srv384064707=63, srv1724151337=37, srv1688525345=33, srv1716821226=36, srv1348485565=13, srv873301559=95, srv83481427=92, srv62409602=76, srv1331931098=12, srv342211078=61, srv1670745778=32, srv1977039332=50, srv818466845=90, srv1275574106=10, srv1764435724=39, srv2115580922=58, srv1369659203=14, srv1833632253=44, srv1944983744=47} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-09 20:56:05,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 20:56:05,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-09 20:56:05,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-09 20:56:05,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-09 20:56:05,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-09 20:56:05,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-09 20:56:05,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-09 20:56:05,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-09 20:56:05,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-09 20:56:05,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-09 20:56:05,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-09 20:56:05,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-09 20:56:05,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-09 20:56:05,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-09 20:56:05,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-09 20:56:05,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-09 20:56:05,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-09 20:56:05,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-09 20:56:05,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-09 20:56:05,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-09 20:56:05,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-09 20:56:05,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-09 20:56:05,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-09 20:56:05,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-09 20:56:05,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-09 20:56:05,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-09 20:56:05,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-09 20:56:05,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-09 20:56:05,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-09 20:56:05,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-09 20:56:05,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-09 20:56:05,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-09 20:56:05,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-09 20:56:05,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-09 20:56:05,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-09 20:56:05,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-09 20:56:05,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-09 20:56:05,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-09 20:56:05,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-09 20:56:05,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-09 20:56:05,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-09 20:56:05,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-09 20:56:05,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-09 20:56:05,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-09 20:56:05,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-09 20:56:05,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-09 20:56:05,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-09 20:56:05,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-09 20:56:05,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-09 20:56:05,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-09 20:56:05,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-09 20:56:05,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-09 20:56:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-09 20:56:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-09 20:56:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-09 20:56:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-09 20:56:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-09 20:56:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-09 20:56:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-09 20:56:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-09 20:56:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-09 20:56:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-09 20:56:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-09 20:56:05,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-09 20:56:05,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-09 20:56:05,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-09 20:56:05,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-09 20:56:05,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-09 20:56:05,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-09 20:56:05,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-09 20:56:05,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-09 20:56:05,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-09 20:56:05,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-09 20:56:05,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-09 20:56:05,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-09 20:56:05,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-09 20:56:05,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-09 20:56:05,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-09 20:56:05,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-09 20:56:05,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-09 20:56:05,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-09 20:56:05,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-09 20:56:05,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-09 20:56:05,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-09 20:56:05,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-09 20:56:05,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-09 20:56:05,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-09 20:56:05,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-09 20:56:05,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-09 20:56:05,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-09 20:56:05,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-09 20:56:05,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-09 20:56:05,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-09 20:56:05,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-09 20:56:05,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-09 20:56:05,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-09 20:56:05,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-09 20:56:05,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-09 20:56:05,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-09 20:56:05,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-09 20:56:05,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-09 20:56:05,501 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.9881631748512408E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-09 20:56:05,501 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv359379696:82 , srv2106357305:83 , srv583589246:83 , srv584165283:83 , srv6909476:83 , srv1139470149:84 , srv1248901967:84 , srv1275574106:84 , srv1331931098:84 , srv1386651195:84 , srv1396083602:84 , srv140240816:84 , srv1437146033:84 , srv143958476:84 , srv1443466124:84 , srv1563947690:84 , srv1582277429:84 , srv1638860738:84 , srv1688525345:84 , srv1724151337:84 , srv1781128148:84 , srv1806534266:84 , srv1822901157:84 , srv1837280168:84 , srv1926896261:84 , srv1948447898:84 , srv1977039332:84 , srv2063398274:84 , srv2088204433:84 , srv2107537787:84 , srv2115580922:84 , srv437802099:84 , srv540733152:84 , srv562065845:84 , srv565336939:84 , srv574813628:84 , srv576148991:84 , srv759158939:84 , srv813862102:84 , srv815551203:84 , srv818466845:84 , srv837629247:84 , srv904591864:84 , srv918530928:84 , srv995786506:84 , srv1011754410:85 , srv1032901515:85 , srv1198130962:85 , srv1238978027:85 , srv1273132569:85 , srv12800004:85 , srv1348485565:85 , srv1369659203:85 , srv1432392741:85 , srv1449534603:85 , srv1458703862:85 , srv1558705672:85 , srv1560378556:85 , srv1591809167:85 , srv159421071:85 , srv1670745778:85 , srv1716821226:85 , srv1746578783:85 , srv1764435724:85 , srv1779133902:85 , srv1944983744:85 , srv1945497026:85 , srv1991649509:85 , srv2040627750:85 , srv2050379766:85 , srv306436941:85 , srv342211078:85 , srv384064707:85 , srv43148850:85 , srv508151738:85 , srv542732284:85 , srv587795278:85 , srv62409602:85 , srv697161722:85 , srv699689301:85 , srv768164256:85 , srv791727720:85 , srv792743265:85 , srv807137969:85 , srv807761083:85 , srv807884969:85 , srv826642930:85 , srv83481427:85 , srv839662882:85 , srv873301559:85 , srv108754540:86 , srv1133532161:86 , srv1135369458:86 , srv1696672136:86 , srv1706069689:86 , srv1833632253:86 , srv2137182738:86 , srv676750122:86 , srv97826298:86 , srv1417622643:627 } 2023-02-09 20:56:05,535 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=140 (was 109) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=10821 (was 11314) 2023-02-09 20:56:05,542 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=140, ProcessCount=168, AvailableMemoryMB=10821 2023-02-09 20:56:05,543 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-09 20:56:05,543 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-02-09 20:56:05,555 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1095075468:84 , srv1203119006:84 , srv1219278212:84 , srv1338663353:84 , srv1359332560:84 , srv1411367282:84 , srv1469414:84 , srv1569579116:84 , srv1600065660:84 , srv1875616458:84 , srv1933731124:84 , srv1949577401:84 , srv198748752:84 , srv1998147017:84 , srv205272268:84 , srv275776158:84 , srv300044374:84 , srv309227341:84 , srv34334058:84 , srv441732888:84 , srv508946555:84 , srv548699033:84 , srv602021441:84 , srv606711764:84 , srv759437033:84 , srv762466448:84 , srv822347121:84 , srv951407166:84 , srv969361912:84 , srv1187687509:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-02-09 20:56:05,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-09 20:56:05,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1095075468=0, srv1600065660=9, srv759437033=25, srv969361912=29, srv441732888=20, srv1569579116=8, srv275776158=16, srv1203119006=2, srv205272268=15, srv762466448=26, srv508946555=21, srv1933731124=11, srv1949577401=12, srv198748752=13, srv34334058=19, srv602021441=23, srv300044374=17, srv1998147017=14, srv1187687509=1, srv1411367282=6, srv951407166=28, srv309227341=18, srv1469414=7, srv606711764=24, srv1359332560=5, srv548699033=22, srv822347121=27, srv1875616458=10, srv1219278212=3, srv1338663353=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-09 20:56:05,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 20:56:05,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-09 20:56:05,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-09 20:56:05,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-09 20:56:05,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-09 20:56:05,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-09 20:56:05,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-09 20:56:05,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-09 20:56:05,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-09 20:56:05,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-09 20:56:05,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-09 20:56:05,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-09 20:56:05,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-09 20:56:05,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-09 20:56:05,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-09 20:56:05,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-09 20:56:05,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-09 20:56:05,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-09 20:56:05,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-09 20:56:05,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-09 20:56:05,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-09 20:56:05,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-09 20:56:05,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-09 20:56:05,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-09 20:56:05,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-09 20:56:05,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-09 20:56:05,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-09 20:56:05,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-09 20:56:05,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-09 20:56:05,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-09 20:56:05,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-09 20:56:05,591 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1485133499813223, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-09 20:56:05,591 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912661003980664, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1485133499813223, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-02-09 20:58:05,576 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6566847 different iterations. Found a solution that moves 1855 regions; Going from a computed imbalance of 0.9912661003980664 to a new imbalance of 4.329662539733032E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.687037037037037, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-09 20:58:05,582 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1095075468:90 , srv1187687509:90 , srv1203119006:90 , srv1219278212:90 , srv1338663353:90 , srv1359332560:90 , srv1411367282:90 , srv1469414:90 , srv1569579116:90 , srv1600065660:90 , srv1875616458:90 , srv1933731124:90 , srv1949577401:90 , srv198748752:90 , srv1998147017:90 , srv205272268:90 , srv275776158:90 , srv300044374:90 , srv309227341:90 , srv34334058:90 , srv441732888:90 , srv508946555:90 , srv548699033:90 , srv602021441:90 , srv606711764:90 , srv759437033:90 , srv762466448:90 , srv822347121:90 , srv951407166:90 , srv969361912:90 } 2023-02-09 20:58:05,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-09 20:58:05,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1095075468=0, srv1600065660=9, srv759437033=25, srv969361912=29, srv441732888=20, srv1569579116=8, srv275776158=16, srv1203119006=2, srv205272268=15, srv762466448=26, srv508946555=21, srv1933731124=11, srv1949577401=12, srv198748752=13, srv34334058=19, srv602021441=23, srv300044374=17, srv1998147017=14, srv1187687509=1, srv1411367282=6, srv951407166=28, srv309227341=18, srv1469414=7, srv606711764=24, srv1359332560=5, srv548699033=22, srv822347121=27, srv1875616458=10, srv1219278212=3, srv1338663353=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-09 20:58:05,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 20:58:05,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-09 20:58:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-09 20:58:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-09 20:58:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-09 20:58:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-09 20:58:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-09 20:58:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-09 20:58:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-09 20:58:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-09 20:58:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-09 20:58:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-09 20:58:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-09 20:58:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-09 20:58:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-09 20:58:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-09 20:58:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-09 20:58:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-09 20:58:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-09 20:58:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-09 20:58:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-09 20:58:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-09 20:58:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-09 20:58:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-09 20:58:05,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-09 20:58:05,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-09 20:58:05,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-09 20:58:05,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-09 20:58:05,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-09 20:58:05,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-09 20:58:05,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-09 20:58:05,599 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-09 20:58:05,599 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1095075468:90 , srv1187687509:90 , srv1203119006:90 , srv1219278212:90 , srv1338663353:90 , srv1359332560:90 , srv1411367282:90 , srv1469414:90 , srv1569579116:90 , srv1600065660:90 , srv1875616458:90 , srv1933731124:90 , srv1949577401:90 , srv198748752:90 , srv1998147017:90 , srv205272268:90 , srv275776158:90 , srv300044374:90 , srv309227341:90 , srv34334058:90 , srv441732888:90 , srv508946555:90 , srv548699033:90 , srv602021441:90 , srv606711764:90 , srv759437033:90 , srv762466448:90 , srv822347121:90 , srv951407166:90 , srv969361912:90 } 2023-02-09 20:58:05,615 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=199 (was 140) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=10645 (was 10821)