2023-04-01 03:59:44,513 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-04-01 03:59:44,786 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-01 03:59:44,888 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-01 03:59:44,965 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-01 03:59:44,974 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-01 03:59:44,991 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=476, ProcessCount=167, AvailableMemoryMB=9587 2023-04-01 03:59:44,993 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-01 03:59:44,993 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-01 03:59:45,126 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv102866622:84 , srv1036111184:84 , srv1054354858:84 , srv1058028508:84 , srv1119419323:84 , srv1125620273:84 , srv1128055035:84 , srv1130918861:84 , srv1158174829:84 , srv1170600797:84 , srv1232888242:84 , srv1233949179:84 , srv1237552190:84 , srv1261952605:84 , srv1268109957:84 , srv1271451079:84 , srv1278807079:84 , srv1340857729:84 , srv1342143617:84 , srv1348628925:84 , srv1399395053:84 , srv1417362131:84 , srv1456083664:84 , srv1458942815:84 , srv1491302864:84 , srv150676298:84 , srv1588076409:84 , srv1599343073:84 , srv1602896747:84 , srv1603396711:84 , srv1603750129:84 , srv1654100984:84 , srv1666972805:84 , srv1674392227:84 , srv1679060575:84 , srv1680987920:84 , srv1684968212:84 , srv1698572321:84 , srv170639145:84 , srv1738886434:84 , srv1766811880:84 , srv1808217702:84 , srv1841280974:84 , srv1845586061:84 , srv1889370679:84 , srv1889807714:84 , srv1890900503:84 , srv1907681838:84 , srv1908248180:84 , srv1915982440:84 , srv1921402618:84 , srv1946596909:84 , srv1957510887:84 , srv1961469328:84 , srv1975894660:84 , srv198719940:84 , srv1997010021:84 , srv2002289510:84 , srv201950313:84 , srv2028851517:84 , srv2045754798:84 , srv2047922043:84 , srv2049863221:84 , srv2072550966:84 , srv2074527133:84 , srv226969518:84 , srv255040105:84 , srv256651103:84 , srv262245399:84 , srv293055971:84 , srv29668983:84 , srv314392539:84 , srv355534485:84 , srv373662136:84 , srv387913170:84 , srv411345888:84 , srv434382004:84 , srv43636674:84 , srv445968766:84 , srv487295681:84 , srv54109376:84 , srv541664019:84 , srv609107286:84 , srv623621618:84 , srv625461079:84 , srv640331504:84 , srv641324213:84 , srv651722412:84 , srv661546627:84 , srv690836760:84 , srv706543699:84 , srv724291470:84 , srv750466578:84 , srv759517640:84 , srv820318764:84 , srv82788003:84 , srv844145936:84 , srv94320653:84 , srv977465688:84 , srv27639952:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-04-01 03:59:45,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-01 03:59:45,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv29668983=71, srv1271451079=15, srv27639952=69, srv314392539=72, srv1232888242=10, srv373662136=74, srv1128055035=6, srv94320653=98, srv355534485=73, srv750466578=93, srv1907681838=47, srv1679060575=34, srv1808217702=41, srv1603396711=29, srv690836760=90, srv1674392227=33, srv1890900503=46, srv2045754798=60, srv1961469328=53, srv1278807079=16, srv150676298=25, srv1119419323=4, srv2002289510=57, srv2049863221=62, srv170639145=38, srv1417362131=21, srv1456083664=22, srv387913170=75, srv1997010021=56, srv641324213=87, srv820318764=95, srv2047922043=61, srv487295681=80, srv1680987920=35, srv1666972805=32, srv1845586061=43, srv661546627=89, srv434382004=77, srv445968766=79, srv1125620273=5, srv2072550966=63, srv1130918861=7, srv1237552190=12, srv1588076409=26, srv256651103=67, srv293055971=70, srv1921402618=50, srv2028851517=59, srv1766811880=40, srv1170600797=9, srv1889370679=44, srv1975894660=54, srv226969518=65, srv1915982440=49, srv1058028508=3, srv255040105=66, srv1654100984=31, srv977465688=99, srv1054354858=2, srv411345888=76, srv1233949179=11, srv541664019=82, srv2074527133=64, srv1342143617=18, srv724291470=92, srv1261952605=13, srv1491302864=24, srv1889807714=45, srv1599343073=27, srv651722412=88, srv759517640=94, srv623621618=84, srv1036111184=1, srv1684968212=36, srv198719940=55, srv1348628925=19, srv201950313=58, srv1841280974=42, srv43636674=78, srv1946596909=51, srv1738886434=39, srv625461079=85, srv1602896747=28, srv844145936=97, srv1458942815=23, srv262245399=68, srv1908248180=48, srv54109376=81, srv1603750129=30, srv1957510887=52, srv640331504=86, srv1399395053=20, srv609107286=83, srv1158174829=8, srv706543699=91, srv1698572321=37, srv82788003=96, srv102866622=0, srv1340857729=17, srv1268109957=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-01 03:59:45,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-01 03:59:45,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-01 03:59:45,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-01 03:59:45,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-01 03:59:45,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-01 03:59:45,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-01 03:59:45,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-01 03:59:45,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-01 03:59:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-01 03:59:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-01 03:59:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-01 03:59:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-01 03:59:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-01 03:59:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-01 03:59:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-01 03:59:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-01 03:59:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-01 03:59:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-01 03:59:45,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-01 03:59:45,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-01 03:59:45,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-01 03:59:45,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-01 03:59:45,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-01 03:59:45,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-01 03:59:45,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-01 03:59:45,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-01 03:59:45,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-01 03:59:45,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-01 03:59:45,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-01 03:59:45,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-01 03:59:45,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-01 03:59:45,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-01 03:59:45,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-01 03:59:45,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-01 03:59:45,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-01 03:59:45,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-01 03:59:45,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-01 03:59:45,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-01 03:59:45,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-01 03:59:45,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-01 03:59:45,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-01 03:59:45,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-01 03:59:45,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-01 03:59:45,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-01 03:59:45,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-01 03:59:45,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-01 03:59:45,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-01 03:59:45,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-01 03:59:45,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-01 03:59:45,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-01 03:59:45,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-01 03:59:45,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-01 03:59:45,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-01 03:59:45,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-01 03:59:45,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-01 03:59:45,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-01 03:59:45,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-01 03:59:45,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-01 03:59:45,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-01 03:59:45,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-01 03:59:45,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-01 03:59:45,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-01 03:59:45,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-01 03:59:45,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-01 03:59:45,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-01 03:59:45,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-01 03:59:45,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-01 03:59:45,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-01 03:59:45,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-01 03:59:45,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-01 03:59:45,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-01 03:59:45,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-01 03:59:45,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-01 03:59:45,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-01 03:59:45,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-01 03:59:45,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-01 03:59:45,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-01 03:59:45,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-01 03:59:45,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-01 03:59:45,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-01 03:59:45,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-01 03:59:45,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-01 03:59:45,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-01 03:59:45,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-01 03:59:45,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-01 03:59:45,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-01 03:59:45,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-01 03:59:45,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-01 03:59:45,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-01 03:59:45,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-01 03:59:45,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-01 03:59:45,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-01 03:59:45,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-01 03:59:45,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-01 03:59:45,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-01 03:59:45,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-01 03:59:45,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-01 03:59:45,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-01 03:59:45,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-01 03:59:45,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-01 03:59:45,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-01 03:59:45,571 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-01 03:59:45,572 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-01 03:59:45,572 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-01 03:59:45,892 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 612 ms to try 5000 different iterations. Found a solution that moves 4695 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17923871648796058. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040920936057919556, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057509087751083654, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5216666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040920936057919556, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24383333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-01 03:59:45,904 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1130918861:81 , srv1908248180:82 , srv2045754798:82 , srv759517640:82 , srv1342143617:83 , srv1915982440:83 , srv724291470:83 , srv820318764:83 , srv1170600797:84 , srv1456083664:84 , srv1602896747:84 , srv170639145:84 , srv1766811880:84 , srv1841280974:84 , srv2002289510:84 , srv2072550966:84 , srv293055971:84 , srv373662136:84 , srv387913170:84 , srv625461079:84 , srv1125620273:85 , srv1261952605:85 , srv1271451079:85 , srv1278807079:85 , srv1599343073:85 , srv1603750129:85 , srv1957510887:85 , srv1975894660:85 , srv201950313:85 , srv2049863221:85 , srv226969518:85 , srv262245399:85 , srv355534485:85 , srv434382004:85 , srv541664019:85 , srv640331504:85 , srv641324213:85 , srv102866622:86 , srv1058028508:86 , srv1268109957:86 , srv1399395053:86 , srv1491302864:86 , srv1588076409:86 , srv1603396711:86 , srv1680987920:86 , srv1808217702:86 , srv1845586061:86 , srv1890900503:86 , srv1907681838:86 , srv255040105:86 , srv411345888:86 , srv690836760:86 , srv1036111184:87 , srv1119419323:87 , srv1232888242:87 , srv1458942815:87 , srv1666972805:87 , srv1889370679:87 , srv1946596909:87 , srv1997010021:87 , srv2028851517:87 , srv29668983:87 , srv314392539:87 , srv43636674:87 , srv651722412:87 , srv661546627:87 , srv844145936:87 , srv94320653:87 , srv1054354858:88 , srv1128055035:88 , srv1237552190:88 , srv1348628925:88 , srv1417362131:88 , srv1654100984:88 , srv1698572321:88 , srv1738886434:88 , srv1921402618:88 , srv1961469328:88 , srv198719940:88 , srv256651103:88 , srv445968766:88 , srv487295681:88 , srv54109376:88 , srv609107286:88 , srv1233949179:89 , srv1679060575:89 , srv2047922043:89 , srv623621618:89 , srv706543699:89 , srv750466578:89 , srv82788003:89 , srv977465688:89 , srv1158174829:90 , srv1340857729:90 , srv150676298:90 , srv1674392227:90 , srv1889807714:90 , srv1684968212:91 , srv2074527133:91 , srv27639952:454 } 2023-04-01 03:59:45,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-01 03:59:45,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv29668983=71, srv1271451079=15, srv27639952=69, srv314392539=72, srv1232888242=10, srv373662136=74, srv1128055035=6, srv94320653=98, srv355534485=73, srv750466578=93, srv1907681838=47, srv1679060575=34, srv1808217702=41, srv1603396711=29, srv690836760=90, srv1674392227=33, srv1890900503=46, srv2045754798=60, srv1961469328=53, srv1278807079=16, srv150676298=25, srv1119419323=4, srv2002289510=57, srv2049863221=62, srv170639145=38, srv1417362131=21, srv1456083664=22, srv387913170=75, srv1997010021=56, srv641324213=87, srv820318764=95, srv2047922043=61, srv487295681=80, srv1680987920=35, srv1666972805=32, srv1845586061=43, srv661546627=89, srv434382004=77, srv445968766=79, srv1125620273=5, srv2072550966=63, srv1130918861=7, srv1237552190=12, srv1588076409=26, srv256651103=67, srv293055971=70, srv1921402618=50, srv2028851517=59, srv1766811880=40, srv1170600797=9, srv1889370679=44, srv1975894660=54, srv226969518=65, srv1915982440=49, srv1058028508=3, srv255040105=66, srv1654100984=31, srv977465688=99, srv1054354858=2, srv411345888=76, srv1233949179=11, srv541664019=82, srv2074527133=64, srv1342143617=18, srv724291470=92, srv1261952605=13, srv1491302864=24, srv1889807714=45, srv1599343073=27, srv651722412=88, srv759517640=94, srv623621618=84, srv1036111184=1, srv1684968212=36, srv198719940=55, srv1348628925=19, srv201950313=58, srv1841280974=42, srv43636674=78, srv1946596909=51, srv1738886434=39, srv625461079=85, srv1602896747=28, srv844145936=97, srv1458942815=23, srv262245399=68, srv1908248180=48, srv54109376=81, srv1603750129=30, srv1957510887=52, srv640331504=86, srv1399395053=20, srv609107286=83, srv1158174829=8, srv706543699=91, srv1698572321=37, srv82788003=96, srv102866622=0, srv1340857729=17, srv1268109957=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-01 03:59:45,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-01 03:59:45,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-01 03:59:45,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-01 03:59:45,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-01 03:59:45,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-01 03:59:45,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-01 03:59:45,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-01 03:59:45,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-01 03:59:45,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-01 03:59:45,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-01 03:59:45,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-01 03:59:45,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-01 03:59:45,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-01 03:59:45,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-01 03:59:45,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-01 03:59:45,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-01 03:59:45,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-01 03:59:45,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-01 03:59:45,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-01 03:59:45,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-01 03:59:45,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-01 03:59:45,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-01 03:59:45,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-01 03:59:45,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-01 03:59:45,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-01 03:59:45,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-01 03:59:45,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-01 03:59:45,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-01 03:59:45,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-01 03:59:45,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-01 03:59:45,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-01 03:59:45,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-01 03:59:45,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-01 03:59:45,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-01 03:59:45,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-01 03:59:45,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-01 03:59:45,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-01 03:59:45,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-01 03:59:45,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-01 03:59:45,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-01 03:59:45,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-01 03:59:45,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-01 03:59:45,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-01 03:59:45,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-01 03:59:45,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-01 03:59:45,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-01 03:59:45,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-01 03:59:45,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-01 03:59:45,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-01 03:59:45,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-01 03:59:45,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-01 03:59:45,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-01 03:59:45,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-01 03:59:45,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-01 03:59:45,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-01 03:59:45,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-01 03:59:45,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-01 03:59:45,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-01 03:59:45,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-01 03:59:45,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-01 03:59:45,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-01 03:59:45,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-01 03:59:45,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-01 03:59:45,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-01 03:59:45,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-01 03:59:45,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-01 03:59:45,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-01 03:59:45,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-01 03:59:45,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-01 03:59:45,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-01 03:59:45,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-01 03:59:45,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-01 03:59:45,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-01 03:59:45,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-01 03:59:45,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-01 03:59:45,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-01 03:59:45,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-01 03:59:45,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-01 03:59:45,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-01 03:59:45,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-01 03:59:45,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-01 03:59:45,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-01 03:59:45,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-01 03:59:45,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-01 03:59:45,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-01 03:59:45,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-01 03:59:45,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-01 03:59:45,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-01 03:59:45,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-01 03:59:45,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-01 03:59:45,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-01 03:59:45,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-01 03:59:45,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-01 03:59:45,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-01 03:59:45,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-01 03:59:45,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-01 03:59:45,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-01 03:59:45,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-01 03:59:45,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-01 03:59:45,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-01 03:59:45,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-01 03:59:46,053 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040920936057919556, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057509087751083654, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040920936057919556, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24383333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-01 03:59:46,053 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-01 03:59:46,054 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17920584139530712, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040920936057919556, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057509087751083654, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040920936057919556, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24383333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-01 03:59:46,304 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 355 ms to try 5000 different iterations. Found a solution that moves 2666 regions; Going from a computed imbalance of 0.17920584139530712 to a new imbalance of 0.016393532272882903. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022523214112595932); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035308485005105784, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2962222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022523214112595932); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-01 03:59:46,312 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2074527133:79 , srv1340857729:81 , srv1679060575:81 , srv1738886434:81 , srv623621618:81 , srv82788003:81 , srv1158174829:82 , srv1417362131:82 , srv1654100984:82 , srv2047922043:82 , srv150676298:83 , srv1588076409:83 , srv1603396711:83 , srv1684968212:83 , srv1698572321:83 , srv1808217702:83 , srv1921402618:83 , srv1961469328:83 , srv201950313:83 , srv29668983:83 , srv445968766:83 , srv609107286:83 , srv1054354858:84 , srv1232888242:84 , srv1233949179:84 , srv1399395053:84 , srv1674392227:84 , srv1889807714:84 , srv2028851517:84 , srv256651103:84 , srv54109376:84 , srv651722412:84 , srv706543699:84 , srv1058028508:85 , srv1348628925:85 , srv1491302864:85 , srv1666972805:85 , srv1907681838:85 , srv1946596909:85 , srv198719940:85 , srv2072550966:85 , srv226969518:85 , srv314392539:85 , srv487295681:85 , srv750466578:85 , srv844145936:85 , srv94320653:85 , srv1119419323:86 , srv1125620273:86 , srv1268109957:86 , srv1456083664:86 , srv1458942815:86 , srv1680987920:86 , srv1915982440:86 , srv1997010021:86 , srv355534485:86 , srv43636674:86 , srv977465688:86 , srv102866622:87 , srv1036111184:87 , srv1128055035:87 , srv1237552190:87 , srv1261952605:87 , srv1271451079:87 , srv1599343073:87 , srv1841280974:87 , srv1845586061:87 , srv1890900503:87 , srv1975894660:87 , srv2002289510:87 , srv293055971:87 , srv387913170:87 , srv411345888:87 , srv541664019:87 , srv640331504:87 , srv661546627:87 , srv1170600797:88 , srv1342143617:88 , srv1766811880:88 , srv1957510887:88 , srv255040105:88 , srv262245399:88 , srv434382004:88 , srv625461079:88 , srv641324213:88 , srv690836760:88 , srv759517640:88 , srv820318764:88 , srv1278807079:89 , srv1602896747:89 , srv170639145:89 , srv1889370679:89 , srv373662136:89 , srv1130918861:90 , srv2049863221:90 , srv724291470:90 , srv2045754798:91 , srv1603750129:92 , srv1908248180:92 , srv27639952:520 } 2023-04-01 03:59:46,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-01 03:59:46,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv29668983=71, srv1271451079=15, srv27639952=69, srv314392539=72, srv1232888242=10, srv373662136=74, srv1128055035=6, srv94320653=98, srv355534485=73, srv750466578=93, srv1907681838=47, srv1679060575=34, srv1808217702=41, srv1603396711=29, srv690836760=90, srv1674392227=33, srv1890900503=46, srv2045754798=60, srv1961469328=53, srv1278807079=16, srv150676298=25, srv1119419323=4, srv2002289510=57, srv2049863221=62, srv170639145=38, srv1417362131=21, srv1456083664=22, srv387913170=75, srv1997010021=56, srv641324213=87, srv820318764=95, srv2047922043=61, srv487295681=80, srv1680987920=35, srv1666972805=32, srv1845586061=43, srv661546627=89, srv434382004=77, srv445968766=79, srv1125620273=5, srv2072550966=63, srv1130918861=7, srv1237552190=12, srv1588076409=26, srv256651103=67, srv293055971=70, srv1921402618=50, srv2028851517=59, srv1766811880=40, srv1170600797=9, srv1889370679=44, srv1975894660=54, srv226969518=65, srv1915982440=49, srv1058028508=3, srv255040105=66, srv1654100984=31, srv977465688=99, srv1054354858=2, srv411345888=76, srv1233949179=11, srv541664019=82, srv2074527133=64, srv1342143617=18, srv724291470=92, srv1261952605=13, srv1491302864=24, srv1889807714=45, srv1599343073=27, srv651722412=88, srv759517640=94, srv623621618=84, srv1036111184=1, srv1684968212=36, srv198719940=55, srv1348628925=19, srv201950313=58, srv1841280974=42, srv43636674=78, srv1946596909=51, srv1738886434=39, srv625461079=85, srv1602896747=28, srv844145936=97, srv1458942815=23, srv262245399=68, srv1908248180=48, srv54109376=81, srv1603750129=30, srv1957510887=52, srv640331504=86, srv1399395053=20, srv609107286=83, srv1158174829=8, srv706543699=91, srv1698572321=37, srv82788003=96, srv102866622=0, srv1340857729=17, srv1268109957=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-01 03:59:46,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-01 03:59:46,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-01 03:59:46,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-01 03:59:46,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-01 03:59:46,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-01 03:59:46,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-01 03:59:46,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-01 03:59:46,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-01 03:59:46,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-01 03:59:46,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-01 03:59:46,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-01 03:59:46,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-01 03:59:46,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-01 03:59:46,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-01 03:59:46,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-01 03:59:46,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-01 03:59:46,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-01 03:59:46,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-01 03:59:46,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-01 03:59:46,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-01 03:59:46,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-01 03:59:46,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-01 03:59:46,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-01 03:59:46,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-01 03:59:46,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-01 03:59:46,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-01 03:59:46,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-01 03:59:46,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-01 03:59:46,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-01 03:59:46,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-01 03:59:46,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-01 03:59:46,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-01 03:59:46,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-01 03:59:46,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-01 03:59:46,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-01 03:59:46,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-01 03:59:46,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-01 03:59:46,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-01 03:59:46,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-01 03:59:46,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-01 03:59:46,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-01 03:59:46,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-01 03:59:46,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-01 03:59:46,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-01 03:59:46,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-01 03:59:46,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-01 03:59:46,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-01 03:59:46,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-01 03:59:46,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-01 03:59:46,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-01 03:59:46,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-01 03:59:46,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-01 03:59:46,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-01 03:59:46,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-01 03:59:46,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-01 03:59:46,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-01 03:59:46,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-01 03:59:46,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-01 03:59:46,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-01 03:59:46,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-01 03:59:46,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-01 03:59:46,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-01 03:59:46,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-01 03:59:46,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-01 03:59:46,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-01 03:59:46,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-01 03:59:46,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-01 03:59:46,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-01 03:59:46,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-01 03:59:46,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-01 03:59:46,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-01 03:59:46,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-01 03:59:46,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-01 03:59:46,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-01 03:59:46,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-01 03:59:46,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-01 03:59:46,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-01 03:59:46,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-01 03:59:46,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-01 03:59:46,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-01 03:59:46,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-01 03:59:46,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-01 03:59:46,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-01 03:59:46,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-01 03:59:46,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-01 03:59:46,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-01 03:59:46,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-01 03:59:46,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-01 03:59:46,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-01 03:59:46,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-01 03:59:46,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-01 03:59:46,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-01 03:59:46,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-01 03:59:46,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-01 03:59:46,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-01 03:59:46,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-01 03:59:46,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-01 03:59:46,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-01 03:59:46,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-01 03:59:46,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-01 03:59:46,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-01 03:59:46,456 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022523214112595932); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035308485005105784, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022523214112595932); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-01 03:59:46,456 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-01 03:59:46,457 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016374864541889484, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022523214112595932); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035308485005105784, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022523214112595932); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-01 03:59:46,624 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 260 ms to try 5000 different iterations. Found a solution that moves 987 regions; Going from a computed imbalance of 0.016374864541889484 to a new imbalance of 8.721919892176971E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006112386358117148); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005685352436149611); MoveCostFunction : (multiplier=7.0, imbalance=0.10966666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006112386358117148); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-01 03:59:46,625 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1603750129:80 , srv1237552190:81 , srv1128055035:83 , srv1674392227:83 , srv1684968212:83 , srv1889370679:83 , srv690836760:83 , srv706543699:83 , srv750466578:83 , srv977465688:83 , srv102866622:84 , srv1119419323:84 , srv1233949179:84 , srv1278807079:84 , srv1458942815:84 , srv150676298:84 , srv1602896747:84 , srv1666972805:84 , srv1845586061:84 , srv1889807714:84 , srv1908248180:84 , srv1957510887:84 , srv2045754798:84 , srv2049863221:84 , srv255040105:84 , srv262245399:84 , srv434382004:84 , srv487295681:84 , srv661546627:84 , srv724291470:84 , srv1036111184:85 , srv1054354858:85 , srv1158174829:85 , srv1170600797:85 , srv1261952605:85 , srv1348628925:85 , srv1599343073:85 , srv170639145:85 , srv1766811880:85 , srv1890900503:85 , srv198719940:85 , srv2047922043:85 , srv314392539:85 , srv411345888:85 , srv43636674:85 , srv541664019:85 , srv625461079:85 , srv640331504:85 , srv641324213:85 , srv844145936:85 , srv1058028508:86 , srv1125620273:86 , srv1130918861:86 , srv1232888242:86 , srv1268109957:86 , srv1491302864:86 , srv1680987920:86 , srv1698572321:86 , srv1841280974:86 , srv1921402618:86 , srv1975894660:86 , srv1997010021:86 , srv2002289510:86 , srv2028851517:86 , srv256651103:86 , srv293055971:86 , srv373662136:86 , srv387913170:86 , srv445968766:86 , srv54109376:86 , srv609107286:86 , srv820318764:86 , srv82788003:86 , srv94320653:86 , srv1271451079:87 , srv1340857729:87 , srv1342143617:87 , srv1417362131:87 , srv1456083664:87 , srv1907681838:87 , srv1946596909:87 , srv1961469328:87 , srv2074527133:87 , srv226969518:87 , srv355534485:87 , srv651722412:87 , srv1399395053:88 , srv1588076409:88 , srv1654100984:88 , srv1679060575:88 , srv1738886434:88 , srv1808217702:88 , srv2072550966:88 , srv29668983:88 , srv623621618:88 , srv1603396711:89 , srv1915982440:89 , srv759517640:89 , srv201950313:90 , srv27639952:538 } 2023-04-01 03:59:46,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-01 03:59:46,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv29668983=71, srv1271451079=15, srv27639952=69, srv314392539=72, srv1232888242=10, srv373662136=74, srv1128055035=6, srv94320653=98, srv355534485=73, srv750466578=93, srv1907681838=47, srv1679060575=34, srv1808217702=41, srv1603396711=29, srv690836760=90, srv1674392227=33, srv1890900503=46, srv2045754798=60, srv1961469328=53, srv1278807079=16, srv150676298=25, srv1119419323=4, srv2002289510=57, srv2049863221=62, srv170639145=38, srv1417362131=21, srv1456083664=22, srv387913170=75, srv1997010021=56, srv641324213=87, srv820318764=95, srv2047922043=61, srv487295681=80, srv1680987920=35, srv1666972805=32, srv1845586061=43, srv661546627=89, srv434382004=77, srv445968766=79, srv1125620273=5, srv2072550966=63, srv1130918861=7, srv1237552190=12, srv1588076409=26, srv256651103=67, srv293055971=70, srv1921402618=50, srv2028851517=59, srv1766811880=40, srv1170600797=9, srv1889370679=44, srv1975894660=54, srv226969518=65, srv1915982440=49, srv1058028508=3, srv255040105=66, srv1654100984=31, srv977465688=99, srv1054354858=2, srv411345888=76, srv1233949179=11, srv541664019=82, srv2074527133=64, srv1342143617=18, srv724291470=92, srv1261952605=13, srv1491302864=24, srv1889807714=45, srv1599343073=27, srv651722412=88, srv759517640=94, srv623621618=84, srv1036111184=1, srv1684968212=36, srv198719940=55, srv1348628925=19, srv201950313=58, srv1841280974=42, srv43636674=78, srv1946596909=51, srv1738886434=39, srv625461079=85, srv1602896747=28, srv844145936=97, srv1458942815=23, srv262245399=68, srv1908248180=48, srv54109376=81, srv1603750129=30, srv1957510887=52, srv640331504=86, srv1399395053=20, srv609107286=83, srv1158174829=8, srv706543699=91, srv1698572321=37, srv82788003=96, srv102866622=0, srv1340857729=17, srv1268109957=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-01 03:59:46,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-01 03:59:46,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-01 03:59:46,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-01 03:59:46,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-01 03:59:46,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-01 03:59:46,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-01 03:59:46,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-01 03:59:46,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-01 03:59:46,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-01 03:59:46,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-01 03:59:46,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-01 03:59:46,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-01 03:59:46,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-01 03:59:46,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-01 03:59:46,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-01 03:59:46,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-01 03:59:46,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-01 03:59:46,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-01 03:59:46,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-01 03:59:46,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-01 03:59:46,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-01 03:59:46,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-01 03:59:46,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-01 03:59:46,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-01 03:59:46,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-01 03:59:46,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-01 03:59:46,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-01 03:59:46,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-01 03:59:46,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-01 03:59:46,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-01 03:59:46,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-01 03:59:46,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-01 03:59:46,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-01 03:59:46,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-01 03:59:46,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-01 03:59:46,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-01 03:59:46,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-01 03:59:46,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-01 03:59:46,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-01 03:59:46,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-01 03:59:46,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-01 03:59:46,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-01 03:59:46,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-01 03:59:46,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-01 03:59:46,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-01 03:59:46,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-01 03:59:46,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-01 03:59:46,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-01 03:59:46,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-01 03:59:46,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-01 03:59:46,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-01 03:59:46,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-01 03:59:46,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-01 03:59:46,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-01 03:59:46,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-01 03:59:46,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-01 03:59:46,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-01 03:59:46,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-01 03:59:46,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-01 03:59:46,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-01 03:59:46,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-01 03:59:46,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-01 03:59:46,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-01 03:59:46,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-01 03:59:46,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-01 03:59:46,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-01 03:59:46,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-01 03:59:46,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-01 03:59:46,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-01 03:59:46,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-01 03:59:46,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-01 03:59:46,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-01 03:59:46,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-01 03:59:46,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-01 03:59:46,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-01 03:59:46,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-01 03:59:46,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-01 03:59:46,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-01 03:59:46,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-01 03:59:46,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-01 03:59:46,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-01 03:59:46,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-01 03:59:46,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-01 03:59:46,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-01 03:59:46,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-01 03:59:46,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-01 03:59:46,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-01 03:59:46,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-01 03:59:46,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-01 03:59:46,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-01 03:59:46,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-01 03:59:46,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-01 03:59:46,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-01 03:59:46,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-01 03:59:46,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-01 03:59:46,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-01 03:59:46,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-01 03:59:46,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-01 03:59:46,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-01 03:59:46,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-01 03:59:46,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-01 03:59:46,733 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006112386358117148); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005685352436149611); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006112386358117148); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-01 03:59:46,734 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-01 03:59:46,734 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.652808675033307E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006112386358117148); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005685352436149611); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006112386358117148); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-01 03:59:46,977 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 324 ms to try 5000 different iterations. Found a solution that moves 260 regions; Going from a computed imbalance of 8.652808675033307E-4 to a new imbalance of 1.9445829026684864E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.028888888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-01 03:59:46,978 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1036111184:83 , srv1271451079:83 , srv1889370679:83 , srv1889807714:83 , srv1946596909:83 , srv2049863221:83 , srv373662136:83 , srv54109376:83 , srv1128055035:84 , srv1268109957:84 , srv1278807079:84 , srv1340857729:84 , srv1342143617:84 , srv1399395053:84 , srv1491302864:84 , srv1603396711:84 , srv1654100984:84 , srv1679060575:84 , srv1680987920:84 , srv170639145:84 , srv1890900503:84 , srv1907681838:84 , srv1915982440:84 , srv1961469328:84 , srv1975894660:84 , srv198719940:84 , srv1997010021:84 , srv201950313:84 , srv255040105:84 , srv256651103:84 , srv29668983:84 , srv355534485:84 , srv411345888:84 , srv43636674:84 , srv623621618:84 , srv641324213:84 , srv651722412:84 , srv661546627:84 , srv724291470:84 , srv750466578:84 , srv759517640:84 , srv94320653:84 , srv977465688:84 , srv102866622:85 , srv1054354858:85 , srv1058028508:85 , srv1119419323:85 , srv1125620273:85 , srv1130918861:85 , srv1158174829:85 , srv1170600797:85 , srv1232888242:85 , srv1233949179:85 , srv1261952605:85 , srv1348628925:85 , srv1417362131:85 , srv1456083664:85 , srv1458942815:85 , srv150676298:85 , srv1588076409:85 , srv1599343073:85 , srv1602896747:85 , srv1603750129:85 , srv1674392227:85 , srv1684968212:85 , srv1698572321:85 , srv1738886434:85 , srv1808217702:85 , srv1841280974:85 , srv1845586061:85 , srv1908248180:85 , srv1921402618:85 , srv1957510887:85 , srv2002289510:85 , srv2028851517:85 , srv2045754798:85 , srv2072550966:85 , srv2074527133:85 , srv226969518:85 , srv262245399:85 , srv293055971:85 , srv314392539:85 , srv387913170:85 , srv434382004:85 , srv487295681:85 , srv541664019:85 , srv609107286:85 , srv625461079:85 , srv640331504:85 , srv690836760:85 , srv820318764:85 , srv82788003:85 , srv844145936:85 , srv1237552190:86 , srv1666972805:86 , srv1766811880:86 , srv2047922043:86 , srv445968766:86 , srv706543699:86 , srv27639952:630 } 2023-04-01 03:59:46,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-01 03:59:46,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv29668983=71, srv1271451079=15, srv27639952=69, srv314392539=72, srv1232888242=10, srv373662136=74, srv1128055035=6, srv94320653=98, srv355534485=73, srv750466578=93, srv1907681838=47, srv1679060575=34, srv1808217702=41, srv1603396711=29, srv690836760=90, srv1674392227=33, srv1890900503=46, srv2045754798=60, srv1961469328=53, srv1278807079=16, srv150676298=25, srv1119419323=4, srv2002289510=57, srv2049863221=62, srv170639145=38, srv1417362131=21, srv1456083664=22, srv387913170=75, srv1997010021=56, srv641324213=87, srv820318764=95, srv2047922043=61, srv487295681=80, srv1680987920=35, srv1666972805=32, srv1845586061=43, srv661546627=89, srv434382004=77, srv445968766=79, srv1125620273=5, srv2072550966=63, srv1130918861=7, srv1237552190=12, srv1588076409=26, srv256651103=67, srv293055971=70, srv1921402618=50, srv2028851517=59, srv1766811880=40, srv1170600797=9, srv1889370679=44, srv1975894660=54, srv226969518=65, srv1915982440=49, srv1058028508=3, srv255040105=66, srv1654100984=31, srv977465688=99, srv1054354858=2, srv411345888=76, srv1233949179=11, srv541664019=82, srv2074527133=64, srv1342143617=18, srv724291470=92, srv1261952605=13, srv1491302864=24, srv1889807714=45, srv1599343073=27, srv651722412=88, srv759517640=94, srv623621618=84, srv1036111184=1, srv1684968212=36, srv198719940=55, srv1348628925=19, srv201950313=58, srv1841280974=42, srv43636674=78, srv1946596909=51, srv1738886434=39, srv625461079=85, srv1602896747=28, srv844145936=97, srv1458942815=23, srv262245399=68, srv1908248180=48, srv54109376=81, srv1603750129=30, srv1957510887=52, srv640331504=86, srv1399395053=20, srv609107286=83, srv1158174829=8, srv706543699=91, srv1698572321=37, srv82788003=96, srv102866622=0, srv1340857729=17, srv1268109957=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-01 03:59:47,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-01 03:59:47,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-01 03:59:47,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-01 03:59:47,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-01 03:59:47,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-01 03:59:47,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-01 03:59:47,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-01 03:59:47,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-01 03:59:47,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-01 03:59:47,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-01 03:59:47,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-01 03:59:47,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-01 03:59:47,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-01 03:59:47,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-01 03:59:47,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-01 03:59:47,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-01 03:59:47,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-01 03:59:47,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-01 03:59:47,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-01 03:59:47,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-01 03:59:47,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-01 03:59:47,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-01 03:59:47,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-01 03:59:47,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-01 03:59:47,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-01 03:59:47,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-01 03:59:47,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-01 03:59:47,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-01 03:59:47,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-01 03:59:47,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-01 03:59:47,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-01 03:59:47,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-01 03:59:47,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-01 03:59:47,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-01 03:59:47,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-01 03:59:47,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-01 03:59:47,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-01 03:59:47,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-01 03:59:47,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-01 03:59:47,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-01 03:59:47,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-01 03:59:47,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-01 03:59:47,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-01 03:59:47,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-01 03:59:47,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-01 03:59:47,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-01 03:59:47,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-01 03:59:47,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-01 03:59:47,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-01 03:59:47,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-01 03:59:47,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-01 03:59:47,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-01 03:59:47,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-01 03:59:47,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-01 03:59:47,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-01 03:59:47,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-01 03:59:47,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-01 03:59:47,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-01 03:59:47,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-01 03:59:47,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-01 03:59:47,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-01 03:59:47,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-01 03:59:47,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-01 03:59:47,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-01 03:59:47,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-01 03:59:47,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-01 03:59:47,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-01 03:59:47,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-01 03:59:47,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-01 03:59:47,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-01 03:59:47,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-01 03:59:47,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-01 03:59:47,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-01 03:59:47,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-01 03:59:47,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-01 03:59:47,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-01 03:59:47,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-01 03:59:47,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-01 03:59:47,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-01 03:59:47,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-01 03:59:47,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-01 03:59:47,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-01 03:59:47,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-01 03:59:47,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-01 03:59:47,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-01 03:59:47,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-01 03:59:47,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-01 03:59:47,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-01 03:59:47,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-01 03:59:47,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-01 03:59:47,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-01 03:59:47,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-01 03:59:47,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-01 03:59:47,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-01 03:59:47,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-01 03:59:47,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-01 03:59:47,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-01 03:59:47,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-01 03:59:47,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-01 03:59:47,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-01 03:59:47,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-01 03:59:47,142 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.7625270115099008E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-01 03:59:47,142 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1036111184:83 , srv1271451079:83 , srv1889370679:83 , srv1889807714:83 , srv1946596909:83 , srv2049863221:83 , srv373662136:83 , srv54109376:83 , srv1128055035:84 , srv1268109957:84 , srv1278807079:84 , srv1340857729:84 , srv1342143617:84 , srv1399395053:84 , srv1491302864:84 , srv1603396711:84 , srv1654100984:84 , srv1679060575:84 , srv1680987920:84 , srv170639145:84 , srv1890900503:84 , srv1907681838:84 , srv1915982440:84 , srv1961469328:84 , srv1975894660:84 , srv198719940:84 , srv1997010021:84 , srv201950313:84 , srv255040105:84 , srv256651103:84 , srv29668983:84 , srv355534485:84 , srv411345888:84 , srv43636674:84 , srv623621618:84 , srv641324213:84 , srv651722412:84 , srv661546627:84 , srv724291470:84 , srv750466578:84 , srv759517640:84 , srv94320653:84 , srv977465688:84 , srv102866622:85 , srv1054354858:85 , srv1058028508:85 , srv1119419323:85 , srv1125620273:85 , srv1130918861:85 , srv1158174829:85 , srv1170600797:85 , srv1232888242:85 , srv1233949179:85 , srv1261952605:85 , srv1348628925:85 , srv1417362131:85 , srv1456083664:85 , srv1458942815:85 , srv150676298:85 , srv1588076409:85 , srv1599343073:85 , srv1602896747:85 , srv1603750129:85 , srv1674392227:85 , srv1684968212:85 , srv1698572321:85 , srv1738886434:85 , srv1808217702:85 , srv1841280974:85 , srv1845586061:85 , srv1908248180:85 , srv1921402618:85 , srv1957510887:85 , srv2002289510:85 , srv2028851517:85 , srv2045754798:85 , srv2072550966:85 , srv2074527133:85 , srv226969518:85 , srv262245399:85 , srv293055971:85 , srv314392539:85 , srv387913170:85 , srv434382004:85 , srv487295681:85 , srv541664019:85 , srv609107286:85 , srv625461079:85 , srv640331504:85 , srv690836760:85 , srv820318764:85 , srv82788003:85 , srv844145936:85 , srv1237552190:86 , srv1666972805:86 , srv1766811880:86 , srv2047922043:86 , srv445968766:86 , srv706543699:86 , srv27639952:630 } 2023-04-01 03:59:47,162 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=501 (was 476) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=9063 (was 9587) 2023-04-01 03:59:47,169 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=501, ProcessCount=167, AvailableMemoryMB=9063 2023-04-01 03:59:47,170 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-01 03:59:47,170 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-04-01 03:59:47,181 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1084520071:84 , srv11319075:84 , srv1177457946:84 , srv118524283:84 , srv1417789926:84 , srv1546346435:84 , srv162726815:84 , srv169583670:84 , srv1721323684:84 , srv1800009680:84 , srv1814684256:84 , srv1860736681:84 , srv1885744314:84 , srv1988091003:84 , srv257927374:84 , srv372033292:84 , srv385793247:84 , srv421470590:84 , srv479439661:84 , srv524367079:84 , srv563968764:84 , srv588168014:84 , srv597222028:84 , srv661950062:84 , srv692873785:84 , srv780027814:84 , srv836633045:84 , srv850254788:84 , srv9873411:84 , srv1494210270:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-04-01 03:59:47,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-01 03:59:47,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv169583670=8, srv1800009680=10, srv1721323684=9, srv118524283=3, srv11319075=1, srv9873411=29, srv162726815=7, srv850254788=28, srv1494210270=5, srv1084520071=0, srv257927374=15, srv1988091003=14, srv588168014=22, srv563968764=21, srv597222028=23, srv692873785=25, srv836633045=27, srv421470590=18, srv1417789926=4, srv661950062=24, srv1546346435=6, srv479439661=19, srv372033292=16, srv1814684256=11, srv1860736681=12, srv524367079=20, srv780027814=26, srv1177457946=2, srv1885744314=13, srv385793247=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-01 03:59:47,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-01 03:59:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-01 03:59:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-01 03:59:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-01 03:59:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-01 03:59:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-01 03:59:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-01 03:59:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-01 03:59:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-01 03:59:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-01 03:59:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-01 03:59:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-01 03:59:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-01 03:59:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-01 03:59:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-01 03:59:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-01 03:59:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-01 03:59:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-01 03:59:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-01 03:59:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-01 03:59:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-01 03:59:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-01 03:59:47,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-01 03:59:47,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-01 03:59:47,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-01 03:59:47,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-01 03:59:47,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-01 03:59:47,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-01 03:59:47,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-01 03:59:47,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-01 03:59:47,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-01 03:59:47,224 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.120290607544579, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-01 03:59:47,224 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912572075040803, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.120290607544579, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-04-01 04:01:47,202 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9432879 different iterations. Found a solution that moves 1840 regions; Going from a computed imbalance of 0.9912572075040803 to a new imbalance of 4.2946517914333035E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6814814814814815, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-01 04:01:47,209 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1084520071:90 , srv11319075:90 , srv1177457946:90 , srv118524283:90 , srv1417789926:90 , srv1494210270:90 , srv1546346435:90 , srv162726815:90 , srv169583670:90 , srv1721323684:90 , srv1800009680:90 , srv1814684256:90 , srv1860736681:90 , srv1885744314:90 , srv1988091003:90 , srv257927374:90 , srv372033292:90 , srv385793247:90 , srv421470590:90 , srv479439661:90 , srv524367079:90 , srv563968764:90 , srv588168014:90 , srv597222028:90 , srv661950062:90 , srv692873785:90 , srv780027814:90 , srv836633045:90 , srv850254788:90 , srv9873411:90 } 2023-04-01 04:01:47,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-01 04:01:47,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv169583670=8, srv1800009680=10, srv1721323684=9, srv118524283=3, srv11319075=1, srv9873411=29, srv162726815=7, srv850254788=28, srv1494210270=5, srv1084520071=0, srv257927374=15, srv1988091003=14, srv588168014=22, srv563968764=21, srv597222028=23, srv692873785=25, srv836633045=27, srv421470590=18, srv1417789926=4, srv661950062=24, srv1546346435=6, srv479439661=19, srv372033292=16, srv1814684256=11, srv1860736681=12, srv524367079=20, srv780027814=26, srv1177457946=2, srv1885744314=13, srv385793247=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-01 04:01:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-01 04:01:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-01 04:01:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-01 04:01:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-01 04:01:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-01 04:01:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-01 04:01:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-01 04:01:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-01 04:01:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-01 04:01:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-01 04:01:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-01 04:01:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-01 04:01:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-01 04:01:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-01 04:01:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-01 04:01:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-01 04:01:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-01 04:01:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-01 04:01:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-01 04:01:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-01 04:01:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-01 04:01:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-01 04:01:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-01 04:01:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-01 04:01:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-01 04:01:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-01 04:01:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-01 04:01:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-01 04:01:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-01 04:01:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-01 04:01:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-01 04:01:47,226 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-01 04:01:47,227 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1084520071:90 , srv11319075:90 , srv1177457946:90 , srv118524283:90 , srv1417789926:90 , srv1494210270:90 , srv1546346435:90 , srv162726815:90 , srv169583670:90 , srv1721323684:90 , srv1800009680:90 , srv1814684256:90 , srv1860736681:90 , srv1885744314:90 , srv1988091003:90 , srv257927374:90 , srv372033292:90 , srv385793247:90 , srv421470590:90 , srv479439661:90 , srv524367079:90 , srv563968764:90 , srv588168014:90 , srv597222028:90 , srv661950062:90 , srv692873785:90 , srv780027814:90 , srv836633045:90 , srv850254788:90 , srv9873411:90 } 2023-04-01 04:01:47,237 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=200 (was 501), ProcessCount=162 (was 167), AvailableMemoryMB=9742 (was 9063) - AvailableMemoryMB LEAK? -