2023-07-14 03:53:12,350 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-07-14 03:53:12,659 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-14 03:53:12,785 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-14 03:53:12,849 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-14 03:53:12,857 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-14 03:53:12,873 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=350, ProcessCount=172, AvailableMemoryMB=8692 2023-07-14 03:53:12,875 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-14 03:53:12,875 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-14 03:53:13,016 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1018303395:84 , srv1024989881:84 , srv1030559665:84 , srv1088113130:84 , srv1118335473:84 , srv112754407:84 , srv1138636656:84 , srv1139024944:84 , srv1145417734:84 , srv1169653468:84 , srv1181555740:84 , srv1192330838:84 , srv1193744900:84 , srv1200643094:84 , srv1220519161:84 , srv1221144344:84 , srv1229683173:84 , srv124516735:84 , srv1253438788:84 , srv1261847163:84 , srv1292189528:84 , srv1296764054:84 , srv1298357617:84 , srv1300183793:84 , srv1329859992:84 , srv1330511390:84 , srv1347650974:84 , srv1352248510:84 , srv135944097:84 , srv1385441644:84 , srv1435076252:84 , srv1438662176:84 , srv1439285646:84 , srv1454621518:84 , srv147408003:84 , srv1498380972:84 , srv1508375145:84 , srv1519147984:84 , srv1525869397:84 , srv1594602753:84 , srv1596645681:84 , srv1617857967:84 , srv1701308557:84 , srv1705252710:84 , srv1718254437:84 , srv1785850819:84 , srv1798508303:84 , srv1810088722:84 , srv1818135584:84 , srv1879838022:84 , srv1895274985:84 , srv1896552389:84 , srv1946473552:84 , srv2033219794:84 , srv2037202269:84 , srv2091094945:84 , srv20945049:84 , srv2123774274:84 , srv213162655:84 , srv23091728:84 , srv247783445:84 , srv250178053:84 , srv265740922:84 , srv287849855:84 , srv306662468:84 , srv388810147:84 , srv420548851:84 , srv454645002:84 , srv462032142:84 , srv477753820:84 , srv479583554:84 , srv484785645:84 , srv519981914:84 , srv574905219:84 , srv615172859:84 , srv632009843:84 , srv64355067:84 , srv649533188:84 , srv671542763:84 , srv707637093:84 , srv712627254:84 , srv717929960:84 , srv727964583:84 , srv746279826:84 , srv751139417:84 , srv758794738:84 , srv844078397:84 , srv857661769:84 , srv86559302:84 , srv871498846:84 , srv880688752:84 , srv88715082:84 , srv901785353:84 , srv934444099:84 , srv946641578:84 , srv959059719:84 , srv969269787:84 , srv984100583:84 , srv998388383:84 , srv784262362:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-07-14 03:53:13,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-14 03:53:13,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1810088722=47, srv1200643094=13, srv1261847163=19, srv1519147984=37, srv1292189528=20, srv2123774274=57, srv23091728=59, srv1718254437=44, srv1330511390=25, srv477753820=69, srv2033219794=53, srv649533188=77, srv707637093=79, srv484785645=71, srv1895274985=50, srv1145417734=8, srv1946473552=52, srv751139417=84, srv2091094945=55, srv1300183793=23, srv1296764054=21, srv1798508303=46, srv519981914=72, srv1138636656=6, srv1221144344=15, srv250178053=61, srv1253438788=18, srv969269787=97, srv420548851=66, srv135944097=28, srv112754407=5, srv1818135584=48, srv984100583=98, srv1139024944=7, srv998388383=99, srv632009843=75, srv1454621518=33, srv1024989881=1, srv247783445=60, srv64355067=76, srv746279826=83, srv844078397=87, srv1229683173=16, srv1298357617=22, srv479583554=70, srv727964583=82, srv784262362=86, srv574905219=73, srv1169653468=9, srv880688752=91, srv901785353=93, srv147408003=34, srv959059719=96, srv1181555740=10, srv1439285646=32, srv671542763=78, srv1352248510=27, srv1617857967=41, srv857661769=88, srv712627254=80, srv1118335473=4, srv1220519161=14, srv1596645681=40, srv1508375145=36, srv615172859=74, srv388810147=65, srv213162655=58, srv946641578=95, srv306662468=64, srv124516735=17, srv1498380972=35, srv1193744900=12, srv287849855=63, srv758794738=85, srv1705252710=43, srv1896552389=51, srv934444099=94, srv265740922=62, srv1385441644=29, srv1438662176=31, srv717929960=81, srv871498846=90, srv1785850819=45, srv88715082=92, srv1018303395=0, srv20945049=56, srv1347650974=26, srv1030559665=2, srv1192330838=11, srv1701308557=42, srv462032142=68, srv1329859992=24, srv454645002=67, srv2037202269=54, srv86559302=89, srv1525869397=38, srv1594602753=39, srv1879838022=49, srv1435076252=30, srv1088113130=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-14 03:53:13,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-14 03:53:13,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-14 03:53:13,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-14 03:53:13,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-14 03:53:13,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-14 03:53:13,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-14 03:53:13,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-14 03:53:13,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-14 03:53:13,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-14 03:53:13,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-14 03:53:13,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-14 03:53:13,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-14 03:53:13,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-14 03:53:13,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-14 03:53:13,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-14 03:53:13,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-14 03:53:13,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-14 03:53:13,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-14 03:53:13,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-14 03:53:13,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-14 03:53:13,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-14 03:53:13,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-14 03:53:13,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-14 03:53:13,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-14 03:53:13,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-14 03:53:13,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-14 03:53:13,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-14 03:53:13,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-14 03:53:13,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-14 03:53:13,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-14 03:53:13,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-14 03:53:13,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-14 03:53:13,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-14 03:53:13,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-14 03:53:13,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-14 03:53:13,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-14 03:53:13,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-14 03:53:13,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-14 03:53:13,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-14 03:53:13,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-14 03:53:13,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-14 03:53:13,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-14 03:53:13,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-14 03:53:13,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-14 03:53:13,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-14 03:53:13,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-14 03:53:13,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-14 03:53:13,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-14 03:53:13,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-14 03:53:13,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-14 03:53:13,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-14 03:53:13,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-14 03:53:13,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-14 03:53:13,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-14 03:53:13,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-14 03:53:13,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-14 03:53:13,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-14 03:53:13,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-14 03:53:13,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-14 03:53:13,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-14 03:53:13,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-14 03:53:13,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-14 03:53:13,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-14 03:53:13,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-14 03:53:13,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-14 03:53:13,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-14 03:53:13,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-14 03:53:13,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-14 03:53:13,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-14 03:53:13,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-14 03:53:13,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-14 03:53:13,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-14 03:53:13,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-14 03:53:13,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-14 03:53:13,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-14 03:53:13,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-14 03:53:13,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-14 03:53:13,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-14 03:53:13,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-14 03:53:13,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-14 03:53:13,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-14 03:53:13,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-14 03:53:13,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-14 03:53:13,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-14 03:53:13,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-14 03:53:13,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-14 03:53:13,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-14 03:53:13,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-14 03:53:13,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-14 03:53:13,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-14 03:53:13,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-14 03:53:13,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-14 03:53:13,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-14 03:53:13,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-14 03:53:13,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-14 03:53:13,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-14 03:53:13,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-14 03:53:13,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-14 03:53:13,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-14 03:53:13,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-14 03:53:13,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-14 03:53:13,335 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-14 03:53:13,335 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-14 03:53:13,336 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-14 03:53:13,657 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 489 ms to try 5000 different iterations. Found a solution that moves 4680 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17788757961751522. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039300848310055744, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057438787991052107, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.52, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039300848310055744, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17308333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23975, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-14 03:53:13,668 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1192330838:81 , srv1118335473:82 , srv1300183793:82 , srv1525869397:82 , srv746279826:82 , srv86559302:82 , srv880688752:82 , srv946641578:82 , srv998388383:82 , srv1030559665:83 , srv1229683173:83 , srv1330511390:83 , srv1439285646:83 , srv1519147984:83 , srv1718254437:83 , srv1818135584:83 , srv649533188:83 , srv901785353:83 , srv1139024944:84 , srv1200643094:84 , srv124516735:84 , srv1454621518:84 , srv1596645681:84 , srv1798508303:84 , srv574905219:84 , srv88715082:84 , srv1145417734:85 , srv1261847163:85 , srv1329859992:85 , srv135944097:85 , srv2037202269:85 , srv479583554:85 , srv519981914:85 , srv64355067:85 , srv758794738:85 , srv969269787:85 , srv984100583:85 , srv1193744900:86 , srv1253438788:86 , srv1296764054:86 , srv147408003:86 , srv1896552389:86 , srv20945049:86 , srv2123774274:86 , srv287849855:86 , srv477753820:86 , srv632009843:86 , srv707637093:86 , srv727964583:86 , srv959059719:86 , srv1018303395:87 , srv1024989881:87 , srv1220519161:87 , srv1221144344:87 , srv1705252710:87 , srv1810088722:87 , srv1895274985:87 , srv1946473552:87 , srv2091094945:87 , srv250178053:87 , srv265740922:87 , srv484785645:87 , srv712627254:87 , srv934444099:87 , srv1088113130:88 , srv112754407:88 , srv1385441644:88 , srv1438662176:88 , srv1617857967:88 , srv1701308557:88 , srv1879838022:88 , srv23091728:88 , srv454645002:88 , srv462032142:88 , srv615172859:88 , srv671542763:88 , srv844078397:88 , srv857661769:88 , srv871498846:88 , srv1169653468:89 , srv1292189528:89 , srv1352248510:89 , srv1498380972:89 , srv247783445:89 , srv751139417:89 , srv1138636656:90 , srv1181555740:90 , srv1298357617:90 , srv1435076252:90 , srv2033219794:90 , srv388810147:90 , srv1347650974:91 , srv1785850819:91 , srv1508375145:92 , srv717929960:92 , srv1594602753:93 , srv306662468:93 , srv213162655:94 , srv420548851:94 , srv784262362:439 } 2023-07-14 03:53:13,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-14 03:53:13,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1810088722=47, srv1200643094=13, srv1261847163=19, srv1519147984=37, srv1292189528=20, srv2123774274=57, srv23091728=59, srv1718254437=44, srv1330511390=25, srv477753820=69, srv2033219794=53, srv649533188=77, srv707637093=79, srv484785645=71, srv1895274985=50, srv1145417734=8, srv1946473552=52, srv751139417=84, srv2091094945=55, srv1300183793=23, srv1296764054=21, srv1798508303=46, srv519981914=72, srv1138636656=6, srv1221144344=15, srv250178053=61, srv1253438788=18, srv969269787=97, srv420548851=66, srv135944097=28, srv112754407=5, srv1818135584=48, srv984100583=98, srv1139024944=7, srv998388383=99, srv632009843=75, srv1454621518=33, srv1024989881=1, srv247783445=60, srv64355067=76, srv746279826=83, srv844078397=87, srv1229683173=16, srv1298357617=22, srv479583554=70, srv727964583=82, srv784262362=86, srv574905219=73, srv1169653468=9, srv880688752=91, srv901785353=93, srv147408003=34, srv959059719=96, srv1181555740=10, srv1439285646=32, srv671542763=78, srv1352248510=27, srv1617857967=41, srv857661769=88, srv712627254=80, srv1118335473=4, srv1220519161=14, srv1596645681=40, srv1508375145=36, srv615172859=74, srv388810147=65, srv213162655=58, srv946641578=95, srv306662468=64, srv124516735=17, srv1498380972=35, srv1193744900=12, srv287849855=63, srv758794738=85, srv1705252710=43, srv1896552389=51, srv934444099=94, srv265740922=62, srv1385441644=29, srv1438662176=31, srv717929960=81, srv871498846=90, srv1785850819=45, srv88715082=92, srv1018303395=0, srv20945049=56, srv1347650974=26, srv1030559665=2, srv1192330838=11, srv1701308557=42, srv462032142=68, srv1329859992=24, srv454645002=67, srv2037202269=54, srv86559302=89, srv1525869397=38, srv1594602753=39, srv1879838022=49, srv1435076252=30, srv1088113130=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-14 03:53:13,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-14 03:53:13,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-14 03:53:13,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-14 03:53:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-14 03:53:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-14 03:53:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-14 03:53:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-14 03:53:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-14 03:53:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-14 03:53:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-14 03:53:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-14 03:53:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-14 03:53:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-14 03:53:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-14 03:53:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-14 03:53:13,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-14 03:53:13,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-14 03:53:13,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-14 03:53:13,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-14 03:53:13,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-14 03:53:13,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-14 03:53:13,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-14 03:53:13,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-14 03:53:13,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-14 03:53:13,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-14 03:53:13,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-14 03:53:13,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-14 03:53:13,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-14 03:53:13,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-14 03:53:13,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-14 03:53:13,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-14 03:53:13,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-14 03:53:13,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-14 03:53:13,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-14 03:53:13,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-14 03:53:13,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-14 03:53:13,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-14 03:53:13,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-14 03:53:13,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-14 03:53:13,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-14 03:53:13,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-14 03:53:13,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-14 03:53:13,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-14 03:53:13,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-14 03:53:13,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-14 03:53:13,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-14 03:53:13,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-14 03:53:13,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-14 03:53:13,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-14 03:53:13,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-14 03:53:13,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-14 03:53:13,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-14 03:53:13,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-14 03:53:13,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-14 03:53:13,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-14 03:53:13,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-14 03:53:13,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-14 03:53:13,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-14 03:53:13,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-14 03:53:13,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-14 03:53:13,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-14 03:53:13,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-14 03:53:13,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-14 03:53:13,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-14 03:53:13,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-14 03:53:13,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-14 03:53:13,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-14 03:53:13,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-14 03:53:13,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-14 03:53:13,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-14 03:53:13,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-14 03:53:13,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-14 03:53:13,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-14 03:53:13,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-14 03:53:13,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-14 03:53:13,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-14 03:53:13,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-14 03:53:13,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-14 03:53:13,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-14 03:53:13,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-14 03:53:13,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-14 03:53:13,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-14 03:53:13,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-14 03:53:13,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-14 03:53:13,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-14 03:53:13,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-14 03:53:13,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-14 03:53:13,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-14 03:53:13,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-14 03:53:13,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-14 03:53:13,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-14 03:53:13,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-14 03:53:13,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-14 03:53:13,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-14 03:53:13,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-14 03:53:13,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-14 03:53:13,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-14 03:53:13,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-14 03:53:13,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-14 03:53:13,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-14 03:53:13,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-14 03:53:13,826 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039300848310055744, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057438787991052107, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039300848310055744, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17308333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23975, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-14 03:53:13,826 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-14 03:53:13,827 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17785480955710667, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039300848310055744, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057438787991052107, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039300848310055744, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17308333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23975, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-14 03:53:14,107 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 379 ms to try 5000 different iterations. Found a solution that moves 2641 regions; Going from a computed imbalance of 0.17785480955710667 to a new imbalance of 0.01313526633746795. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023516198618315033); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035448069856702444, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29344444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023516198618315033); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0435, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-14 03:53:14,116 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1594602753:78 , srv420548851:78 , srv213162655:79 , srv1298357617:80 , srv388810147:80 , srv1138636656:81 , srv1508375145:81 , srv306662468:81 , srv717929960:81 , srv1024989881:82 , srv112754407:82 , srv1292189528:82 , srv1347650974:82 , srv2033219794:82 , srv247783445:82 , srv751139417:82 , srv1181555740:83 , srv1701308557:83 , srv1879838022:83 , srv23091728:83 , srv671542763:83 , srv1018303395:84 , srv1221144344:84 , srv1296764054:84 , srv1385441644:84 , srv1435076252:84 , srv1438662176:84 , srv1498380972:84 , srv1705252710:84 , srv1785850819:84 , srv1946473552:84 , srv265740922:84 , srv479583554:84 , srv484785645:84 , srv857661769:84 , srv871498846:84 , srv984100583:84 , srv1088113130:85 , srv1253438788:85 , srv1352248510:85 , srv135944097:85 , srv147408003:85 , srv1596645681:85 , srv1617857967:85 , srv1810088722:85 , srv1895274985:85 , srv1896552389:85 , srv2037202269:85 , srv2091094945:85 , srv462032142:85 , srv632009843:85 , srv712627254:85 , srv727964583:85 , srv934444099:85 , srv1169653468:86 , srv1220519161:86 , srv1261847163:86 , srv1329859992:86 , srv1718254437:86 , srv250178053:86 , srv287849855:86 , srv454645002:86 , srv615172859:86 , srv844078397:86 , srv959059719:86 , srv969269787:86 , srv1118335473:87 , srv1193744900:87 , srv124516735:87 , srv1798508303:87 , srv1818135584:87 , srv2123774274:87 , srv477753820:87 , srv519981914:87 , srv649533188:87 , srv1139024944:88 , srv1145417734:88 , srv1229683173:88 , srv1454621518:88 , srv64355067:88 , srv707637093:88 , srv1300183793:89 , srv1330511390:89 , srv1439285646:89 , srv1519147984:89 , srv574905219:89 , srv746279826:89 , srv758794738:89 , srv880688752:89 , srv901785353:89 , srv1525869397:90 , srv86559302:90 , srv88715082:90 , srv946641578:90 , srv998388383:90 , srv1030559665:91 , srv1200643094:91 , srv20945049:91 , srv1192330838:92 , srv784262362:544 } 2023-07-14 03:53:14,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-14 03:53:14,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1810088722=47, srv1200643094=13, srv1261847163=19, srv1519147984=37, srv1292189528=20, srv2123774274=57, srv23091728=59, srv1718254437=44, srv1330511390=25, srv477753820=69, srv2033219794=53, srv649533188=77, srv707637093=79, srv484785645=71, srv1895274985=50, srv1145417734=8, srv1946473552=52, srv751139417=84, srv2091094945=55, srv1300183793=23, srv1296764054=21, srv1798508303=46, srv519981914=72, srv1138636656=6, srv1221144344=15, srv250178053=61, srv1253438788=18, srv969269787=97, srv420548851=66, srv135944097=28, srv112754407=5, srv1818135584=48, srv984100583=98, srv1139024944=7, srv998388383=99, srv632009843=75, srv1454621518=33, srv1024989881=1, srv247783445=60, srv64355067=76, srv746279826=83, srv844078397=87, srv1229683173=16, srv1298357617=22, srv479583554=70, srv727964583=82, srv784262362=86, srv574905219=73, srv1169653468=9, srv880688752=91, srv901785353=93, srv147408003=34, srv959059719=96, srv1181555740=10, srv1439285646=32, srv671542763=78, srv1352248510=27, srv1617857967=41, srv857661769=88, srv712627254=80, srv1118335473=4, srv1220519161=14, srv1596645681=40, srv1508375145=36, srv615172859=74, srv388810147=65, srv213162655=58, srv946641578=95, srv306662468=64, srv124516735=17, srv1498380972=35, srv1193744900=12, srv287849855=63, srv758794738=85, srv1705252710=43, srv1896552389=51, srv934444099=94, srv265740922=62, srv1385441644=29, srv1438662176=31, srv717929960=81, srv871498846=90, srv1785850819=45, srv88715082=92, srv1018303395=0, srv20945049=56, srv1347650974=26, srv1030559665=2, srv1192330838=11, srv1701308557=42, srv462032142=68, srv1329859992=24, srv454645002=67, srv2037202269=54, srv86559302=89, srv1525869397=38, srv1594602753=39, srv1879838022=49, srv1435076252=30, srv1088113130=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-14 03:53:14,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-14 03:53:14,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-14 03:53:14,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-14 03:53:14,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-14 03:53:14,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-14 03:53:14,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-14 03:53:14,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-14 03:53:14,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-14 03:53:14,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-14 03:53:14,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-14 03:53:14,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-14 03:53:14,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-14 03:53:14,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-14 03:53:14,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-14 03:53:14,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-14 03:53:14,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-14 03:53:14,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-14 03:53:14,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-14 03:53:14,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-14 03:53:14,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-14 03:53:14,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-14 03:53:14,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-14 03:53:14,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-14 03:53:14,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-14 03:53:14,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-14 03:53:14,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-14 03:53:14,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-14 03:53:14,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-14 03:53:14,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-14 03:53:14,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-14 03:53:14,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-14 03:53:14,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-14 03:53:14,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-14 03:53:14,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-14 03:53:14,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-14 03:53:14,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-14 03:53:14,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-14 03:53:14,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-14 03:53:14,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-14 03:53:14,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-14 03:53:14,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-14 03:53:14,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-14 03:53:14,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-14 03:53:14,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-14 03:53:14,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-14 03:53:14,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-14 03:53:14,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-14 03:53:14,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-14 03:53:14,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-14 03:53:14,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-14 03:53:14,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-14 03:53:14,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-14 03:53:14,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-14 03:53:14,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-14 03:53:14,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-14 03:53:14,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-14 03:53:14,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-14 03:53:14,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-14 03:53:14,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-14 03:53:14,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-14 03:53:14,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-14 03:53:14,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-14 03:53:14,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-14 03:53:14,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-14 03:53:14,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-14 03:53:14,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-14 03:53:14,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-14 03:53:14,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-14 03:53:14,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-14 03:53:14,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-14 03:53:14,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-14 03:53:14,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-14 03:53:14,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-14 03:53:14,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-14 03:53:14,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-14 03:53:14,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-14 03:53:14,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-14 03:53:14,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-14 03:53:14,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-14 03:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-14 03:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-14 03:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-14 03:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-14 03:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-14 03:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-14 03:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-14 03:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-14 03:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-14 03:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-14 03:53:14,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-14 03:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-14 03:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-14 03:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-14 03:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-14 03:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-14 03:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-14 03:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-14 03:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-14 03:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-14 03:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-14 03:53:14,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-14 03:53:14,261 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023516198618315033); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035448069856702444, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023516198618315033); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0435, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-14 03:53:14,261 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-14 03:53:14,261 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013116773660216035, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023516198618315033); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035448069856702444, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023516198618315033); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0435, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-14 03:53:14,544 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 381 ms to try 5000 different iterations. Found a solution that moves 964 regions; Going from a computed imbalance of 0.013116773660216035 to a new imbalance of 5.467405730680533E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006995268562901327); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007476112351500336); MoveCostFunction : (multiplier=7.0, imbalance=0.10711111111111112, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006995268562901327); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-14 03:53:14,547 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv20945049:81 , srv1200643094:82 , srv1030559665:83 , srv1169653468:83 , srv1352248510:83 , srv1435076252:83 , srv1498380972:83 , srv1785850819:83 , srv306662468:83 , srv454645002:83 , srv615172859:83 , srv707637093:83 , srv1088113130:84 , srv1192330838:84 , srv1220519161:84 , srv1347650974:84 , srv1508375145:84 , srv2123774274:84 , srv574905219:84 , srv64355067:84 , srv717929960:84 , srv758794738:84 , srv844078397:84 , srv88715082:84 , srv1145417734:85 , srv1181555740:85 , srv1193744900:85 , srv1330511390:85 , srv1438662176:85 , srv1519147984:85 , srv1525869397:85 , srv1617857967:85 , srv1705252710:85 , srv1895274985:85 , srv2033219794:85 , srv2091094945:85 , srv213162655:85 , srv250178053:85 , srv420548851:85 , srv462032142:85 , srv477753820:85 , srv519981914:85 , srv712627254:85 , srv871498846:85 , srv880688752:85 , srv901785353:85 , srv934444099:85 , srv946641578:85 , srv959059719:85 , srv998388383:85 , srv1138636656:86 , srv1139024944:86 , srv1221144344:86 , srv1229683173:86 , srv1253438788:86 , srv1261847163:86 , srv1300183793:86 , srv1385441644:86 , srv1439285646:86 , srv1594602753:86 , srv1701308557:86 , srv1798508303:86 , srv1810088722:86 , srv1879838022:86 , srv1896552389:86 , srv2037202269:86 , srv23091728:86 , srv265740922:86 , srv484785645:86 , srv632009843:86 , srv671542763:86 , srv727964583:86 , srv746279826:86 , srv857661769:86 , srv86559302:86 , srv969269787:86 , srv1018303395:87 , srv124516735:87 , srv1292189528:87 , srv1296764054:87 , srv1329859992:87 , srv135944097:87 , srv147408003:87 , srv1818135584:87 , srv1946473552:87 , srv287849855:87 , srv751139417:87 , srv1118335473:88 , srv112754407:88 , srv1298357617:88 , srv1454621518:88 , srv1718254437:88 , srv247783445:88 , srv388810147:88 , srv649533188:88 , srv984100583:88 , srv1024989881:89 , srv1596645681:89 , srv479583554:89 , srv784262362:537 } 2023-07-14 03:53:14,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-14 03:53:14,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1810088722=47, srv1200643094=13, srv1261847163=19, srv1519147984=37, srv1292189528=20, srv2123774274=57, srv23091728=59, srv1718254437=44, srv1330511390=25, srv477753820=69, srv2033219794=53, srv649533188=77, srv707637093=79, srv484785645=71, srv1895274985=50, srv1145417734=8, srv1946473552=52, srv751139417=84, srv2091094945=55, srv1300183793=23, srv1296764054=21, srv1798508303=46, srv519981914=72, srv1138636656=6, srv1221144344=15, srv250178053=61, srv1253438788=18, srv969269787=97, srv420548851=66, srv135944097=28, srv112754407=5, srv1818135584=48, srv984100583=98, srv1139024944=7, srv998388383=99, srv632009843=75, srv1454621518=33, srv1024989881=1, srv247783445=60, srv64355067=76, srv746279826=83, srv844078397=87, srv1229683173=16, srv1298357617=22, srv479583554=70, srv727964583=82, srv784262362=86, srv574905219=73, srv1169653468=9, srv880688752=91, srv901785353=93, srv147408003=34, srv959059719=96, srv1181555740=10, srv1439285646=32, srv671542763=78, srv1352248510=27, srv1617857967=41, srv857661769=88, srv712627254=80, srv1118335473=4, srv1220519161=14, srv1596645681=40, srv1508375145=36, srv615172859=74, srv388810147=65, srv213162655=58, srv946641578=95, srv306662468=64, srv124516735=17, srv1498380972=35, srv1193744900=12, srv287849855=63, srv758794738=85, srv1705252710=43, srv1896552389=51, srv934444099=94, srv265740922=62, srv1385441644=29, srv1438662176=31, srv717929960=81, srv871498846=90, srv1785850819=45, srv88715082=92, srv1018303395=0, srv20945049=56, srv1347650974=26, srv1030559665=2, srv1192330838=11, srv1701308557=42, srv462032142=68, srv1329859992=24, srv454645002=67, srv2037202269=54, srv86559302=89, srv1525869397=38, srv1594602753=39, srv1879838022=49, srv1435076252=30, srv1088113130=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-14 03:53:14,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-14 03:53:14,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-14 03:53:14,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-14 03:53:14,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-14 03:53:14,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-14 03:53:14,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-14 03:53:14,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-14 03:53:14,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-14 03:53:14,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-14 03:53:14,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-14 03:53:14,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-14 03:53:14,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-14 03:53:14,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-14 03:53:14,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-14 03:53:14,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-14 03:53:14,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-14 03:53:14,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-14 03:53:14,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-14 03:53:14,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-14 03:53:14,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-14 03:53:14,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-14 03:53:14,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-14 03:53:14,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-14 03:53:14,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-14 03:53:14,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-14 03:53:14,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-14 03:53:14,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-14 03:53:14,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-14 03:53:14,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-14 03:53:14,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-14 03:53:14,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-14 03:53:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-14 03:53:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-14 03:53:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-14 03:53:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-14 03:53:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-14 03:53:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-14 03:53:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-14 03:53:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-14 03:53:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-14 03:53:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-14 03:53:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-14 03:53:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-14 03:53:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-14 03:53:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-14 03:53:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-14 03:53:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-14 03:53:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-14 03:53:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-14 03:53:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-14 03:53:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-14 03:53:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-14 03:53:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-14 03:53:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-14 03:53:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-14 03:53:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-14 03:53:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-14 03:53:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-14 03:53:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-14 03:53:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-14 03:53:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-14 03:53:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-14 03:53:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-14 03:53:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-14 03:53:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-14 03:53:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-14 03:53:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-14 03:53:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-14 03:53:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-14 03:53:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-14 03:53:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-14 03:53:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-14 03:53:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-14 03:53:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-14 03:53:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-14 03:53:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-14 03:53:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-14 03:53:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-14 03:53:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-14 03:53:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-14 03:53:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-14 03:53:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-14 03:53:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-14 03:53:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-14 03:53:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-14 03:53:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-14 03:53:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-14 03:53:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-14 03:53:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-14 03:53:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-14 03:53:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-14 03:53:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-14 03:53:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-14 03:53:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-14 03:53:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-14 03:53:14,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-14 03:53:14,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-14 03:53:14,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-14 03:53:14,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-14 03:53:14,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-14 03:53:14,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-14 03:53:14,692 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006995268562901327); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007476112351500336); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006995268562901327); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-14 03:53:14,692 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-14 03:53:14,692 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.399905007958658E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006995268562901327); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007476112351500336); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006995268562901327); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-14 03:53:14,957 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 364 ms to try 5000 different iterations. Found a solution that moves 237 regions; Going from a computed imbalance of 5.399905007958658E-4 to a new imbalance of 2.6507008922639875E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.026333333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-14 03:53:14,958 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1454621518:82 , srv247783445:83 , srv287849855:83 , srv1018303395:84 , srv1024989881:84 , srv112754407:84 , srv1139024944:84 , srv1145417734:84 , srv1169653468:84 , srv1181555740:84 , srv124516735:84 , srv1253438788:84 , srv1292189528:84 , srv1298357617:84 , srv1329859992:84 , srv1385441644:84 , srv1435076252:84 , srv1439285646:84 , srv147408003:84 , srv1596645681:84 , srv1617857967:84 , srv1785850819:84 , srv1810088722:84 , srv1946473552:84 , srv20945049:84 , srv213162655:84 , srv250178053:84 , srv265740922:84 , srv388810147:84 , srv462032142:84 , srv477753820:84 , srv479583554:84 , srv649533188:84 , srv751139417:84 , srv758794738:84 , srv844078397:84 , srv857661769:84 , srv86559302:84 , srv88715082:84 , srv934444099:84 , srv998388383:84 , srv1030559665:85 , srv1088113130:85 , srv1118335473:85 , srv1138636656:85 , srv1192330838:85 , srv1193744900:85 , srv1200643094:85 , srv1220519161:85 , srv1229683173:85 , srv1261847163:85 , srv1296764054:85 , srv1300183793:85 , srv1330511390:85 , srv1347650974:85 , srv1352248510:85 , srv135944097:85 , srv1438662176:85 , srv1508375145:85 , srv1519147984:85 , srv1525869397:85 , srv1594602753:85 , srv1701308557:85 , srv1718254437:85 , srv1798508303:85 , srv1818135584:85 , srv1895274985:85 , srv1896552389:85 , srv2033219794:85 , srv2037202269:85 , srv2091094945:85 , srv23091728:85 , srv306662468:85 , srv420548851:85 , srv454645002:85 , srv484785645:85 , srv519981914:85 , srv615172859:85 , srv632009843:85 , srv64355067:85 , srv671542763:85 , srv707637093:85 , srv712627254:85 , srv717929960:85 , srv727964583:85 , srv746279826:85 , srv871498846:85 , srv901785353:85 , srv946641578:85 , srv959059719:85 , srv984100583:85 , srv1221144344:86 , srv1498380972:86 , srv1705252710:86 , srv1879838022:86 , srv2123774274:86 , srv574905219:86 , srv880688752:86 , srv969269787:86 , srv784262362:622 } 2023-07-14 03:53:14,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-14 03:53:14,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1810088722=47, srv1200643094=13, srv1261847163=19, srv1519147984=37, srv1292189528=20, srv2123774274=57, srv23091728=59, srv1718254437=44, srv1330511390=25, srv477753820=69, srv2033219794=53, srv649533188=77, srv707637093=79, srv484785645=71, srv1895274985=50, srv1145417734=8, srv1946473552=52, srv751139417=84, srv2091094945=55, srv1300183793=23, srv1296764054=21, srv1798508303=46, srv519981914=72, srv1138636656=6, srv1221144344=15, srv250178053=61, srv1253438788=18, srv969269787=97, srv420548851=66, srv135944097=28, srv112754407=5, srv1818135584=48, srv984100583=98, srv1139024944=7, srv998388383=99, srv632009843=75, srv1454621518=33, srv1024989881=1, srv247783445=60, srv64355067=76, srv746279826=83, srv844078397=87, srv1229683173=16, srv1298357617=22, srv479583554=70, srv727964583=82, srv784262362=86, srv574905219=73, srv1169653468=9, srv880688752=91, srv901785353=93, srv147408003=34, srv959059719=96, srv1181555740=10, srv1439285646=32, srv671542763=78, srv1352248510=27, srv1617857967=41, srv857661769=88, srv712627254=80, srv1118335473=4, srv1220519161=14, srv1596645681=40, srv1508375145=36, srv615172859=74, srv388810147=65, srv213162655=58, srv946641578=95, srv306662468=64, srv124516735=17, srv1498380972=35, srv1193744900=12, srv287849855=63, srv758794738=85, srv1705252710=43, srv1896552389=51, srv934444099=94, srv265740922=62, srv1385441644=29, srv1438662176=31, srv717929960=81, srv871498846=90, srv1785850819=45, srv88715082=92, srv1018303395=0, srv20945049=56, srv1347650974=26, srv1030559665=2, srv1192330838=11, srv1701308557=42, srv462032142=68, srv1329859992=24, srv454645002=67, srv2037202269=54, srv86559302=89, srv1525869397=38, srv1594602753=39, srv1879838022=49, srv1435076252=30, srv1088113130=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-14 03:53:14,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-14 03:53:14,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-14 03:53:14,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-14 03:53:14,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-14 03:53:14,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-14 03:53:14,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-14 03:53:14,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-14 03:53:14,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-14 03:53:14,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-14 03:53:14,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-14 03:53:14,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-14 03:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-14 03:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-14 03:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-14 03:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-14 03:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-14 03:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-14 03:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-14 03:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-14 03:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-14 03:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-14 03:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-14 03:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-14 03:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-14 03:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-14 03:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-14 03:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-14 03:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-14 03:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-14 03:53:14,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-14 03:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-14 03:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-14 03:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-14 03:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-14 03:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-14 03:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-14 03:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-14 03:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-14 03:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-14 03:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-14 03:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-14 03:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-14 03:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-14 03:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-14 03:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-14 03:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-14 03:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-14 03:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-14 03:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-14 03:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-14 03:53:14,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-14 03:53:14,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-14 03:53:14,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-14 03:53:14,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-14 03:53:14,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-14 03:53:14,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-14 03:53:14,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-14 03:53:14,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-14 03:53:14,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-14 03:53:14,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-14 03:53:14,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-14 03:53:14,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-14 03:53:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-14 03:53:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-14 03:53:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-14 03:53:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-14 03:53:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-14 03:53:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-14 03:53:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-14 03:53:14,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-14 03:53:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-14 03:53:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-14 03:53:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-14 03:53:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-14 03:53:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-14 03:53:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-14 03:53:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-14 03:53:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-14 03:53:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-14 03:53:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-14 03:53:14,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-14 03:53:14,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-14 03:53:14,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-14 03:53:14,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-14 03:53:14,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-14 03:53:14,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-14 03:53:14,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-14 03:53:14,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-14 03:53:14,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-14 03:53:14,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-14 03:53:14,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-14 03:53:14,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-14 03:53:14,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-14 03:53:14,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-14 03:53:14,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-14 03:53:14,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-14 03:53:14,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-14 03:53:14,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-14 03:53:14,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-14 03:53:14,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-14 03:53:14,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-14 03:53:15,114 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-14 03:53:15,115 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-14 03:53:15,115 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.4847499453232764E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-14 03:53:15,360 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 371 ms to try 5000 different iterations. Found a solution that moves 14 regions; Going from a computed imbalance of 2.4847499453232764E-5 to a new imbalance of 8.077720212279826E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0015555555555555555); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-14 03:53:15,360 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1193744900:83 , srv1221144344:83 , srv1879838022:83 , srv2123774274:83 , srv574905219:83 , srv969269787:83 , srv1018303395:84 , srv1024989881:84 , srv1030559665:84 , srv1088113130:84 , srv1118335473:84 , srv112754407:84 , srv1138636656:84 , srv1139024944:84 , srv1145417734:84 , srv1169653468:84 , srv1181555740:84 , srv1192330838:84 , srv1200643094:84 , srv1220519161:84 , srv1229683173:84 , srv124516735:84 , srv1261847163:84 , srv1292189528:84 , srv1296764054:84 , srv1298357617:84 , srv1300183793:84 , srv1329859992:84 , srv1330511390:84 , srv1347650974:84 , srv1352248510:84 , srv135944097:84 , srv1385441644:84 , srv1438662176:84 , srv1439285646:84 , srv1454621518:84 , srv147408003:84 , srv1498380972:84 , srv1508375145:84 , srv1519147984:84 , srv1525869397:84 , srv1594602753:84 , srv1596645681:84 , srv1617857967:84 , srv1701308557:84 , srv1705252710:84 , srv1718254437:84 , srv1785850819:84 , srv1798508303:84 , srv1810088722:84 , srv1818135584:84 , srv1895274985:84 , srv1896552389:84 , srv1946473552:84 , srv2033219794:84 , srv2091094945:84 , srv20945049:84 , srv213162655:84 , srv23091728:84 , srv247783445:84 , srv250178053:84 , srv287849855:84 , srv306662468:84 , srv388810147:84 , srv420548851:84 , srv454645002:84 , srv462032142:84 , srv477753820:84 , srv479583554:84 , srv484785645:84 , srv519981914:84 , srv615172859:84 , srv632009843:84 , srv64355067:84 , srv649533188:84 , srv671542763:84 , srv707637093:84 , srv712627254:84 , srv717929960:84 , srv727964583:84 , srv746279826:84 , srv751139417:84 , srv758794738:84 , srv844078397:84 , srv857661769:84 , srv86559302:84 , srv871498846:84 , srv880688752:84 , srv88715082:84 , srv901785353:84 , srv946641578:84 , srv959059719:84 , srv984100583:84 , srv1253438788:85 , srv1435076252:85 , srv2037202269:85 , srv265740922:85 , srv934444099:85 , srv998388383:85 , srv784262362:684 } 2023-07-14 03:53:15,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-14 03:53:15,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1810088722=47, srv1200643094=13, srv1261847163=19, srv1519147984=37, srv1292189528=20, srv2123774274=57, srv23091728=59, srv1718254437=44, srv1330511390=25, srv477753820=69, srv2033219794=53, srv649533188=77, srv707637093=79, srv484785645=71, srv1895274985=50, srv1145417734=8, srv1946473552=52, srv751139417=84, srv2091094945=55, srv1300183793=23, srv1296764054=21, srv1798508303=46, srv519981914=72, srv1138636656=6, srv1221144344=15, srv250178053=61, srv1253438788=18, srv969269787=97, srv420548851=66, srv135944097=28, srv112754407=5, srv1818135584=48, srv984100583=98, srv1139024944=7, srv998388383=99, srv632009843=75, srv1454621518=33, srv1024989881=1, srv247783445=60, srv64355067=76, srv746279826=83, srv844078397=87, srv1229683173=16, srv1298357617=22, srv479583554=70, srv727964583=82, srv784262362=86, srv574905219=73, srv1169653468=9, srv880688752=91, srv901785353=93, srv147408003=34, srv959059719=96, srv1181555740=10, srv1439285646=32, srv671542763=78, srv1352248510=27, srv1617857967=41, srv857661769=88, srv712627254=80, srv1118335473=4, srv1220519161=14, srv1596645681=40, srv1508375145=36, srv615172859=74, srv388810147=65, srv213162655=58, srv946641578=95, srv306662468=64, srv124516735=17, srv1498380972=35, srv1193744900=12, srv287849855=63, srv758794738=85, srv1705252710=43, srv1896552389=51, srv934444099=94, srv265740922=62, srv1385441644=29, srv1438662176=31, srv717929960=81, srv871498846=90, srv1785850819=45, srv88715082=92, srv1018303395=0, srv20945049=56, srv1347650974=26, srv1030559665=2, srv1192330838=11, srv1701308557=42, srv462032142=68, srv1329859992=24, srv454645002=67, srv2037202269=54, srv86559302=89, srv1525869397=38, srv1594602753=39, srv1879838022=49, srv1435076252=30, srv1088113130=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-14 03:53:15,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-14 03:53:15,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-14 03:53:15,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-14 03:53:15,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-14 03:53:15,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-14 03:53:15,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-14 03:53:15,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-14 03:53:15,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-14 03:53:15,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-14 03:53:15,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-14 03:53:15,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-14 03:53:15,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-14 03:53:15,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-14 03:53:15,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-14 03:53:15,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-14 03:53:15,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-14 03:53:15,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-14 03:53:15,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-14 03:53:15,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-14 03:53:15,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-14 03:53:15,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-14 03:53:15,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-14 03:53:15,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-14 03:53:15,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-14 03:53:15,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-14 03:53:15,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-14 03:53:15,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-14 03:53:15,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-14 03:53:15,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-14 03:53:15,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-14 03:53:15,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-14 03:53:15,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-14 03:53:15,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-14 03:53:15,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-14 03:53:15,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-14 03:53:15,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-14 03:53:15,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-14 03:53:15,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-14 03:53:15,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-14 03:53:15,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-14 03:53:15,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-14 03:53:15,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-14 03:53:15,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-14 03:53:15,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-14 03:53:15,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-14 03:53:15,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-14 03:53:15,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-14 03:53:15,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-14 03:53:15,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-14 03:53:15,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-14 03:53:15,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-14 03:53:15,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-14 03:53:15,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-14 03:53:15,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-14 03:53:15,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-14 03:53:15,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-14 03:53:15,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-14 03:53:15,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-14 03:53:15,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-14 03:53:15,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-14 03:53:15,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-14 03:53:15,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-14 03:53:15,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-14 03:53:15,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-14 03:53:15,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-14 03:53:15,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-14 03:53:15,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-14 03:53:15,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-14 03:53:15,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-14 03:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-14 03:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-14 03:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-14 03:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-14 03:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-14 03:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-14 03:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-14 03:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-14 03:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-14 03:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-14 03:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-14 03:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-14 03:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-14 03:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-14 03:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-14 03:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-14 03:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-14 03:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-14 03:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-14 03:53:15,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-14 03:53:15,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-14 03:53:15,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-14 03:53:15,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-14 03:53:15,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-14 03:53:15,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-14 03:53:15,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-14 03:53:15,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-14 03:53:15,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-14 03:53:15,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-14 03:53:15,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-14 03:53:15,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-14 03:53:15,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-14 03:53:15,501 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.979690117040589E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-14 03:53:15,502 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1193744900:83 , srv1221144344:83 , srv1879838022:83 , srv2123774274:83 , srv574905219:83 , srv969269787:83 , srv1018303395:84 , srv1024989881:84 , srv1030559665:84 , srv1088113130:84 , srv1118335473:84 , srv112754407:84 , srv1138636656:84 , srv1139024944:84 , srv1145417734:84 , srv1169653468:84 , srv1181555740:84 , srv1192330838:84 , srv1200643094:84 , srv1220519161:84 , srv1229683173:84 , srv124516735:84 , srv1261847163:84 , srv1292189528:84 , srv1296764054:84 , srv1298357617:84 , srv1300183793:84 , srv1329859992:84 , srv1330511390:84 , srv1347650974:84 , srv1352248510:84 , srv135944097:84 , srv1385441644:84 , srv1438662176:84 , srv1439285646:84 , srv1454621518:84 , srv147408003:84 , srv1498380972:84 , srv1508375145:84 , srv1519147984:84 , srv1525869397:84 , srv1594602753:84 , srv1596645681:84 , srv1617857967:84 , srv1701308557:84 , srv1705252710:84 , srv1718254437:84 , srv1785850819:84 , srv1798508303:84 , srv1810088722:84 , srv1818135584:84 , srv1895274985:84 , srv1896552389:84 , srv1946473552:84 , srv2033219794:84 , srv2091094945:84 , srv20945049:84 , srv213162655:84 , srv23091728:84 , srv247783445:84 , srv250178053:84 , srv287849855:84 , srv306662468:84 , srv388810147:84 , srv420548851:84 , srv454645002:84 , srv462032142:84 , srv477753820:84 , srv479583554:84 , srv484785645:84 , srv519981914:84 , srv615172859:84 , srv632009843:84 , srv64355067:84 , srv649533188:84 , srv671542763:84 , srv707637093:84 , srv712627254:84 , srv717929960:84 , srv727964583:84 , srv746279826:84 , srv751139417:84 , srv758794738:84 , srv844078397:84 , srv857661769:84 , srv86559302:84 , srv871498846:84 , srv880688752:84 , srv88715082:84 , srv901785353:84 , srv946641578:84 , srv959059719:84 , srv984100583:84 , srv1253438788:85 , srv1435076252:85 , srv2037202269:85 , srv265740922:85 , srv934444099:85 , srv998388383:85 , srv784262362:684 } 2023-07-14 03:53:15,536 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=350 (was 350), ProcessCount=172 (was 172), AvailableMemoryMB=7887 (was 8692) 2023-07-14 03:53:15,545 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=350, ProcessCount=172, AvailableMemoryMB=7885 2023-07-14 03:53:15,545 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-14 03:53:15,546 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-07-14 03:53:15,556 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1026465674:84 , srv118433595:84 , srv1210219429:84 , srv1485374116:84 , srv1517923104:84 , srv152349875:84 , srv1544483491:84 , srv1602358494:84 , srv1614508097:84 , srv1615221966:84 , srv169612242:84 , srv1753780709:84 , srv1882459540:84 , srv1929519775:84 , srv1942458575:84 , srv2028057585:84 , srv2043035047:84 , srv2079222300:84 , srv391133:84 , srv441982237:84 , srv457020595:84 , srv468118393:84 , srv506582666:84 , srv526785020:84 , srv635093146:84 , srv712673819:84 , srv772840189:84 , srv85806854:84 , srv931993952:84 , srv742751418:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-07-14 03:53:15,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-14 03:53:15,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv742751418=26, srv1026465674=0, srv2079222300=17, srv712673819=25, srv2028057585=15, srv1753780709=11, srv1602358494=7, srv931993952=29, srv118433595=1, srv1882459540=12, srv1929519775=13, srv457020595=20, srv1942458575=14, srv506582666=22, srv169612242=10, srv1544483491=6, srv1614508097=8, srv526785020=23, srv1615221966=9, srv1210219429=2, srv2043035047=16, srv635093146=24, srv1485374116=3, srv1517923104=4, srv772840189=27, srv468118393=21, srv85806854=28, srv152349875=5, srv391133=18, srv441982237=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-14 03:53:15,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-14 03:53:15,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-14 03:53:15,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-14 03:53:15,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-14 03:53:15,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-14 03:53:15,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-14 03:53:15,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-14 03:53:15,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-14 03:53:15,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-14 03:53:15,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-14 03:53:15,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-14 03:53:15,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-14 03:53:15,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-14 03:53:15,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-14 03:53:15,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-14 03:53:15,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-14 03:53:15,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-14 03:53:15,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-14 03:53:15,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-14 03:53:15,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-14 03:53:15,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-14 03:53:15,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-14 03:53:15,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-14 03:53:15,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-14 03:53:15,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-14 03:53:15,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-14 03:53:15,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-14 03:53:15,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-14 03:53:15,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-14 03:53:15,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-14 03:53:15,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-07-14 03:53:15,595 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.083265438844763, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-14 03:53:15,596 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912455409943214, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.083265438844763, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-07-14 03:55:15,576 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6195673 different iterations. Found a solution that moves 1859 regions; Going from a computed imbalance of 0.9912455409943214 to a new imbalance of 4.338998739279625E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6885185185185185, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-14 03:55:15,581 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1026465674:90 , srv118433595:90 , srv1210219429:90 , srv1485374116:90 , srv1517923104:90 , srv152349875:90 , srv1544483491:90 , srv1602358494:90 , srv1614508097:90 , srv1615221966:90 , srv169612242:90 , srv1753780709:90 , srv1882459540:90 , srv1929519775:90 , srv1942458575:90 , srv2028057585:90 , srv2043035047:90 , srv2079222300:90 , srv391133:90 , srv441982237:90 , srv457020595:90 , srv468118393:90 , srv506582666:90 , srv526785020:90 , srv635093146:90 , srv712673819:90 , srv742751418:90 , srv772840189:90 , srv85806854:90 , srv931993952:90 } 2023-07-14 03:55:15,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-14 03:55:15,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv742751418=26, srv1026465674=0, srv2079222300=17, srv712673819=25, srv2028057585=15, srv1753780709=11, srv1602358494=7, srv931993952=29, srv118433595=1, srv1882459540=12, srv1929519775=13, srv457020595=20, srv1942458575=14, srv506582666=22, srv169612242=10, srv1544483491=6, srv1614508097=8, srv526785020=23, srv1615221966=9, srv1210219429=2, srv2043035047=16, srv635093146=24, srv1485374116=3, srv1517923104=4, srv772840189=27, srv468118393=21, srv85806854=28, srv152349875=5, srv391133=18, srv441982237=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-14 03:55:15,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-14 03:55:15,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-14 03:55:15,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-14 03:55:15,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-14 03:55:15,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-14 03:55:15,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-14 03:55:15,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-14 03:55:15,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-14 03:55:15,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-14 03:55:15,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-14 03:55:15,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-14 03:55:15,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-14 03:55:15,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-14 03:55:15,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-14 03:55:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-14 03:55:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-14 03:55:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-14 03:55:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-14 03:55:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-14 03:55:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-14 03:55:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-14 03:55:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-14 03:55:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-14 03:55:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-14 03:55:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-14 03:55:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-14 03:55:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-14 03:55:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-14 03:55:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-14 03:55:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-14 03:55:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-07-14 03:55:15,599 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-14 03:55:15,599 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1026465674:90 , srv118433595:90 , srv1210219429:90 , srv1485374116:90 , srv1517923104:90 , srv152349875:90 , srv1544483491:90 , srv1602358494:90 , srv1614508097:90 , srv1615221966:90 , srv169612242:90 , srv1753780709:90 , srv1882459540:90 , srv1929519775:90 , srv1942458575:90 , srv2028057585:90 , srv2043035047:90 , srv2079222300:90 , srv391133:90 , srv441982237:90 , srv457020595:90 , srv468118393:90 , srv506582666:90 , srv526785020:90 , srv635093146:90 , srv712673819:90 , srv742751418:90 , srv772840189:90 , srv85806854:90 , srv931993952:90 } 2023-07-14 03:55:15,611 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=235 (was 350), ProcessCount=170 (was 172), AvailableMemoryMB=7904 (was 7885) - AvailableMemoryMB LEAK? -