2024-01-07 09:52:43,111 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-01-07 09:52:43,386 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-01-07 09:52:43,494 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-01-07 09:52:43,549 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-07 09:52:43,555 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-01-07 09:52:43,568 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=225, ProcessCount=171, AvailableMemoryMB=8956
2024-01-07 09:52:43,571 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-07 09:52:43,571 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-01-07 09:52:43,672 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1026773405:84 , srv1030692155:84 , srv1037481150:84 , srv1040952143:84 , srv1043676890:84 , srv1097437432:84 , srv1180558809:84 , srv1236688994:84 , srv1259352198:84 , srv1293690230:84 , srv1304260895:84 , srv1305103776:84 , srv1333393766:84 , srv1339276499:84 , srv1365988966:84 , srv1385800727:84 , srv1400868084:84 , srv1442840318:84 , srv1446132686:84 , srv1447389035:84 , srv1453490985:84 , srv148237400:84 , srv1491402007:84 , srv1552276371:84 , srv1622424132:84 , srv1649148498:84 , srv1658232959:84 , srv1659181806:84 , srv1699005953:84 , srv1759814693:84 , srv1779545126:84 , srv178957770:84 , srv1790706073:84 , srv182576142:84 , srv1882731567:84 , srv1914808462:84 , srv1916979175:84 , srv191733607:84 , srv1918508905:84 , srv1924568106:84 , srv1929281322:84 , srv194936169:84 , srv195399547:84 , srv1961078525:84 , srv1975539928:84 , srv1986011769:84 , srv199547396:84 , srv2026421512:84 , srv2081461023:84 , srv2141359456:84 , srv2144584551:84 , srv247790537:84 , srv257132246:84 , srv292310705:84 , srv295912660:84 , srv300848463:84 , srv336310221:84 , srv351607088:84 , srv35277849:84 , srv360015690:84 , srv360347817:84 , srv367653964:84 , srv414668918:84 , srv423255805:84 , srv42734397:84 , srv443264422:84 , srv458956559:84 , srv475271309:84 , srv501055142:84 , srv521655284:84 , srv524140729:84 , srv543283116:84 , srv553224078:84 , srv570975144:84 , srv572345063:84 , srv574726844:84 , srv589407294:84 , srv600736271:84 , srv642118181:84 , srv66985806:84 , srv671060865:84 , srv690593055:84 , srv707264884:84 , srv710008515:84 , srv716758768:84 , srv760910455:84 , srv780850396:84 , srv782058838:84 , srv810496251:84 , srv812359989:84 , srv826099201:84 , srv832916311:84 , srv846663724:84 , srv859688762:84 , srv888472054:84 , srv914449457:84 , srv969747086:84 , srv971269441:84 , srv975461612:84 , srv1190893350:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-01-07 09:52:43,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-07 09:52:43,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv336310221=57, srv2144584551=51, srv443264422=66, srv1190893350=7, srv1043676890=4, srv475271309=68, srv971269441=98, srv1400868084=17, srv148237400=22, srv501055142=69, srv671060865=81, srv1961078525=44, srv1914808462=36, srv859688762=94, srv1442840318=18, srv360015690=60, srv35277849=59, srv191733607=38, srv199547396=47, srv360347817=61, srv1293690230=10, srv2141359456=50, srv367653964=62, srv66985806=80, srv1259352198=9, srv1365988966=15, srv1333393766=13, srv247790537=52, srv1699005953=29, srv1385800727=16, srv257132246=53, srv543283116=72, srv1916979175=37, srv194936169=42, srv1026773405=0, srv521655284=70, srv1097437432=5, srv888472054=95, srv846663724=93, srv1918508905=39, srv1037481150=2, srv2081461023=49, srv600736271=78, srv553224078=73, srv1453490985=21, srv178957770=32, srv1779545126=31, srv812359989=90, srv195399547=43, srv1180558809=6, srv1236688994=8, srv458956559=67, srv1040952143=3, srv975461612=99, srv1659181806=28, srv1790706073=33, srv1649148498=26, srv832916311=92, srv1491402007=23, srv1304260895=11, srv1658232959=27, srv589407294=77, srv351607088=58, srv690593055=82, srv780850396=87, srv1339276499=14, srv572345063=75, srv182576142=34, srv914449457=96, srv1305103776=12, srv1622424132=25, srv2026421512=48, srv1975539928=45, srv414668918=63, srv969747086=97, srv716758768=85, srv292310705=54, srv300848463=56, srv574726844=76, srv42734397=65, srv423255805=64, srv782058838=88, srv826099201=91, srv642118181=79, srv1929281322=41, srv1446132686=19, srv524140729=71, srv570975144=74, srv810496251=89, srv707264884=83, srv1552276371=24, srv1924568106=40, srv295912660=55, srv1759814693=30, srv1986011769=46, srv1030692155=1, srv1447389035=20, srv1882731567=35, srv760910455=86, srv710008515=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-07 09:52:43,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-07 09:52:43,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-07 09:52:43,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-07 09:52:43,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-07 09:52:43,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-07 09:52:43,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-07 09:52:43,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-07 09:52:43,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-07 09:52:43,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-07 09:52:43,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-07 09:52:43,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-07 09:52:43,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-07 09:52:43,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-07 09:52:43,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-07 09:52:43,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-07 09:52:43,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-07 09:52:43,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-07 09:52:43,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-07 09:52:43,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-07 09:52:43,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-07 09:52:43,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-07 09:52:43,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-07 09:52:43,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-07 09:52:43,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-07 09:52:43,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-07 09:52:43,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-07 09:52:43,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-07 09:52:43,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-07 09:52:43,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-07 09:52:43,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-07 09:52:43,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-07 09:52:43,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-07 09:52:43,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-07 09:52:43,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-07 09:52:43,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-07 09:52:43,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-07 09:52:43,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-07 09:52:43,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-07 09:52:43,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-07 09:52:43,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-07 09:52:43,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-07 09:52:43,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-07 09:52:43,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-07 09:52:43,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-07 09:52:43,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-07 09:52:43,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-07 09:52:43,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-07 09:52:43,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-07 09:52:43,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-07 09:52:43,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-07 09:52:43,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-07 09:52:43,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-07 09:52:43,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-07 09:52:43,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-07 09:52:43,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-07 09:52:43,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-07 09:52:43,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-07 09:52:43,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-07 09:52:43,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-07 09:52:43,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-07 09:52:43,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-07 09:52:43,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-07 09:52:43,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-07 09:52:43,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-07 09:52:43,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-07 09:52:43,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-07 09:52:43,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-07 09:52:43,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-07 09:52:43,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-07 09:52:43,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-07 09:52:43,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-07 09:52:43,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-07 09:52:43,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-07 09:52:43,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-07 09:52:43,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-07 09:52:43,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-07 09:52:43,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-07 09:52:43,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-07 09:52:43,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-07 09:52:43,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-07 09:52:43,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-07 09:52:43,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-07 09:52:43,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-07 09:52:43,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-07 09:52:43,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-07 09:52:43,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-07 09:52:43,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-07 09:52:43,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-07 09:52:43,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-07 09:52:43,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-07 09:52:43,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-07 09:52:43,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-07 09:52:43,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-07 09:52:43,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-07 09:52:43,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-07 09:52:43,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-07 09:52:43,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-07 09:52:43,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-07 09:52:43,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-07 09:52:43,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-07 09:52:43,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-07 09:52:44,088 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 09:52:44,089 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-07 09:52:44,089 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-07 09:52:44,424 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 595 ms to try 5000 different iterations.  Found a solution that moves 4678 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18101362977626012. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04037176039370149, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05409390347180594, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5197777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04037176039370149, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17616666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 09:52:44,445 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv971269441:81 , srv182576142:82 , srv35277849:82 , srv1333393766:83 , srv1442840318:83 , srv1447389035:83 , srv1882731567:83 , srv247790537:83 , srv367653964:83 , srv589407294:83 , srv1305103776:84 , srv1649148498:84 , srv1759814693:84 , srv1779545126:84 , srv501055142:84 , srv574726844:84 , srv716758768:84 , srv846663724:84 , srv975461612:84 , srv1040952143:85 , srv1365988966:85 , srv1914808462:85 , srv1929281322:85 , srv295912660:85 , srv336310221:85 , srv360347817:85 , srv642118181:85 , srv671060865:85 , srv1030692155:86 , srv1043676890:86 , srv1293690230:86 , srv1385800727:86 , srv1400868084:86 , srv148237400:86 , srv1491402007:86 , srv1659181806:86 , srv191733607:86 , srv1924568106:86 , srv194936169:86 , srv195399547:86 , srv199547396:86 , srv2026421512:86 , srv2081461023:86 , srv2141359456:86 , srv257132246:86 , srv292310705:86 , srv351607088:86 , srv42734397:86 , srv443264422:86 , srv458956559:86 , srv475271309:86 , srv521655284:86 , srv572345063:86 , srv780850396:86 , srv782058838:86 , srv810496251:86 , srv812359989:86 , srv859688762:86 , srv1304260895:87 , srv1339276499:87 , srv1552276371:87 , srv1699005953:87 , srv178957770:87 , srv1961078525:87 , srv1986011769:87 , srv300848463:87 , srv360015690:87 , srv414668918:87 , srv524140729:87 , srv543283116:87 , srv707264884:87 , srv710008515:87 , srv760910455:87 , srv832916311:87 , srv888472054:87 , srv969747086:87 , srv1037481150:88 , srv1180558809:88 , srv1236688994:88 , srv1259352198:88 , srv1622424132:88 , srv1790706073:88 , srv423255805:88 , srv553224078:88 , srv66985806:88 , srv1453490985:89 , srv1658232959:89 , srv1975539928:89 , srv600736271:89 , srv690593055:89 , srv826099201:89 , srv1026773405:90 , srv570975144:90 , srv914449457:90 , srv2144584551:91 , srv1446132686:92 , srv1918508905:92 , srv1097437432:93 , srv1916979175:94 , srv1190893350:449 }
2024-01-07 09:52:44,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-07 09:52:44,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv336310221=57, srv2144584551=51, srv443264422=66, srv1190893350=7, srv1043676890=4, srv475271309=68, srv971269441=98, srv1400868084=17, srv148237400=22, srv501055142=69, srv671060865=81, srv1961078525=44, srv1914808462=36, srv859688762=94, srv1442840318=18, srv360015690=60, srv35277849=59, srv191733607=38, srv199547396=47, srv360347817=61, srv1293690230=10, srv2141359456=50, srv367653964=62, srv66985806=80, srv1259352198=9, srv1365988966=15, srv1333393766=13, srv247790537=52, srv1699005953=29, srv1385800727=16, srv257132246=53, srv543283116=72, srv1916979175=37, srv194936169=42, srv1026773405=0, srv521655284=70, srv1097437432=5, srv888472054=95, srv846663724=93, srv1918508905=39, srv1037481150=2, srv2081461023=49, srv600736271=78, srv553224078=73, srv1453490985=21, srv178957770=32, srv1779545126=31, srv812359989=90, srv195399547=43, srv1180558809=6, srv1236688994=8, srv458956559=67, srv1040952143=3, srv975461612=99, srv1659181806=28, srv1790706073=33, srv1649148498=26, srv832916311=92, srv1491402007=23, srv1304260895=11, srv1658232959=27, srv589407294=77, srv351607088=58, srv690593055=82, srv780850396=87, srv1339276499=14, srv572345063=75, srv182576142=34, srv914449457=96, srv1305103776=12, srv1622424132=25, srv2026421512=48, srv1975539928=45, srv414668918=63, srv969747086=97, srv716758768=85, srv292310705=54, srv300848463=56, srv574726844=76, srv42734397=65, srv423255805=64, srv782058838=88, srv826099201=91, srv642118181=79, srv1929281322=41, srv1446132686=19, srv524140729=71, srv570975144=74, srv810496251=89, srv707264884=83, srv1552276371=24, srv1924568106=40, srv295912660=55, srv1759814693=30, srv1986011769=46, srv1030692155=1, srv1447389035=20, srv1882731567=35, srv760910455=86, srv710008515=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-07 09:52:44,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-07 09:52:44,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-07 09:52:44,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-07 09:52:44,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-07 09:52:44,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-07 09:52:44,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-07 09:52:44,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-07 09:52:44,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-07 09:52:44,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-07 09:52:44,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-07 09:52:44,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-07 09:52:44,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-07 09:52:44,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-07 09:52:44,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-07 09:52:44,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-07 09:52:44,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-07 09:52:44,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-07 09:52:44,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-07 09:52:44,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-07 09:52:44,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-07 09:52:44,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-07 09:52:44,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-07 09:52:44,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-07 09:52:44,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-07 09:52:44,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-07 09:52:44,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-07 09:52:44,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-07 09:52:44,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-07 09:52:44,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-07 09:52:44,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-07 09:52:44,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-07 09:52:44,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-07 09:52:44,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-07 09:52:44,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-07 09:52:44,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-07 09:52:44,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-07 09:52:44,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-07 09:52:44,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-07 09:52:44,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-07 09:52:44,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-07 09:52:44,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-07 09:52:44,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-07 09:52:44,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-07 09:52:44,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-07 09:52:44,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-07 09:52:44,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-07 09:52:44,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-07 09:52:44,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-07 09:52:44,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-07 09:52:44,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-07 09:52:44,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-07 09:52:44,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-07 09:52:44,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-07 09:52:44,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-07 09:52:44,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-07 09:52:44,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-07 09:52:44,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-07 09:52:44,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-07 09:52:44,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-07 09:52:44,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-07 09:52:44,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-07 09:52:44,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-07 09:52:44,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-07 09:52:44,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-07 09:52:44,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-07 09:52:44,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-07 09:52:44,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-07 09:52:44,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-07 09:52:44,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-07 09:52:44,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-07 09:52:44,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-07 09:52:44,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-07 09:52:44,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-07 09:52:44,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-07 09:52:44,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-07 09:52:44,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-07 09:52:44,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-07 09:52:44,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-07 09:52:44,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-07 09:52:44,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-07 09:52:44,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-07 09:52:44,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-07 09:52:44,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-07 09:52:44,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-07 09:52:44,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-07 09:52:44,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-07 09:52:44,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-07 09:52:44,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-07 09:52:44,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-07 09:52:44,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-07 09:52:44,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-07 09:52:44,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-07 09:52:44,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-07 09:52:44,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-07 09:52:44,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-07 09:52:44,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-07 09:52:44,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-07 09:52:44,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-07 09:52:44,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-07 09:52:44,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-07 09:52:44,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-07 09:52:44,608 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04037176039370149, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05409390347180594, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04037176039370149, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17616666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 09:52:44,608 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-07 09:52:44,608 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1809808737201509, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04037176039370149, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05409390347180594, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04037176039370149, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17616666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-07 09:52:44,767 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 265 ms to try 5000 different iterations.  Found a solution that moves 2680 regions; Going from a computed imbalance of 0.1809808737201509 to a new imbalance of 0.015401378925196734. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020942382016124392); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029900696102013423, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29777777777777775, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020942382016124392); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012416666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 09:52:44,779 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1446132686:79 , srv1916979175:80 , srv1918508905:80 , srv2144584551:80 , srv1097437432:81 , srv914449457:81 , srv1180558809:82 , srv1658232959:82 , srv570975144:82 , srv690593055:82 , srv826099201:82 , srv1026773405:83 , srv1037481150:83 , srv1453490985:83 , srv300848463:83 , srv524140729:83 , srv66985806:83 , srv707264884:83 , srv969747086:83 , srv1236688994:84 , srv1659181806:84 , srv1699005953:84 , srv178957770:84 , srv1924568106:84 , srv1961078525:84 , srv1975539928:84 , srv1986011769:84 , srv2081461023:84 , srv351607088:84 , srv414668918:84 , srv423255805:84 , srv553224078:84 , srv710008515:84 , srv760910455:84 , srv1040952143:85 , srv1304260895:85 , srv1622424132:85 , srv199547396:85 , srv2141359456:85 , srv257132246:85 , srv458956559:85 , srv782058838:85 , srv812359989:85 , srv1339276499:86 , srv1790706073:86 , srv1914808462:86 , srv1929281322:86 , srv292310705:86 , srv360015690:86 , srv360347817:86 , srv443264422:86 , srv475271309:86 , srv574726844:86 , srv600736271:86 , srv671060865:86 , srv780850396:86 , srv832916311:86 , srv888472054:86 , srv975461612:86 , srv1030692155:87 , srv1259352198:87 , srv1305103776:87 , srv1385800727:87 , srv1400868084:87 , srv1552276371:87 , srv194936169:87 , srv195399547:87 , srv2026421512:87 , srv42734397:87 , srv521655284:87 , srv810496251:87 , srv859688762:87 , srv1043676890:88 , srv1293690230:88 , srv148237400:88 , srv1491402007:88 , srv1649148498:88 , srv1779545126:88 , srv247790537:88 , srv336310221:88 , srv35277849:88 , srv543283116:88 , srv572345063:88 , srv589407294:88 , srv716758768:88 , srv1365988966:89 , srv1442840318:89 , srv1759814693:89 , srv191733607:89 , srv295912660:89 , srv846663724:89 , srv971269441:89 , srv182576142:90 , srv1882731567:90 , srv501055142:90 , srv1333393766:91 , srv1447389035:91 , srv367653964:91 , srv642118181:92 , srv1190893350:511 }
2024-01-07 09:52:44,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-07 09:52:44,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv336310221=57, srv2144584551=51, srv443264422=66, srv1190893350=7, srv1043676890=4, srv475271309=68, srv971269441=98, srv1400868084=17, srv148237400=22, srv501055142=69, srv671060865=81, srv1961078525=44, srv1914808462=36, srv859688762=94, srv1442840318=18, srv360015690=60, srv35277849=59, srv191733607=38, srv199547396=47, srv360347817=61, srv1293690230=10, srv2141359456=50, srv367653964=62, srv66985806=80, srv1259352198=9, srv1365988966=15, srv1333393766=13, srv247790537=52, srv1699005953=29, srv1385800727=16, srv257132246=53, srv543283116=72, srv1916979175=37, srv194936169=42, srv1026773405=0, srv521655284=70, srv1097437432=5, srv888472054=95, srv846663724=93, srv1918508905=39, srv1037481150=2, srv2081461023=49, srv600736271=78, srv553224078=73, srv1453490985=21, srv178957770=32, srv1779545126=31, srv812359989=90, srv195399547=43, srv1180558809=6, srv1236688994=8, srv458956559=67, srv1040952143=3, srv975461612=99, srv1659181806=28, srv1790706073=33, srv1649148498=26, srv832916311=92, srv1491402007=23, srv1304260895=11, srv1658232959=27, srv589407294=77, srv351607088=58, srv690593055=82, srv780850396=87, srv1339276499=14, srv572345063=75, srv182576142=34, srv914449457=96, srv1305103776=12, srv1622424132=25, srv2026421512=48, srv1975539928=45, srv414668918=63, srv969747086=97, srv716758768=85, srv292310705=54, srv300848463=56, srv574726844=76, srv42734397=65, srv423255805=64, srv782058838=88, srv826099201=91, srv642118181=79, srv1929281322=41, srv1446132686=19, srv524140729=71, srv570975144=74, srv810496251=89, srv707264884=83, srv1552276371=24, srv1924568106=40, srv295912660=55, srv1759814693=30, srv1986011769=46, srv1030692155=1, srv1447389035=20, srv1882731567=35, srv760910455=86, srv710008515=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-07 09:52:44,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-07 09:52:44,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-07 09:52:44,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-07 09:52:44,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-07 09:52:44,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-07 09:52:44,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-07 09:52:44,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-07 09:52:44,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-07 09:52:44,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-07 09:52:44,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-07 09:52:44,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-07 09:52:44,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-07 09:52:44,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-07 09:52:44,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-07 09:52:44,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-07 09:52:44,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-07 09:52:44,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-07 09:52:44,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-07 09:52:44,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-07 09:52:44,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-07 09:52:44,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-07 09:52:44,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-07 09:52:44,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-07 09:52:44,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-07 09:52:44,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-07 09:52:44,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-07 09:52:44,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-07 09:52:44,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-07 09:52:44,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-07 09:52:44,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-07 09:52:44,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-07 09:52:44,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-07 09:52:44,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-07 09:52:44,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-07 09:52:44,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-07 09:52:44,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-07 09:52:44,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-07 09:52:44,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-07 09:52:44,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-07 09:52:44,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-07 09:52:44,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-07 09:52:44,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-07 09:52:44,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-07 09:52:44,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-07 09:52:44,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-07 09:52:44,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-07 09:52:44,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-07 09:52:44,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-07 09:52:44,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-07 09:52:44,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-07 09:52:44,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-07 09:52:44,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-07 09:52:44,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-07 09:52:44,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-07 09:52:44,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-07 09:52:44,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-07 09:52:44,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-07 09:52:44,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-07 09:52:44,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-07 09:52:44,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-07 09:52:44,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-07 09:52:44,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-07 09:52:44,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-07 09:52:44,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-07 09:52:44,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-07 09:52:44,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-07 09:52:44,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-07 09:52:44,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-07 09:52:44,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-07 09:52:44,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-07 09:52:44,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-07 09:52:44,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-07 09:52:44,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-07 09:52:44,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-07 09:52:44,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-07 09:52:44,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-07 09:52:44,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-07 09:52:44,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-07 09:52:44,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-07 09:52:44,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-07 09:52:44,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-07 09:52:44,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-07 09:52:44,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-07 09:52:44,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-07 09:52:44,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-07 09:52:44,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-07 09:52:44,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-07 09:52:44,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-07 09:52:44,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-07 09:52:44,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-07 09:52:44,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-07 09:52:44,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-07 09:52:44,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-07 09:52:44,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-07 09:52:44,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-07 09:52:44,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-07 09:52:44,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-07 09:52:44,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-07 09:52:44,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-07 09:52:44,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-07 09:52:44,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-07 09:52:44,896 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020942382016124392); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029900696102013423, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020942382016124392); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012416666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 09:52:44,896 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-07 09:52:44,896 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015382613164108081, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020942382016124392); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029900696102013423, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020942382016124392); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012416666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-07 09:52:45,057 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 232 ms to try 5000 different iterations.  Found a solution that moves 977 regions; Going from a computed imbalance of 0.015382613164108081 to a new imbalance of 7.693859203544768E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0052019929608213104); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0071695828379051965); MoveCostFunction : (multiplier=7.0, imbalance=0.10855555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0052019929608213104); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 09:52:45,059 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv642118181:80 , srv191733607:82 , srv1259352198:83 , srv1365988966:83 , srv1447389035:83 , srv1790706073:83 , srv1916979175:83 , srv295912660:83 , srv600736271:83 , srv1026773405:84 , srv1043676890:84 , srv1097437432:84 , srv1293690230:84 , srv1333393766:84 , srv1339276499:84 , srv1385800727:84 , srv1491402007:84 , srv1552276371:84 , srv1759814693:84 , srv1882731567:84 , srv195399547:84 , srv2026421512:84 , srv336310221:84 , srv367653964:84 , srv521655284:84 , srv543283116:84 , srv572345063:84 , srv846663724:84 , srv888472054:84 , srv1030692155:85 , srv1400868084:85 , srv1442840318:85 , srv1453490985:85 , srv148237400:85 , srv1622424132:85 , srv1649148498:85 , srv1918508905:85 , srv194936169:85 , srv1975539928:85 , srv292310705:85 , srv360015690:85 , srv423255805:85 , srv42734397:85 , srv443264422:85 , srv475271309:85 , srv501055142:85 , srv570975144:85 , srv780850396:85 , srv810496251:85 , srv832916311:85 , srv859688762:85 , srv1037481150:86 , srv1304260895:86 , srv1305103776:86 , srv1446132686:86 , srv1658232959:86 , srv1779545126:86 , srv182576142:86 , srv2141359456:86 , srv2144584551:86 , srv247790537:86 , srv360347817:86 , srv458956559:86 , srv553224078:86 , srv671060865:86 , srv710008515:86 , srv716758768:86 , srv760910455:86 , srv782058838:86 , srv826099201:86 , srv914449457:86 , srv1236688994:87 , srv1659181806:87 , srv1699005953:87 , srv178957770:87 , srv1914808462:87 , srv1929281322:87 , srv1961078525:87 , srv1986011769:87 , srv199547396:87 , srv2081461023:87 , srv257132246:87 , srv300848463:87 , srv35277849:87 , srv414668918:87 , srv589407294:87 , srv66985806:87 , srv690593055:87 , srv812359989:87 , srv969747086:87 , srv971269441:87 , srv975461612:87 , srv1040952143:88 , srv1180558809:88 , srv1924568106:88 , srv351607088:88 , srv524140729:88 , srv574726844:88 , srv707264884:88 , srv1190893350:544 }
2024-01-07 09:52:45,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-07 09:52:45,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv336310221=57, srv2144584551=51, srv443264422=66, srv1190893350=7, srv1043676890=4, srv475271309=68, srv971269441=98, srv1400868084=17, srv148237400=22, srv501055142=69, srv671060865=81, srv1961078525=44, srv1914808462=36, srv859688762=94, srv1442840318=18, srv360015690=60, srv35277849=59, srv191733607=38, srv199547396=47, srv360347817=61, srv1293690230=10, srv2141359456=50, srv367653964=62, srv66985806=80, srv1259352198=9, srv1365988966=15, srv1333393766=13, srv247790537=52, srv1699005953=29, srv1385800727=16, srv257132246=53, srv543283116=72, srv1916979175=37, srv194936169=42, srv1026773405=0, srv521655284=70, srv1097437432=5, srv888472054=95, srv846663724=93, srv1918508905=39, srv1037481150=2, srv2081461023=49, srv600736271=78, srv553224078=73, srv1453490985=21, srv178957770=32, srv1779545126=31, srv812359989=90, srv195399547=43, srv1180558809=6, srv1236688994=8, srv458956559=67, srv1040952143=3, srv975461612=99, srv1659181806=28, srv1790706073=33, srv1649148498=26, srv832916311=92, srv1491402007=23, srv1304260895=11, srv1658232959=27, srv589407294=77, srv351607088=58, srv690593055=82, srv780850396=87, srv1339276499=14, srv572345063=75, srv182576142=34, srv914449457=96, srv1305103776=12, srv1622424132=25, srv2026421512=48, srv1975539928=45, srv414668918=63, srv969747086=97, srv716758768=85, srv292310705=54, srv300848463=56, srv574726844=76, srv42734397=65, srv423255805=64, srv782058838=88, srv826099201=91, srv642118181=79, srv1929281322=41, srv1446132686=19, srv524140729=71, srv570975144=74, srv810496251=89, srv707264884=83, srv1552276371=24, srv1924568106=40, srv295912660=55, srv1759814693=30, srv1986011769=46, srv1030692155=1, srv1447389035=20, srv1882731567=35, srv760910455=86, srv710008515=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-07 09:52:45,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-07 09:52:45,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-07 09:52:45,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-07 09:52:45,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-07 09:52:45,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-07 09:52:45,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-07 09:52:45,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-07 09:52:45,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-07 09:52:45,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-07 09:52:45,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-07 09:52:45,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-07 09:52:45,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-07 09:52:45,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-07 09:52:45,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-07 09:52:45,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-07 09:52:45,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-07 09:52:45,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-07 09:52:45,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-07 09:52:45,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-07 09:52:45,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-07 09:52:45,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-07 09:52:45,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-07 09:52:45,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-07 09:52:45,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-07 09:52:45,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-07 09:52:45,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-07 09:52:45,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-07 09:52:45,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-07 09:52:45,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-07 09:52:45,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-07 09:52:45,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-07 09:52:45,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-07 09:52:45,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-07 09:52:45,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-07 09:52:45,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-07 09:52:45,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-07 09:52:45,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-07 09:52:45,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-07 09:52:45,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-07 09:52:45,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-07 09:52:45,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-07 09:52:45,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-07 09:52:45,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-07 09:52:45,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-07 09:52:45,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-07 09:52:45,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-07 09:52:45,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-07 09:52:45,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-07 09:52:45,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-07 09:52:45,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-07 09:52:45,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-07 09:52:45,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-07 09:52:45,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-07 09:52:45,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-07 09:52:45,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-07 09:52:45,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-07 09:52:45,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-07 09:52:45,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-07 09:52:45,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-07 09:52:45,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-07 09:52:45,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-07 09:52:45,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-07 09:52:45,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-07 09:52:45,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-07 09:52:45,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-07 09:52:45,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-07 09:52:45,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-07 09:52:45,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-07 09:52:45,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-07 09:52:45,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-07 09:52:45,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-07 09:52:45,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-07 09:52:45,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-07 09:52:45,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-07 09:52:45,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-07 09:52:45,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-07 09:52:45,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-07 09:52:45,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-07 09:52:45,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-07 09:52:45,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-07 09:52:45,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-07 09:52:45,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-07 09:52:45,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-07 09:52:45,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-07 09:52:45,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-07 09:52:45,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-07 09:52:45,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-07 09:52:45,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-07 09:52:45,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-07 09:52:45,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-07 09:52:45,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-07 09:52:45,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-07 09:52:45,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-07 09:52:45,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-07 09:52:45,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-07 09:52:45,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-07 09:52:45,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-07 09:52:45,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-07 09:52:45,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-07 09:52:45,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-07 09:52:45,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-07 09:52:45,158 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0052019929608213104); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0071695828379051965); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0052019929608213104); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 09:52:45,158 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-07 09:52:45,158 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.6254482013671E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0052019929608213104); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0071695828379051965); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0052019929608213104); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-07 09:52:45,396 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 311 ms to try 5000 different iterations.  Found a solution that moves 240 regions; Going from a computed imbalance of 7.6254482013671E-4 to a new imbalance of 1.9455072279154107E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.02666666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 09:52:45,397 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1236688994:83 , srv148237400:83 , srv501055142:83 , srv543283116:83 , srv572345063:83 , srv1030692155:84 , srv1040952143:84 , srv1043676890:84 , srv1097437432:84 , srv1180558809:84 , srv1259352198:84 , srv1304260895:84 , srv1333393766:84 , srv1365988966:84 , srv1400868084:84 , srv1453490985:84 , srv1491402007:84 , srv1552276371:84 , srv1622424132:84 , srv1658232959:84 , srv1699005953:84 , srv1779545126:84 , srv178957770:84 , srv182576142:84 , srv1914808462:84 , srv1916979175:84 , srv1924568106:84 , srv1929281322:84 , srv194936169:84 , srv1961078525:84 , srv199547396:84 , srv2144584551:84 , srv257132246:84 , srv351607088:84 , srv35277849:84 , srv360015690:84 , srv367653964:84 , srv414668918:84 , srv42734397:84 , srv524140729:84 , srv574726844:84 , srv600736271:84 , srv66985806:84 , srv690593055:84 , srv707264884:84 , srv716758768:84 , srv810496251:84 , srv812359989:84 , srv832916311:84 , srv859688762:84 , srv1026773405:85 , srv1037481150:85 , srv1293690230:85 , srv1305103776:85 , srv1339276499:85 , srv1385800727:85 , srv1442840318:85 , srv1446132686:85 , srv1447389035:85 , srv1649148498:85 , srv1659181806:85 , srv1759814693:85 , srv1790706073:85 , srv1882731567:85 , srv191733607:85 , srv1918508905:85 , srv195399547:85 , srv1975539928:85 , srv1986011769:85 , srv2026421512:85 , srv2081461023:85 , srv2141359456:85 , srv247790537:85 , srv292310705:85 , srv295912660:85 , srv300848463:85 , srv336310221:85 , srv360347817:85 , srv443264422:85 , srv458956559:85 , srv475271309:85 , srv521655284:85 , srv553224078:85 , srv570975144:85 , srv589407294:85 , srv642118181:85 , srv671060865:85 , srv710008515:85 , srv760910455:85 , srv780850396:85 , srv782058838:85 , srv826099201:85 , srv846663724:85 , srv888472054:85 , srv914449457:85 , srv971269441:85 , srv975461612:85 , srv423255805:86 , srv969747086:86 , srv1190893350:638 }
2024-01-07 09:52:45,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-07 09:52:45,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv336310221=57, srv2144584551=51, srv443264422=66, srv1190893350=7, srv1043676890=4, srv475271309=68, srv971269441=98, srv1400868084=17, srv148237400=22, srv501055142=69, srv671060865=81, srv1961078525=44, srv1914808462=36, srv859688762=94, srv1442840318=18, srv360015690=60, srv35277849=59, srv191733607=38, srv199547396=47, srv360347817=61, srv1293690230=10, srv2141359456=50, srv367653964=62, srv66985806=80, srv1259352198=9, srv1365988966=15, srv1333393766=13, srv247790537=52, srv1699005953=29, srv1385800727=16, srv257132246=53, srv543283116=72, srv1916979175=37, srv194936169=42, srv1026773405=0, srv521655284=70, srv1097437432=5, srv888472054=95, srv846663724=93, srv1918508905=39, srv1037481150=2, srv2081461023=49, srv600736271=78, srv553224078=73, srv1453490985=21, srv178957770=32, srv1779545126=31, srv812359989=90, srv195399547=43, srv1180558809=6, srv1236688994=8, srv458956559=67, srv1040952143=3, srv975461612=99, srv1659181806=28, srv1790706073=33, srv1649148498=26, srv832916311=92, srv1491402007=23, srv1304260895=11, srv1658232959=27, srv589407294=77, srv351607088=58, srv690593055=82, srv780850396=87, srv1339276499=14, srv572345063=75, srv182576142=34, srv914449457=96, srv1305103776=12, srv1622424132=25, srv2026421512=48, srv1975539928=45, srv414668918=63, srv969747086=97, srv716758768=85, srv292310705=54, srv300848463=56, srv574726844=76, srv42734397=65, srv423255805=64, srv782058838=88, srv826099201=91, srv642118181=79, srv1929281322=41, srv1446132686=19, srv524140729=71, srv570975144=74, srv810496251=89, srv707264884=83, srv1552276371=24, srv1924568106=40, srv295912660=55, srv1759814693=30, srv1986011769=46, srv1030692155=1, srv1447389035=20, srv1882731567=35, srv760910455=86, srv710008515=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-07 09:52:45,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-07 09:52:45,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-07 09:52:45,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-07 09:52:45,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-07 09:52:45,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-07 09:52:45,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-07 09:52:45,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-07 09:52:45,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-07 09:52:45,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-07 09:52:45,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-07 09:52:45,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-07 09:52:45,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-07 09:52:45,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-07 09:52:45,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-07 09:52:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-07 09:52:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-07 09:52:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-07 09:52:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-07 09:52:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-07 09:52:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-07 09:52:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-07 09:52:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-07 09:52:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-07 09:52:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-07 09:52:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-07 09:52:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-07 09:52:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-07 09:52:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-07 09:52:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-07 09:52:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-07 09:52:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-07 09:52:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-07 09:52:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-07 09:52:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-07 09:52:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-07 09:52:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-07 09:52:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-07 09:52:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-07 09:52:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-07 09:52:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-07 09:52:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-07 09:52:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-07 09:52:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-07 09:52:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-07 09:52:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-07 09:52:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-07 09:52:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-07 09:52:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-07 09:52:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-07 09:52:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-07 09:52:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-07 09:52:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-07 09:52:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-07 09:52:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-07 09:52:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-07 09:52:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-07 09:52:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-07 09:52:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-07 09:52:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-07 09:52:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-07 09:52:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-07 09:52:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-07 09:52:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-07 09:52:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-07 09:52:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-07 09:52:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-07 09:52:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-07 09:52:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-07 09:52:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-07 09:52:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-07 09:52:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-07 09:52:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-07 09:52:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-07 09:52:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-07 09:52:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-07 09:52:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-07 09:52:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-07 09:52:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-07 09:52:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-07 09:52:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-07 09:52:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-07 09:52:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-07 09:52:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-07 09:52:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-07 09:52:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-07 09:52:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-07 09:52:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-07 09:52:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-07 09:52:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-07 09:52:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-07 09:52:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-07 09:52:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-07 09:52:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-07 09:52:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-07 09:52:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-07 09:52:45,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-07 09:52:45,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-07 09:52:45,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-07 09:52:45,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-07 09:52:45,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-07 09:52:45,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-07 09:52:45,502 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.777455636076716E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 09:52:45,503 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1236688994:83 , srv148237400:83 , srv501055142:83 , srv543283116:83 , srv572345063:83 , srv1030692155:84 , srv1040952143:84 , srv1043676890:84 , srv1097437432:84 , srv1180558809:84 , srv1259352198:84 , srv1304260895:84 , srv1333393766:84 , srv1365988966:84 , srv1400868084:84 , srv1453490985:84 , srv1491402007:84 , srv1552276371:84 , srv1622424132:84 , srv1658232959:84 , srv1699005953:84 , srv1779545126:84 , srv178957770:84 , srv182576142:84 , srv1914808462:84 , srv1916979175:84 , srv1924568106:84 , srv1929281322:84 , srv194936169:84 , srv1961078525:84 , srv199547396:84 , srv2144584551:84 , srv257132246:84 , srv351607088:84 , srv35277849:84 , srv360015690:84 , srv367653964:84 , srv414668918:84 , srv42734397:84 , srv524140729:84 , srv574726844:84 , srv600736271:84 , srv66985806:84 , srv690593055:84 , srv707264884:84 , srv716758768:84 , srv810496251:84 , srv812359989:84 , srv832916311:84 , srv859688762:84 , srv1026773405:85 , srv1037481150:85 , srv1293690230:85 , srv1305103776:85 , srv1339276499:85 , srv1385800727:85 , srv1442840318:85 , srv1446132686:85 , srv1447389035:85 , srv1649148498:85 , srv1659181806:85 , srv1759814693:85 , srv1790706073:85 , srv1882731567:85 , srv191733607:85 , srv1918508905:85 , srv195399547:85 , srv1975539928:85 , srv1986011769:85 , srv2026421512:85 , srv2081461023:85 , srv2141359456:85 , srv247790537:85 , srv292310705:85 , srv295912660:85 , srv300848463:85 , srv336310221:85 , srv360347817:85 , srv443264422:85 , srv458956559:85 , srv475271309:85 , srv521655284:85 , srv553224078:85 , srv570975144:85 , srv589407294:85 , srv642118181:85 , srv671060865:85 , srv710008515:85 , srv760910455:85 , srv780850396:85 , srv782058838:85 , srv826099201:85 , srv846663724:85 , srv888472054:85 , srv914449457:85 , srv971269441:85 , srv975461612:85 , srv423255805:86 , srv969747086:86 , srv1190893350:638 }
2024-01-07 09:52:45,535 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=225 (was 225), ProcessCount=171 (was 171), AvailableMemoryMB=8295 (was 8956)
2024-01-07 09:52:45,545 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=225, ProcessCount=171, AvailableMemoryMB=8293
2024-01-07 09:52:45,545 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-07 09:52:45,546 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-01-07 09:52:45,557 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1048131902:84 , srv1082376372:84 , srv1094414393:84 , srv11396741:84 , srv1222064174:84 , srv1222261345:84 , srv1236438330:84 , srv1275334922:84 , srv1303500118:84 , srv1390779457:84 , srv1411548573:84 , srv1434605181:84 , srv1597555536:84 , srv1620651709:84 , srv1697683200:84 , srv1838512941:84 , srv1938558977:84 , srv1960284310:84 , srv199181750:84 , srv222783948:84 , srv287305008:84 , srv302580854:84 , srv423768957:84 , srv52151442:84 , srv606306668:84 , srv65397710:84 , srv680502775:84 , srv742498722:84 , srv991712315:84 , srv1309453763:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-01-07 09:52:45,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-07 09:52:45,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1222064174=4, srv287305008=21, srv742498722=28, srv1048131902=0, srv302580854=22, srv1309453763=9, srv1303500118=8, srv423768957=23, srv1236438330=6, srv1390779457=10, srv1434605181=12, srv11396741=3, srv1697683200=15, srv1938558977=17, srv1411548573=11, srv1275334922=7, srv1222261345=5, srv680502775=27, srv1620651709=14, srv52151442=24, srv1082376372=1, srv1960284310=18, srv199181750=19, srv991712315=29, srv222783948=20, srv65397710=26, srv1094414393=2, srv1597555536=13, srv606306668=25, srv1838512941=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-07 09:52:45,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-07 09:52:45,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-07 09:52:45,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-07 09:52:45,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-07 09:52:45,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-07 09:52:45,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-07 09:52:45,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-07 09:52:45,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-07 09:52:45,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-07 09:52:45,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-07 09:52:45,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-07 09:52:45,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-07 09:52:45,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-07 09:52:45,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-07 09:52:45,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-07 09:52:45,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-07 09:52:45,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-07 09:52:45,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-07 09:52:45,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-07 09:52:45,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-07 09:52:45,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-07 09:52:45,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-07 09:52:45,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-07 09:52:45,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-07 09:52:45,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-07 09:52:45,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-07 09:52:45,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-07 09:52:45,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-07 09:52:45,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-07 09:52:45,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-07 09:52:45,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-01-07 09:52:45,598 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1392628625954055, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 09:52:45,599 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912631855996967, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1392628625954055, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-01-07 09:54:45,580 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9271889 different iterations.  Found a solution that moves 1836 regions; Going from a computed imbalance of 0.9912631855996967 to a new imbalance of 4.2853155918867096E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.68, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 09:54:45,587 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1048131902:90 , srv1082376372:90 , srv1094414393:90 , srv11396741:90 , srv1222064174:90 , srv1222261345:90 , srv1236438330:90 , srv1275334922:90 , srv1303500118:90 , srv1309453763:90 , srv1390779457:90 , srv1411548573:90 , srv1434605181:90 , srv1597555536:90 , srv1620651709:90 , srv1697683200:90 , srv1838512941:90 , srv1938558977:90 , srv1960284310:90 , srv199181750:90 , srv222783948:90 , srv287305008:90 , srv302580854:90 , srv423768957:90 , srv52151442:90 , srv606306668:90 , srv65397710:90 , srv680502775:90 , srv742498722:90 , srv991712315:90 }
2024-01-07 09:54:45,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-07 09:54:45,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1222064174=4, srv287305008=21, srv742498722=28, srv1048131902=0, srv302580854=22, srv1309453763=9, srv1303500118=8, srv423768957=23, srv1236438330=6, srv1390779457=10, srv1434605181=12, srv11396741=3, srv1697683200=15, srv1938558977=17, srv1411548573=11, srv1275334922=7, srv1222261345=5, srv680502775=27, srv1620651709=14, srv52151442=24, srv1082376372=1, srv1960284310=18, srv199181750=19, srv991712315=29, srv222783948=20, srv65397710=26, srv1094414393=2, srv1597555536=13, srv606306668=25, srv1838512941=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-07 09:54:45,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-07 09:54:45,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-07 09:54:45,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-07 09:54:45,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-07 09:54:45,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-07 09:54:45,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-07 09:54:45,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-07 09:54:45,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-07 09:54:45,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-07 09:54:45,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-07 09:54:45,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-07 09:54:45,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-07 09:54:45,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-07 09:54:45,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-07 09:54:45,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-07 09:54:45,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-07 09:54:45,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-07 09:54:45,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-07 09:54:45,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-07 09:54:45,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-07 09:54:45,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-07 09:54:45,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-07 09:54:45,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-07 09:54:45,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-07 09:54:45,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-07 09:54:45,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-07 09:54:45,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-07 09:54:45,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-07 09:54:45,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-07 09:54:45,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-07 09:54:45,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-01-07 09:54:45,609 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 09:54:45,609 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1048131902:90 , srv1082376372:90 , srv1094414393:90 , srv11396741:90 , srv1222064174:90 , srv1222261345:90 , srv1236438330:90 , srv1275334922:90 , srv1303500118:90 , srv1309453763:90 , srv1390779457:90 , srv1411548573:90 , srv1434605181:90 , srv1597555536:90 , srv1620651709:90 , srv1697683200:90 , srv1838512941:90 , srv1938558977:90 , srv1960284310:90 , srv199181750:90 , srv222783948:90 , srv287305008:90 , srv302580854:90 , srv423768957:90 , srv52151442:90 , srv606306668:90 , srv65397710:90 , srv680502775:90 , srv742498722:90 , srv991712315:90 }
2024-01-07 09:54:45,624 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=192 (was 225), ProcessCount=170 (was 171), AvailableMemoryMB=7926 (was 8293)