2024-01-07 07:53:24,138 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-01-07 07:53:24,454 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-01-07 07:53:24,585 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-01-07 07:53:24,652 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-07 07:53:24,661 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-01-07 07:53:24,675 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=250, ProcessCount=171, AvailableMemoryMB=5898
2024-01-07 07:53:24,677 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-07 07:53:24,677 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-01-07 07:53:24,813 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1000256854:84 , srv1034274189:84 , srv1049295110:84 , srv1056914167:84 , srv107815306:84 , srv1115171982:84 , srv1118343237:84 , srv1137894105:84 , srv1148341834:84 , srv1172002187:84 , srv117208764:84 , srv1234745791:84 , srv1245544468:84 , srv1246878756:84 , srv1252335426:84 , srv1301021950:84 , srv1317304865:84 , srv1352791832:84 , srv1392892783:84 , srv1415370891:84 , srv1422263059:84 , srv1425670069:84 , srv144042496:84 , srv144645033:84 , srv1451857466:84 , srv1475500839:84 , srv1503046141:84 , srv1509974082:84 , srv1531365110:84 , srv1539539724:84 , srv1556367236:84 , srv1623704124:84 , srv1630882345:84 , srv1672481475:84 , srv1679806667:84 , srv1685411057:84 , srv1726578933:84 , srv1744525582:84 , srv1746876502:84 , srv1749578455:84 , srv1754893405:84 , srv1767101005:84 , srv1845959590:84 , srv1869444078:84 , srv1885079703:84 , srv1895627848:84 , srv1917298322:84 , srv1922770668:84 , srv1970229668:84 , srv1993523546:84 , srv2001305541:84 , srv2008187806:84 , srv2015290889:84 , srv2037150902:84 , srv2087997076:84 , srv2090662685:84 , srv2123057283:84 , srv2128243241:84 , srv216978501:84 , srv309044619:84 , srv318463757:84 , srv320821317:84 , srv327263254:84 , srv339753112:84 , srv347103307:84 , srv353603842:84 , srv358435077:84 , srv367044667:84 , srv485579961:84 , srv502713006:84 , srv502726876:84 , srv513981468:84 , srv514698096:84 , srv538791836:84 , srv549122227:84 , srv559865834:84 , srv582287616:84 , srv618450514:84 , srv623958618:84 , srv669445521:84 , srv670742236:84 , srv680736779:84 , srv698255664:84 , srv718745631:84 , srv758667178:84 , srv764775912:84 , srv774652726:84 , srv790211841:84 , srv827168712:84 , srv835831708:84 , srv850147381:84 , srv871215253:84 , srv884018944:84 , srv889315543:84 , srv926243154:84 , srv929779201:84 , srv9321105:84 , srv95502530:84 , srv980768806:84 , srv490255074:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-01-07 07:53:24,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-07 07:53:24,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2087997076=54, srv1000256854=0, srv790211841=88, srv1503046141=26, srv2123057283=56, srv367044667=67, srv889315543=94, srv1767101005=41, srv549122227=75, srv929779201=96, srv758667178=85, srv1148341834=8, srv850147381=91, srv698255664=83, srv1726578933=36, srv559865834=76, srv2001305541=50, srv1970229668=48, srv1754893405=40, srv320821317=61, srv827168712=89, srv1422263059=20, srv485579961=68, srv871215253=92, srv2008187806=51, srv1415370891=19, srv1509974082=27, srv1301021950=15, srv582287616=77, srv2037150902=53, srv1049295110=2, srv1451857466=24, srv680736779=82, srv1922770668=47, srv718745631=84, srv1425670069=21, srv2090662685=55, srv835831708=90, srv1869444078=43, srv1352791832=17, srv669445521=80, srv490255074=69, srv980768806=99, srv1895627848=45, srv670742236=81, srv623958618=79, srv764775912=86, srv1845959590=42, srv1246878756=13, srv1531365110=28, srv1317304865=16, srv1556367236=30, srv1744525582=37, srv358435077=66, srv1917298322=46, srv144042496=22, srv1993523546=49, srv502726876=71, srv513981468=72, srv538791836=74, srv1672481475=33, srv1885079703=44, srv144645033=23, srv1539539724=29, srv926243154=95, srv1034274189=1, srv1234745791=11, srv216978501=58, srv514698096=73, srv2015290889=52, srv774652726=87, srv117208764=10, srv339753112=63, srv1392892783=18, srv9321105=97, srv1685411057=35, srv1115171982=5, srv347103307=64, srv1475500839=25, srv618450514=78, srv1056914167=3, srv1749578455=39, srv327263254=62, srv1630882345=32, srv1137894105=7, srv107815306=4, srv1118343237=6, srv1245544468=12, srv502713006=70, srv1679806667=34, srv2128243241=57, srv353603842=65, srv309044619=59, srv1252335426=14, srv884018944=93, srv318463757=60, srv1172002187=9, srv1623704124=31, srv95502530=98, srv1746876502=38} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-07 07:53:24,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-07 07:53:24,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-07 07:53:24,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-07 07:53:24,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-07 07:53:24,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-07 07:53:24,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-07 07:53:24,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-07 07:53:24,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-07 07:53:24,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-07 07:53:24,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-07 07:53:24,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-07 07:53:24,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-07 07:53:24,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-07 07:53:24,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-07 07:53:24,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-07 07:53:24,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-07 07:53:24,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-07 07:53:24,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-07 07:53:24,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-07 07:53:24,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-07 07:53:24,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-07 07:53:24,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-07 07:53:24,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-07 07:53:24,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-07 07:53:24,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-07 07:53:24,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-07 07:53:24,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-07 07:53:24,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-07 07:53:24,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-07 07:53:24,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-07 07:53:24,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-07 07:53:24,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-07 07:53:24,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-07 07:53:24,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-07 07:53:24,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-07 07:53:24,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-07 07:53:24,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-07 07:53:24,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-07 07:53:24,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-07 07:53:24,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-07 07:53:24,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-07 07:53:24,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-07 07:53:24,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-07 07:53:24,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-07 07:53:24,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-07 07:53:24,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-07 07:53:24,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-07 07:53:24,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-07 07:53:24,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-07 07:53:24,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-07 07:53:24,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-07 07:53:24,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-07 07:53:24,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-07 07:53:24,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-07 07:53:24,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-07 07:53:24,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-07 07:53:24,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-07 07:53:24,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-07 07:53:24,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-07 07:53:24,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-07 07:53:24,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-07 07:53:24,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-07 07:53:24,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-07 07:53:24,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-07 07:53:24,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-07 07:53:24,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-07 07:53:24,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-07 07:53:24,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-07 07:53:24,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-07 07:53:24,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-07 07:53:24,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-07 07:53:24,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-07 07:53:24,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-07 07:53:24,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-07 07:53:24,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-07 07:53:24,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-07 07:53:24,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-07 07:53:24,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-07 07:53:24,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-07 07:53:24,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-07 07:53:24,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-07 07:53:24,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-07 07:53:24,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-07 07:53:24,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-07 07:53:24,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-07 07:53:24,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-07 07:53:24,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-07 07:53:24,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-07 07:53:24,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-07 07:53:24,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-07 07:53:24,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-07 07:53:24,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-07 07:53:24,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-07 07:53:24,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-07 07:53:24,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-07 07:53:24,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-07 07:53:24,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-07 07:53:24,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-07 07:53:24,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-07 07:53:24,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-07 07:53:24,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-07 07:53:25,160 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 07:53:25,160 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-07 07:53:25,161 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-07 07:53:25,580 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 604 ms to try 5000 different iterations.  Found a solution that moves 4713 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17863240987111884. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03903595728098643, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05647124155120423, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5236666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03903595728098643, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.174, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23891666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 07:53:25,603 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1234745791:81 , srv1115171982:82 , srv1034274189:83 , srv107815306:83 , srv144042496:83 , srv2090662685:83 , srv367044667:83 , srv582287616:83 , srv680736779:83 , srv884018944:83 , srv1056914167:84 , srv1148341834:84 , srv1352791832:84 , srv144645033:84 , srv1503046141:84 , srv1556367236:84 , srv1744525582:84 , srv514698096:84 , srv764775912:84 , srv1245544468:85 , srv1749578455:85 , srv1754893405:85 , srv1869444078:85 , srv1993523546:85 , srv216978501:85 , srv358435077:85 , srv502713006:85 , srv502726876:85 , srv513981468:85 , srv698255664:85 , srv835831708:85 , srv850147381:85 , srv871215253:85 , srv929779201:85 , srv1301021950:86 , srv1392892783:86 , srv1415370891:86 , srv1425670069:86 , srv1539539724:86 , srv1630882345:86 , srv2015290889:86 , srv2087997076:86 , srv2123057283:86 , srv318463757:86 , srv353603842:86 , srv549122227:86 , srv669445521:86 , srv670742236:86 , srv718745631:86 , srv774652726:86 , srv790211841:86 , srv889315543:86 , srv95502530:86 , srv1118343237:87 , srv117208764:87 , srv1317304865:87 , srv1475500839:87 , srv1623704124:87 , srv1685411057:87 , srv1746876502:87 , srv1845959590:87 , srv1895627848:87 , srv1917298322:87 , srv309044619:87 , srv320821317:87 , srv327263254:87 , srv347103307:87 , srv623958618:87 , srv758667178:87 , srv926243154:87 , srv980768806:87 , srv1137894105:88 , srv1172002187:88 , srv1422263059:88 , srv1726578933:88 , srv1767101005:88 , srv1970229668:88 , srv538791836:88 , srv618450514:88 , srv1000256854:89 , srv1246878756:89 , srv1252335426:89 , srv1509974082:89 , srv1531365110:89 , srv1679806667:89 , srv2001305541:89 , srv2037150902:89 , srv339753112:89 , srv1049295110:90 , srv1451857466:90 , srv1885079703:90 , srv2008187806:90 , srv485579961:90 , srv827168712:90 , srv9321105:90 , srv1922770668:91 , srv2128243241:91 , srv559865834:92 , srv1672481475:96 , srv490255074:437 }
2024-01-07 07:53:25,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-07 07:53:25,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2087997076=54, srv1000256854=0, srv790211841=88, srv1503046141=26, srv2123057283=56, srv367044667=67, srv889315543=94, srv1767101005=41, srv549122227=75, srv929779201=96, srv758667178=85, srv1148341834=8, srv850147381=91, srv698255664=83, srv1726578933=36, srv559865834=76, srv2001305541=50, srv1970229668=48, srv1754893405=40, srv320821317=61, srv827168712=89, srv1422263059=20, srv485579961=68, srv871215253=92, srv2008187806=51, srv1415370891=19, srv1509974082=27, srv1301021950=15, srv582287616=77, srv2037150902=53, srv1049295110=2, srv1451857466=24, srv680736779=82, srv1922770668=47, srv718745631=84, srv1425670069=21, srv2090662685=55, srv835831708=90, srv1869444078=43, srv1352791832=17, srv669445521=80, srv490255074=69, srv980768806=99, srv1895627848=45, srv670742236=81, srv623958618=79, srv764775912=86, srv1845959590=42, srv1246878756=13, srv1531365110=28, srv1317304865=16, srv1556367236=30, srv1744525582=37, srv358435077=66, srv1917298322=46, srv144042496=22, srv1993523546=49, srv502726876=71, srv513981468=72, srv538791836=74, srv1672481475=33, srv1885079703=44, srv144645033=23, srv1539539724=29, srv926243154=95, srv1034274189=1, srv1234745791=11, srv216978501=58, srv514698096=73, srv2015290889=52, srv774652726=87, srv117208764=10, srv339753112=63, srv1392892783=18, srv9321105=97, srv1685411057=35, srv1115171982=5, srv347103307=64, srv1475500839=25, srv618450514=78, srv1056914167=3, srv1749578455=39, srv327263254=62, srv1630882345=32, srv1137894105=7, srv107815306=4, srv1118343237=6, srv1245544468=12, srv502713006=70, srv1679806667=34, srv2128243241=57, srv353603842=65, srv309044619=59, srv1252335426=14, srv884018944=93, srv318463757=60, srv1172002187=9, srv1623704124=31, srv95502530=98, srv1746876502=38} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-07 07:53:25,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-07 07:53:25,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-07 07:53:25,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-07 07:53:25,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-07 07:53:25,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-07 07:53:25,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-07 07:53:25,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-07 07:53:25,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-07 07:53:25,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-07 07:53:25,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-07 07:53:25,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-07 07:53:25,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-07 07:53:25,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-07 07:53:25,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-07 07:53:25,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-07 07:53:25,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-07 07:53:25,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-07 07:53:25,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-07 07:53:25,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-07 07:53:25,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-07 07:53:25,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-07 07:53:25,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-07 07:53:25,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-07 07:53:25,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-07 07:53:25,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-07 07:53:25,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-07 07:53:25,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-07 07:53:25,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-07 07:53:25,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-07 07:53:25,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-07 07:53:25,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-07 07:53:25,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-07 07:53:25,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-07 07:53:25,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-07 07:53:25,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-07 07:53:25,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-07 07:53:25,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-07 07:53:25,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-07 07:53:25,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-07 07:53:25,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-07 07:53:25,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-07 07:53:25,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-07 07:53:25,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-07 07:53:25,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-07 07:53:25,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-07 07:53:25,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-07 07:53:25,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-07 07:53:25,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-07 07:53:25,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-07 07:53:25,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-07 07:53:25,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-07 07:53:25,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-07 07:53:25,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-07 07:53:25,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-07 07:53:25,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-07 07:53:25,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-07 07:53:25,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-07 07:53:25,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-07 07:53:25,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-07 07:53:25,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-07 07:53:25,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-07 07:53:25,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-07 07:53:25,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-07 07:53:25,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-07 07:53:25,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-07 07:53:25,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-07 07:53:25,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-07 07:53:25,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-07 07:53:25,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-07 07:53:25,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-07 07:53:25,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-07 07:53:25,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-07 07:53:25,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-07 07:53:25,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-07 07:53:25,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-07 07:53:25,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-07 07:53:25,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-07 07:53:25,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-07 07:53:25,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-07 07:53:25,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-07 07:53:25,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-07 07:53:25,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-07 07:53:25,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-07 07:53:25,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-07 07:53:25,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-07 07:53:25,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-07 07:53:25,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-07 07:53:25,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-07 07:53:25,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-07 07:53:25,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-07 07:53:25,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-07 07:53:25,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-07 07:53:25,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-07 07:53:25,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-07 07:53:25,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-07 07:53:25,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-07 07:53:25,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-07 07:53:25,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-07 07:53:25,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-07 07:53:25,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-07 07:53:25,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-07 07:53:25,828 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03903595728098643, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05647124155120423, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03903595728098643, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.174, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23891666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 07:53:25,829 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-07 07:53:25,829 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17859940873977148, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03903595728098643, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05647124155120423, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03903595728098643, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.174, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23891666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-07 07:53:26,120 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 448 ms to try 5000 different iterations.  Found a solution that moves 2634 regions; Going from a computed imbalance of 0.17859940873977148 to a new imbalance of 0.015389349511272587. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020938808942134252); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02897035909249957, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2926666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020938808942134252); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04566666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 07:53:26,139 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1672481475:75 , srv2128243241:80 , srv1049295110:81 , srv2037150902:81 , srv485579961:81 , srv559865834:81 , srv827168712:81 , srv1000256854:82 , srv1422263059:82 , srv1509974082:82 , srv1726578933:82 , srv1922770668:82 , srv2008187806:82 , srv339753112:82 , srv1172002187:83 , srv117208764:83 , srv1246878756:83 , srv1531365110:83 , srv1746876502:83 , srv1885079703:83 , srv327263254:83 , srv926243154:83 , srv1252335426:84 , srv1317304865:84 , srv1392892783:84 , srv1415370891:84 , srv1451857466:84 , srv1475500839:84 , srv1623704124:84 , srv1679806667:84 , srv1970229668:84 , srv2001305541:84 , srv347103307:84 , srv618450514:84 , srv623958618:84 , srv758667178:84 , srv1118343237:85 , srv1137894105:85 , srv1685411057:85 , srv1767101005:85 , srv1895627848:85 , srv1917298322:85 , srv309044619:85 , srv318463757:85 , srv358435077:85 , srv538791836:85 , srv670742236:85 , srv718745631:85 , srv980768806:85 , srv1301021950:86 , srv1539539724:86 , srv1869444078:86 , srv353603842:86 , srv502726876:86 , srv513981468:86 , srv790211841:86 , srv850147381:86 , srv871215253:86 , srv1056914167:87 , srv1352791832:87 , srv1425670069:87 , srv1556367236:87 , srv1630882345:87 , srv1744525582:87 , srv1845959590:87 , srv1993523546:87 , srv2015290889:87 , srv2087997076:87 , srv320821317:87 , srv669445521:87 , srv774652726:87 , srv1148341834:88 , srv1754893405:88 , srv2123057283:88 , srv216978501:88 , srv502713006:88 , srv835831708:88 , srv889315543:88 , srv929779201:88 , srv95502530:88 , srv107815306:89 , srv1245544468:89 , srv144042496:89 , srv144645033:89 , srv1503046141:89 , srv1749578455:89 , srv367044667:89 , srv514698096:89 , srv680736779:89 , srv698255664:89 , srv764775912:89 , srv884018944:89 , srv1034274189:90 , srv2090662685:90 , srv549122227:90 , srv582287616:90 , srv9321105:90 , srv1234745791:91 , srv1115171982:92 , srv490255074:523 }
2024-01-07 07:53:26,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-07 07:53:26,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2087997076=54, srv1000256854=0, srv790211841=88, srv1503046141=26, srv2123057283=56, srv367044667=67, srv889315543=94, srv1767101005=41, srv549122227=75, srv929779201=96, srv758667178=85, srv1148341834=8, srv850147381=91, srv698255664=83, srv1726578933=36, srv559865834=76, srv2001305541=50, srv1970229668=48, srv1754893405=40, srv320821317=61, srv827168712=89, srv1422263059=20, srv485579961=68, srv871215253=92, srv2008187806=51, srv1415370891=19, srv1509974082=27, srv1301021950=15, srv582287616=77, srv2037150902=53, srv1049295110=2, srv1451857466=24, srv680736779=82, srv1922770668=47, srv718745631=84, srv1425670069=21, srv2090662685=55, srv835831708=90, srv1869444078=43, srv1352791832=17, srv669445521=80, srv490255074=69, srv980768806=99, srv1895627848=45, srv670742236=81, srv623958618=79, srv764775912=86, srv1845959590=42, srv1246878756=13, srv1531365110=28, srv1317304865=16, srv1556367236=30, srv1744525582=37, srv358435077=66, srv1917298322=46, srv144042496=22, srv1993523546=49, srv502726876=71, srv513981468=72, srv538791836=74, srv1672481475=33, srv1885079703=44, srv144645033=23, srv1539539724=29, srv926243154=95, srv1034274189=1, srv1234745791=11, srv216978501=58, srv514698096=73, srv2015290889=52, srv774652726=87, srv117208764=10, srv339753112=63, srv1392892783=18, srv9321105=97, srv1685411057=35, srv1115171982=5, srv347103307=64, srv1475500839=25, srv618450514=78, srv1056914167=3, srv1749578455=39, srv327263254=62, srv1630882345=32, srv1137894105=7, srv107815306=4, srv1118343237=6, srv1245544468=12, srv502713006=70, srv1679806667=34, srv2128243241=57, srv353603842=65, srv309044619=59, srv1252335426=14, srv884018944=93, srv318463757=60, srv1172002187=9, srv1623704124=31, srv95502530=98, srv1746876502=38} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-07 07:53:26,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-07 07:53:26,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-07 07:53:26,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-07 07:53:26,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-07 07:53:26,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-07 07:53:26,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-07 07:53:26,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-07 07:53:26,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-07 07:53:26,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-07 07:53:26,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-07 07:53:26,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-07 07:53:26,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-07 07:53:26,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-07 07:53:26,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-07 07:53:26,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-07 07:53:26,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-07 07:53:26,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-07 07:53:26,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-07 07:53:26,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-07 07:53:26,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-07 07:53:26,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-07 07:53:26,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-07 07:53:26,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-07 07:53:26,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-07 07:53:26,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-07 07:53:26,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-07 07:53:26,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-07 07:53:26,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-07 07:53:26,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-07 07:53:26,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-07 07:53:26,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-07 07:53:26,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-07 07:53:26,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-07 07:53:26,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-07 07:53:26,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-07 07:53:26,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-07 07:53:26,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-07 07:53:26,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-07 07:53:26,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-07 07:53:26,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-07 07:53:26,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-07 07:53:26,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-07 07:53:26,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-07 07:53:26,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-07 07:53:26,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-07 07:53:26,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-07 07:53:26,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-07 07:53:26,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-07 07:53:26,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-07 07:53:26,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-07 07:53:26,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-07 07:53:26,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-07 07:53:26,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-07 07:53:26,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-07 07:53:26,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-07 07:53:26,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-07 07:53:26,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-07 07:53:26,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-07 07:53:26,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-07 07:53:26,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-07 07:53:26,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-07 07:53:26,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-07 07:53:26,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-07 07:53:26,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-07 07:53:26,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-07 07:53:26,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-07 07:53:26,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-07 07:53:26,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-07 07:53:26,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-07 07:53:26,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-07 07:53:26,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-07 07:53:26,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-07 07:53:26,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-07 07:53:26,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-07 07:53:26,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-07 07:53:26,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-07 07:53:26,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-07 07:53:26,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-07 07:53:26,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-07 07:53:26,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-07 07:53:26,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-07 07:53:26,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-07 07:53:26,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-07 07:53:26,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-07 07:53:26,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-07 07:53:26,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-07 07:53:26,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-07 07:53:26,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-07 07:53:26,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-07 07:53:26,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-07 07:53:26,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-07 07:53:26,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-07 07:53:26,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-07 07:53:26,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-07 07:53:26,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-07 07:53:26,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-07 07:53:26,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-07 07:53:26,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-07 07:53:26,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-07 07:53:26,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-07 07:53:26,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-07 07:53:26,299 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020938808942134252); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02897035909249957, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020938808942134252); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04566666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 07:53:26,299 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-07 07:53:26,299 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015370905849068289, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020938808942134252); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02897035909249957, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020938808942134252); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04566666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-07 07:53:26,549 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 350 ms to try 5000 different iterations.  Found a solution that moves 985 regions; Going from a computed imbalance of 0.015370905849068289 to a new imbalance of 8.599666807516689E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006661052630583241); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0073854894587599635); MoveCostFunction : (multiplier=7.0, imbalance=0.10944444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006661052630583241); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 07:53:26,551 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv9321105:77 , srv549122227:81 , srv1115171982:83 , srv1845959590:83 , srv1034274189:84 , srv1137894105:84 , srv1245544468:84 , srv1425670069:84 , srv144645033:84 , srv1630882345:84 , srv1749578455:84 , srv1754893405:84 , srv1885079703:84 , srv2015290889:84 , srv2087997076:84 , srv2090662685:84 , srv2123057283:84 , srv320821317:84 , srv367044667:84 , srv502713006:84 , srv514698096:84 , srv538791836:84 , srv559865834:84 , srv582287616:84 , srv618450514:84 , srv698255664:84 , srv764775912:84 , srv835831708:84 , srv889315543:84 , srv95502530:84 , srv107815306:85 , srv1148341834:85 , srv1172002187:85 , srv1246878756:85 , srv1252335426:85 , srv1451857466:85 , srv1503046141:85 , srv1531365110:85 , srv1679806667:85 , srv1767101005:85 , srv1922770668:85 , srv1970229668:85 , srv1993523546:85 , srv2001305541:85 , srv2008187806:85 , srv309044619:85 , srv353603842:85 , srv774652726:85 , srv790211841:85 , srv884018944:85 , srv929779201:85 , srv980768806:85 , srv1049295110:86 , srv1118343237:86 , srv1234745791:86 , srv1301021950:86 , srv1352791832:86 , srv144042496:86 , srv1539539724:86 , srv1556367236:86 , srv1623704124:86 , srv1672481475:86 , srv1685411057:86 , srv1726578933:86 , srv1744525582:86 , srv1869444078:86 , srv1895627848:86 , srv1917298322:86 , srv2128243241:86 , srv216978501:86 , srv318463757:86 , srv339753112:86 , srv347103307:86 , srv485579961:86 , srv513981468:86 , srv623958618:86 , srv669445521:86 , srv670742236:86 , srv680736779:86 , srv758667178:86 , srv850147381:86 , srv1000256854:87 , srv1056914167:87 , srv117208764:87 , srv1392892783:87 , srv1415370891:87 , srv1475500839:87 , srv1509974082:87 , srv1746876502:87 , srv2037150902:87 , srv358435077:87 , srv502726876:87 , srv718745631:87 , srv827168712:87 , srv871215253:87 , srv1317304865:88 , srv1422263059:88 , srv327263254:88 , srv926243154:89 , srv490255074:557 }
2024-01-07 07:53:26,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-07 07:53:26,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2087997076=54, srv1000256854=0, srv790211841=88, srv1503046141=26, srv2123057283=56, srv367044667=67, srv889315543=94, srv1767101005=41, srv549122227=75, srv929779201=96, srv758667178=85, srv1148341834=8, srv850147381=91, srv698255664=83, srv1726578933=36, srv559865834=76, srv2001305541=50, srv1970229668=48, srv1754893405=40, srv320821317=61, srv827168712=89, srv1422263059=20, srv485579961=68, srv871215253=92, srv2008187806=51, srv1415370891=19, srv1509974082=27, srv1301021950=15, srv582287616=77, srv2037150902=53, srv1049295110=2, srv1451857466=24, srv680736779=82, srv1922770668=47, srv718745631=84, srv1425670069=21, srv2090662685=55, srv835831708=90, srv1869444078=43, srv1352791832=17, srv669445521=80, srv490255074=69, srv980768806=99, srv1895627848=45, srv670742236=81, srv623958618=79, srv764775912=86, srv1845959590=42, srv1246878756=13, srv1531365110=28, srv1317304865=16, srv1556367236=30, srv1744525582=37, srv358435077=66, srv1917298322=46, srv144042496=22, srv1993523546=49, srv502726876=71, srv513981468=72, srv538791836=74, srv1672481475=33, srv1885079703=44, srv144645033=23, srv1539539724=29, srv926243154=95, srv1034274189=1, srv1234745791=11, srv216978501=58, srv514698096=73, srv2015290889=52, srv774652726=87, srv117208764=10, srv339753112=63, srv1392892783=18, srv9321105=97, srv1685411057=35, srv1115171982=5, srv347103307=64, srv1475500839=25, srv618450514=78, srv1056914167=3, srv1749578455=39, srv327263254=62, srv1630882345=32, srv1137894105=7, srv107815306=4, srv1118343237=6, srv1245544468=12, srv502713006=70, srv1679806667=34, srv2128243241=57, srv353603842=65, srv309044619=59, srv1252335426=14, srv884018944=93, srv318463757=60, srv1172002187=9, srv1623704124=31, srv95502530=98, srv1746876502=38} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-07 07:53:26,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-07 07:53:26,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-07 07:53:26,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-07 07:53:26,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-07 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-07 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-07 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-07 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-07 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-07 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-07 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-07 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-07 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-07 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-07 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-07 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-07 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-07 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-07 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-07 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-07 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-07 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-07 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-07 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-07 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-07 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-07 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-07 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-07 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-07 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-07 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-07 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-07 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-07 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-07 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-07 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-07 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-07 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-07 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-07 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-07 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-07 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-07 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-07 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-07 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-07 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-07 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-07 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-07 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-07 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-07 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-07 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-07 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-07 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-07 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-07 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-07 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-07 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-07 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-07 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-07 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-07 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-07 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-07 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-07 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-07 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-07 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-07 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-07 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-07 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-07 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-07 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-07 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-07 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-07 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-07 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-07 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-07 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-07 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-07 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-07 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-07 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-07 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-07 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-07 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-07 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-07 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-07 07:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-07 07:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-07 07:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-07 07:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-07 07:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-07 07:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-07 07:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-07 07:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-07 07:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-07 07:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-07 07:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-07 07:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-07 07:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-07 07:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-07 07:53:26,685 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006661052630583241); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0073854894587599635); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006661052630583241); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 07:53:26,685 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-07 07:53:26,685 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.530695633366224E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006661052630583241); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0073854894587599635); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006661052630583241); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-07 07:53:27,042 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 458 ms to try 5000 different iterations.  Found a solution that moves 240 regions; Going from a computed imbalance of 8.530695633366224E-4 to a new imbalance of 2.05128854581736E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.02666666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 07:53:27,043 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1451857466:83 , srv1895627848:83 , srv216978501:83 , srv669445521:83 , srv926243154:83 , srv1000256854:84 , srv1056914167:84 , srv1115171982:84 , srv1118343237:84 , srv1234745791:84 , srv1245544468:84 , srv1246878756:84 , srv1301021950:84 , srv1317304865:84 , srv1352791832:84 , srv1422263059:84 , srv1475500839:84 , srv1503046141:84 , srv1539539724:84 , srv1672481475:84 , srv1679806667:84 , srv1685411057:84 , srv1749578455:84 , srv1767101005:84 , srv1917298322:84 , srv1922770668:84 , srv2001305541:84 , srv2015290889:84 , srv2037150902:84 , srv2090662685:84 , srv2123057283:84 , srv2128243241:84 , srv320821317:84 , srv327263254:84 , srv502726876:84 , srv680736779:84 , srv698255664:84 , srv718745631:84 , srv827168712:84 , srv871215253:84 , srv889315543:84 , srv929779201:84 , srv95502530:84 , srv1034274189:85 , srv1049295110:85 , srv107815306:85 , srv1137894105:85 , srv117208764:85 , srv1252335426:85 , srv1392892783:85 , srv1415370891:85 , srv1425670069:85 , srv144042496:85 , srv144645033:85 , srv1509974082:85 , srv1531365110:85 , srv1556367236:85 , srv1623704124:85 , srv1630882345:85 , srv1744525582:85 , srv1746876502:85 , srv1754893405:85 , srv1869444078:85 , srv1885079703:85 , srv1970229668:85 , srv1993523546:85 , srv2008187806:85 , srv2087997076:85 , srv309044619:85 , srv318463757:85 , srv339753112:85 , srv347103307:85 , srv353603842:85 , srv358435077:85 , srv485579961:85 , srv502713006:85 , srv513981468:85 , srv538791836:85 , srv549122227:85 , srv559865834:85 , srv582287616:85 , srv623958618:85 , srv670742236:85 , srv758667178:85 , srv764775912:85 , srv774652726:85 , srv790211841:85 , srv835831708:85 , srv850147381:85 , srv884018944:85 , srv980768806:85 , srv1148341834:86 , srv1172002187:86 , srv1726578933:86 , srv1845959590:86 , srv367044667:86 , srv514698096:86 , srv618450514:86 , srv9321105:86 , srv490255074:625 }
2024-01-07 07:53:27,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-07 07:53:27,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2087997076=54, srv1000256854=0, srv790211841=88, srv1503046141=26, srv2123057283=56, srv367044667=67, srv889315543=94, srv1767101005=41, srv549122227=75, srv929779201=96, srv758667178=85, srv1148341834=8, srv850147381=91, srv698255664=83, srv1726578933=36, srv559865834=76, srv2001305541=50, srv1970229668=48, srv1754893405=40, srv320821317=61, srv827168712=89, srv1422263059=20, srv485579961=68, srv871215253=92, srv2008187806=51, srv1415370891=19, srv1509974082=27, srv1301021950=15, srv582287616=77, srv2037150902=53, srv1049295110=2, srv1451857466=24, srv680736779=82, srv1922770668=47, srv718745631=84, srv1425670069=21, srv2090662685=55, srv835831708=90, srv1869444078=43, srv1352791832=17, srv669445521=80, srv490255074=69, srv980768806=99, srv1895627848=45, srv670742236=81, srv623958618=79, srv764775912=86, srv1845959590=42, srv1246878756=13, srv1531365110=28, srv1317304865=16, srv1556367236=30, srv1744525582=37, srv358435077=66, srv1917298322=46, srv144042496=22, srv1993523546=49, srv502726876=71, srv513981468=72, srv538791836=74, srv1672481475=33, srv1885079703=44, srv144645033=23, srv1539539724=29, srv926243154=95, srv1034274189=1, srv1234745791=11, srv216978501=58, srv514698096=73, srv2015290889=52, srv774652726=87, srv117208764=10, srv339753112=63, srv1392892783=18, srv9321105=97, srv1685411057=35, srv1115171982=5, srv347103307=64, srv1475500839=25, srv618450514=78, srv1056914167=3, srv1749578455=39, srv327263254=62, srv1630882345=32, srv1137894105=7, srv107815306=4, srv1118343237=6, srv1245544468=12, srv502713006=70, srv1679806667=34, srv2128243241=57, srv353603842=65, srv309044619=59, srv1252335426=14, srv884018944=93, srv318463757=60, srv1172002187=9, srv1623704124=31, srv95502530=98, srv1746876502=38} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-07 07:53:27,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-07 07:53:27,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-07 07:53:27,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-07 07:53:27,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-07 07:53:27,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-07 07:53:27,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-07 07:53:27,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-07 07:53:27,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-07 07:53:27,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-07 07:53:27,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-07 07:53:27,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-07 07:53:27,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-07 07:53:27,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-07 07:53:27,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-07 07:53:27,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-07 07:53:27,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-07 07:53:27,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-07 07:53:27,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-07 07:53:27,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-07 07:53:27,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-07 07:53:27,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-07 07:53:27,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-07 07:53:27,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-07 07:53:27,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-07 07:53:27,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-07 07:53:27,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-07 07:53:27,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-07 07:53:27,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-07 07:53:27,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-07 07:53:27,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-07 07:53:27,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-07 07:53:27,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-07 07:53:27,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-07 07:53:27,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-07 07:53:27,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-07 07:53:27,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-07 07:53:27,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-07 07:53:27,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-07 07:53:27,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-07 07:53:27,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-07 07:53:27,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-07 07:53:27,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-07 07:53:27,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-07 07:53:27,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-07 07:53:27,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-07 07:53:27,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-07 07:53:27,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-07 07:53:27,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-07 07:53:27,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-07 07:53:27,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-07 07:53:27,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-07 07:53:27,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-07 07:53:27,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-07 07:53:27,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-07 07:53:27,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-07 07:53:27,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-07 07:53:27,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-07 07:53:27,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-07 07:53:27,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-07 07:53:27,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-07 07:53:27,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-07 07:53:27,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-07 07:53:27,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-07 07:53:27,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-07 07:53:27,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-07 07:53:27,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-07 07:53:27,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-07 07:53:27,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-07 07:53:27,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-07 07:53:27,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-07 07:53:27,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-07 07:53:27,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-07 07:53:27,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-07 07:53:27,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-07 07:53:27,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-07 07:53:27,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-07 07:53:27,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-07 07:53:27,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-07 07:53:27,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-07 07:53:27,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-07 07:53:27,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-07 07:53:27,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-07 07:53:27,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-07 07:53:27,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-07 07:53:27,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-07 07:53:27,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-07 07:53:27,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-07 07:53:27,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-07 07:53:27,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-07 07:53:27,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-07 07:53:27,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-07 07:53:27,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-07 07:53:27,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-07 07:53:27,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-07 07:53:27,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-07 07:53:27,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-07 07:53:27,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-07 07:53:27,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-07 07:53:27,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-07 07:53:27,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-07 07:53:27,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-07 07:53:27,175 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.8832369539786655E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 07:53:27,176 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1451857466:83 , srv1895627848:83 , srv216978501:83 , srv669445521:83 , srv926243154:83 , srv1000256854:84 , srv1056914167:84 , srv1115171982:84 , srv1118343237:84 , srv1234745791:84 , srv1245544468:84 , srv1246878756:84 , srv1301021950:84 , srv1317304865:84 , srv1352791832:84 , srv1422263059:84 , srv1475500839:84 , srv1503046141:84 , srv1539539724:84 , srv1672481475:84 , srv1679806667:84 , srv1685411057:84 , srv1749578455:84 , srv1767101005:84 , srv1917298322:84 , srv1922770668:84 , srv2001305541:84 , srv2015290889:84 , srv2037150902:84 , srv2090662685:84 , srv2123057283:84 , srv2128243241:84 , srv320821317:84 , srv327263254:84 , srv502726876:84 , srv680736779:84 , srv698255664:84 , srv718745631:84 , srv827168712:84 , srv871215253:84 , srv889315543:84 , srv929779201:84 , srv95502530:84 , srv1034274189:85 , srv1049295110:85 , srv107815306:85 , srv1137894105:85 , srv117208764:85 , srv1252335426:85 , srv1392892783:85 , srv1415370891:85 , srv1425670069:85 , srv144042496:85 , srv144645033:85 , srv1509974082:85 , srv1531365110:85 , srv1556367236:85 , srv1623704124:85 , srv1630882345:85 , srv1744525582:85 , srv1746876502:85 , srv1754893405:85 , srv1869444078:85 , srv1885079703:85 , srv1970229668:85 , srv1993523546:85 , srv2008187806:85 , srv2087997076:85 , srv309044619:85 , srv318463757:85 , srv339753112:85 , srv347103307:85 , srv353603842:85 , srv358435077:85 , srv485579961:85 , srv502713006:85 , srv513981468:85 , srv538791836:85 , srv549122227:85 , srv559865834:85 , srv582287616:85 , srv623958618:85 , srv670742236:85 , srv758667178:85 , srv764775912:85 , srv774652726:85 , srv790211841:85 , srv835831708:85 , srv850147381:85 , srv884018944:85 , srv980768806:85 , srv1148341834:86 , srv1172002187:86 , srv1726578933:86 , srv1845959590:86 , srv367044667:86 , srv514698096:86 , srv618450514:86 , srv9321105:86 , srv490255074:625 }
2024-01-07 07:53:27,204 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=250 (was 250), ProcessCount=171 (was 171), AvailableMemoryMB=5322 (was 5898)
2024-01-07 07:53:27,211 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=250, ProcessCount=171, AvailableMemoryMB=5321
2024-01-07 07:53:27,212 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-07 07:53:27,212 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-01-07 07:53:27,222 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1127827889:84 , srv1182888566:84 , srv1247829138:84 , srv136443153:84 , srv1379422117:84 , srv1397746121:84 , srv1590005709:84 , srv1592685614:84 , srv164785903:84 , srv1766664302:84 , srv1774322143:84 , srv2017040839:84 , srv2028429644:84 , srv2112667816:84 , srv249935190:84 , srv277062356:84 , srv320982321:84 , srv361302730:84 , srv368967512:84 , srv406680817:84 , srv526348154:84 , srv645969697:84 , srv664027237:84 , srv821156373:84 , srv859036928:84 , srv860885289:84 , srv867488337:84 , srv904028759:84 , srv973234995:84 , srv332653023:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-01-07 07:53:27,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-07 07:53:27,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2112667816=13, srv164785903=8, srv332653023=17, srv1182888566=1, srv136443153=3, srv277062356=15, srv320982321=16, srv1590005709=6, srv973234995=29, srv1379422117=4, srv645969697=22, srv1127827889=0, srv867488337=27, srv361302730=18, srv249935190=14, srv860885289=26, srv526348154=21, srv664027237=23, srv2028429644=12, srv406680817=20, srv368967512=19, srv904028759=28, srv1397746121=5, srv1592685614=7, srv1766664302=9, srv2017040839=11, srv821156373=24, srv859036928=25, srv1247829138=2, srv1774322143=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-07 07:53:27,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-07 07:53:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-07 07:53:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-07 07:53:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-07 07:53:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-07 07:53:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-07 07:53:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-07 07:53:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-07 07:53:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-07 07:53:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-07 07:53:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-07 07:53:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-07 07:53:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-07 07:53:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-07 07:53:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-07 07:53:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-07 07:53:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-07 07:53:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-07 07:53:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-07 07:53:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-07 07:53:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-07 07:53:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-07 07:53:27,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-07 07:53:27,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-07 07:53:27,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-07 07:53:27,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-07 07:53:27,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-07 07:53:27,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-07 07:53:27,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-07 07:53:27,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-07 07:53:27,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-01-07 07:53:27,249 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.136764344754456, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 07:53:27,249 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912623983248834, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.136764344754456, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-01-07 07:55:27,236 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5479614 different iterations.  Found a solution that moves 1875 regions; Going from a computed imbalance of 0.9912623983248834 to a new imbalance of 4.376343537466002E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6944444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 07:55:27,244 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1127827889:90 , srv1182888566:90 , srv1247829138:90 , srv136443153:90 , srv1379422117:90 , srv1397746121:90 , srv1590005709:90 , srv1592685614:90 , srv164785903:90 , srv1766664302:90 , srv1774322143:90 , srv2017040839:90 , srv2028429644:90 , srv2112667816:90 , srv249935190:90 , srv277062356:90 , srv320982321:90 , srv332653023:90 , srv361302730:90 , srv368967512:90 , srv406680817:90 , srv526348154:90 , srv645969697:90 , srv664027237:90 , srv821156373:90 , srv859036928:90 , srv860885289:90 , srv867488337:90 , srv904028759:90 , srv973234995:90 }
2024-01-07 07:55:27,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-07 07:55:27,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2112667816=13, srv164785903=8, srv332653023=17, srv1182888566=1, srv136443153=3, srv277062356=15, srv320982321=16, srv1590005709=6, srv973234995=29, srv1379422117=4, srv645969697=22, srv1127827889=0, srv867488337=27, srv361302730=18, srv249935190=14, srv860885289=26, srv526348154=21, srv664027237=23, srv2028429644=12, srv406680817=20, srv368967512=19, srv904028759=28, srv1397746121=5, srv1592685614=7, srv1766664302=9, srv2017040839=11, srv821156373=24, srv859036928=25, srv1247829138=2, srv1774322143=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-07 07:55:27,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-07 07:55:27,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-07 07:55:27,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-07 07:55:27,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-07 07:55:27,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-07 07:55:27,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-07 07:55:27,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-07 07:55:27,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-07 07:55:27,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-07 07:55:27,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-07 07:55:27,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-07 07:55:27,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-07 07:55:27,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-07 07:55:27,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-07 07:55:27,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-07 07:55:27,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-07 07:55:27,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-07 07:55:27,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-07 07:55:27,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-07 07:55:27,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-07 07:55:27,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-07 07:55:27,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-07 07:55:27,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-07 07:55:27,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-07 07:55:27,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-07 07:55:27,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-07 07:55:27,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-07 07:55:27,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-07 07:55:27,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-07 07:55:27,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-07 07:55:27,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-01-07 07:55:27,265 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-07 07:55:27,265 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1127827889:90 , srv1182888566:90 , srv1247829138:90 , srv136443153:90 , srv1379422117:90 , srv1397746121:90 , srv1590005709:90 , srv1592685614:90 , srv164785903:90 , srv1766664302:90 , srv1774322143:90 , srv2017040839:90 , srv2028429644:90 , srv2112667816:90 , srv249935190:90 , srv277062356:90 , srv320982321:90 , srv332653023:90 , srv361302730:90 , srv368967512:90 , srv406680817:90 , srv526348154:90 , srv645969697:90 , srv664027237:90 , srv821156373:90 , srv859036928:90 , srv860885289:90 , srv867488337:90 , srv904028759:90 , srv973234995:90 }
2024-01-07 07:55:27,278 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=234 (was 250), ProcessCount=171 (was 171), AvailableMemoryMB=4585 (was 5321)