2024-01-14 12:53:18,829 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-01-14 12:53:19,136 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-01-14 12:53:19,258 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-01-14 12:53:19,324 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-14 12:53:19,332 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-01-14 12:53:19,345 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=270, ProcessCount=167, AvailableMemoryMB=15655
2024-01-14 12:53:19,347 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-14 12:53:19,348 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-01-14 12:53:19,505 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1010419103:84 , srv1040293972:84 , srv1042863083:84 , srv1080173317:84 , srv1098814130:84 , srv110516319:84 , srv1126417795:84 , srv1135705250:84 , srv1166723891:84 , srv1185398299:84 , srv1208079275:84 , srv1222154051:84 , srv1231889775:84 , srv1241994342:84 , srv13184598:84 , srv1359072074:84 , srv1408769694:84 , srv1434815708:84 , srv1436361316:84 , srv1440795719:84 , srv144946619:84 , srv1451062709:84 , srv1487411533:84 , srv1494982340:84 , srv1512474112:84 , srv1516756773:84 , srv1532004618:84 , srv1571123625:84 , srv1581978829:84 , srv1583036613:84 , srv1590252523:84 , srv1591574578:84 , srv1617432811:84 , srv1674977367:84 , srv1734494479:84 , srv1737022961:84 , srv1738987469:84 , srv174896352:84 , srv1760363979:84 , srv1774753965:84 , srv1801020605:84 , srv1836005714:84 , srv1860685199:84 , srv1891176072:84 , srv1904365002:84 , srv1923927174:84 , srv1927735168:84 , srv1954068424:84 , srv1989899155:84 , srv2019920118:84 , srv2031413057:84 , srv2038428799:84 , srv2057119221:84 , srv2072224834:84 , srv2110486827:84 , srv2112341730:84 , srv217165825:84 , srv254110001:84 , srv267839090:84 , srv271683615:84 , srv290885374:84 , srv317003199:84 , srv326757103:84 , srv397411768:84 , srv408495778:84 , srv425210355:84 , srv44378023:84 , srv507177496:84 , srv542986781:84 , srv589243199:84 , srv59097105:84 , srv603697832:84 , srv620716800:84 , srv621720230:84 , srv634811507:84 , srv64655934:84 , srv658694445:84 , srv714886462:84 , srv730089392:84 , srv755368311:84 , srv756256194:84 , srv756793927:84 , srv765551783:84 , srv769762654:84 , srv798425135:84 , srv828866544:84 , srv834491649:84 , srv839061776:84 , srv841610357:84 , srv844783841:84 , srv862539780:84 , srv887823969:84 , srv89107260:84 , srv920232234:84 , srv925836617:84 , srv927556371:84 , srv951627724:84 , srv97365025:84 , srv988021445:84 , srv1086614602:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-01-14 12:53:19,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-14 12:53:19,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv290885374=61, srv769762654=84, srv1126417795=7, srv1954068424=48, srv2031413057=51, srv1487411533=23, srv326757103=63, srv64655934=76, srv988021445=99, srv1923927174=46, srv828866544=86, srv887823969=92, srv2019920118=50, srv1086614602=4, srv839061776=88, srv13184598=15, srv110516319=6, srv1989899155=49, srv2072224834=54, srv730089392=79, srv1135705250=8, srv1185398299=10, srv765551783=83, srv1737022961=36, srv714886462=78, srv798425135=85, srv97365025=98, srv834491649=87, srv927556371=96, srv254110001=58, srv603697832=72, srv620716800=73, srv144946619=21, srv267839090=59, srv1440795719=20, srv1098814130=5, srv1591574578=32, srv920232234=94, srv862539780=91, srv174896352=38, srv1674977367=34, srv1010419103=0, srv1801020605=41, srv925836617=95, srv271683615=60, srv397411768=64, srv2110486827=55, srv756256194=81, srv1408769694=17, srv2112341730=56, srv755368311=80, srv1241994342=14, srv89107260=93, srv1860685199=43, srv756793927=82, srv59097105=71, srv634811507=75, srv1590252523=31, srv658694445=77, srv217165825=57, srv1080173317=3, srv1512474112=25, srv317003199=62, srv844783841=90, srv1760363979=39, srv2038428799=52, srv1581978829=29, srv44378023=67, srv1222154051=12, srv1532004618=27, srv1836005714=42, srv1571123625=28, srv2057119221=53, srv1231889775=13, srv841610357=89, srv1434815708=18, srv1738987469=37, srv507177496=68, srv1734494479=35, srv1774753965=40, srv408495778=65, srv1516756773=26, srv1040293972=1, srv1904365002=45, srv1166723891=9, srv1359072074=16, srv425210355=66, srv542986781=69, srv589243199=70, srv1891176072=44, srv1042863083=2, srv1208079275=11, srv1583036613=30, srv621720230=74, srv1451062709=22, srv1617432811=33, srv951627724=97, srv1436361316=19, srv1494982340=24, srv1927735168=47} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-14 12:53:19,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-14 12:53:19,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-14 12:53:19,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-14 12:53:19,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-14 12:53:19,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-14 12:53:19,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-14 12:53:19,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-14 12:53:19,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-14 12:53:19,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-14 12:53:19,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-14 12:53:19,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-14 12:53:19,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-14 12:53:19,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-14 12:53:19,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-14 12:53:19,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-14 12:53:19,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-14 12:53:19,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-14 12:53:19,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-14 12:53:19,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-14 12:53:19,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-14 12:53:19,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-14 12:53:19,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-14 12:53:19,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-14 12:53:19,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-14 12:53:19,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-14 12:53:19,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-14 12:53:19,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-14 12:53:19,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-14 12:53:19,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-14 12:53:19,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-14 12:53:19,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-14 12:53:19,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-14 12:53:19,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-14 12:53:19,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-14 12:53:19,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-14 12:53:19,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-14 12:53:19,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-14 12:53:19,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-14 12:53:19,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-14 12:53:19,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-14 12:53:19,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-14 12:53:19,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-14 12:53:19,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-14 12:53:19,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-14 12:53:19,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-14 12:53:19,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-14 12:53:19,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-14 12:53:19,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-14 12:53:19,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-14 12:53:19,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-14 12:53:19,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-14 12:53:19,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-14 12:53:19,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-14 12:53:19,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-14 12:53:19,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-14 12:53:19,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-14 12:53:19,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-14 12:53:19,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-14 12:53:19,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-14 12:53:19,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-14 12:53:19,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-14 12:53:19,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-14 12:53:19,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-14 12:53:19,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-14 12:53:19,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-14 12:53:19,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-14 12:53:19,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-14 12:53:19,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-14 12:53:19,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-14 12:53:19,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-14 12:53:19,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-14 12:53:19,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-14 12:53:19,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-14 12:53:19,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-14 12:53:19,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-14 12:53:19,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-14 12:53:19,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-14 12:53:19,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-14 12:53:19,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-14 12:53:19,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-14 12:53:19,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-14 12:53:19,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-14 12:53:19,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-14 12:53:19,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-14 12:53:19,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-14 12:53:19,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-14 12:53:19,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-14 12:53:19,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-14 12:53:19,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-14 12:53:19,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-14 12:53:19,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-14 12:53:19,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-14 12:53:19,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-14 12:53:19,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-14 12:53:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-14 12:53:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-14 12:53:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-14 12:53:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-14 12:53:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-14 12:53:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-14 12:53:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-14 12:53:19,814 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-14 12:53:19,814 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-14 12:53:19,815 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-14 12:53:20,265 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 602 ms to try 5000 different iterations.  Found a solution that moves 4683 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1832879566712321. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03830131716560836, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05821898364036186, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5203333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03830131716560836, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1785, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24558333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-14 12:53:20,289 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1208079275:81 , srv1516756773:81 , srv798425135:81 , srv1408769694:82 , srv1434815708:82 , srv397411768:82 , srv1010419103:83 , srv1040293972:83 , srv1512474112:83 , srv1904365002:83 , srv2112341730:83 , srv290885374:83 , srv756793927:83 , srv828866544:83 , srv1222154051:84 , srv2019920118:84 , srv2031413057:84 , srv326757103:84 , srv408495778:84 , srv425210355:84 , srv756256194:84 , srv1080173317:85 , srv110516319:85 , srv1126417795:85 , srv1231889775:85 , srv1241994342:85 , srv144946619:85 , srv1583036613:85 , srv1590252523:85 , srv1738987469:85 , srv1860685199:85 , srv1891176072:85 , srv2057119221:85 , srv2110486827:85 , srv271683615:85 , srv603697832:85 , srv844783841:85 , srv887823969:85 , srv1185398299:86 , srv13184598:86 , srv1440795719:86 , srv1581978829:86 , srv1674977367:86 , srv1734494479:86 , srv1737022961:86 , srv1760363979:86 , srv1836005714:86 , srv1923927174:86 , srv317003199:86 , srv589243199:86 , srv621720230:86 , srv755368311:86 , srv89107260:86 , srv988021445:86 , srv1042863083:87 , srv1359072074:87 , srv1801020605:87 , srv1954068424:87 , srv1989899155:87 , srv2038428799:87 , srv267839090:87 , srv634811507:87 , srv658694445:87 , srv714886462:87 , srv730089392:87 , srv765551783:87 , srv927556371:87 , srv1166723891:88 , srv1451062709:88 , srv1774753965:88 , srv1927735168:88 , srv44378023:88 , srv620716800:88 , srv862539780:88 , srv920232234:88 , srv925836617:88 , srv97365025:88 , srv1135705250:89 , srv1494982340:89 , srv1532004618:89 , srv1591574578:89 , srv174896352:89 , srv542986781:89 , srv59097105:89 , srv839061776:89 , srv1098814130:90 , srv1487411533:90 , srv1571123625:90 , srv217165825:90 , srv254110001:90 , srv64655934:90 , srv834491649:90 , srv2072224834:91 , srv951627724:91 , srv841610357:93 , srv1617432811:94 , srv1436361316:95 , srv507177496:95 , srv769762654:96 , srv1086614602:430 }
2024-01-14 12:53:20,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-14 12:53:20,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv290885374=61, srv769762654=84, srv1126417795=7, srv1954068424=48, srv2031413057=51, srv1487411533=23, srv326757103=63, srv64655934=76, srv988021445=99, srv1923927174=46, srv828866544=86, srv887823969=92, srv2019920118=50, srv1086614602=4, srv839061776=88, srv13184598=15, srv110516319=6, srv1989899155=49, srv2072224834=54, srv730089392=79, srv1135705250=8, srv1185398299=10, srv765551783=83, srv1737022961=36, srv714886462=78, srv798425135=85, srv97365025=98, srv834491649=87, srv927556371=96, srv254110001=58, srv603697832=72, srv620716800=73, srv144946619=21, srv267839090=59, srv1440795719=20, srv1098814130=5, srv1591574578=32, srv920232234=94, srv862539780=91, srv174896352=38, srv1674977367=34, srv1010419103=0, srv1801020605=41, srv925836617=95, srv271683615=60, srv397411768=64, srv2110486827=55, srv756256194=81, srv1408769694=17, srv2112341730=56, srv755368311=80, srv1241994342=14, srv89107260=93, srv1860685199=43, srv756793927=82, srv59097105=71, srv634811507=75, srv1590252523=31, srv658694445=77, srv217165825=57, srv1080173317=3, srv1512474112=25, srv317003199=62, srv844783841=90, srv1760363979=39, srv2038428799=52, srv1581978829=29, srv44378023=67, srv1222154051=12, srv1532004618=27, srv1836005714=42, srv1571123625=28, srv2057119221=53, srv1231889775=13, srv841610357=89, srv1434815708=18, srv1738987469=37, srv507177496=68, srv1734494479=35, srv1774753965=40, srv408495778=65, srv1516756773=26, srv1040293972=1, srv1904365002=45, srv1166723891=9, srv1359072074=16, srv425210355=66, srv542986781=69, srv589243199=70, srv1891176072=44, srv1042863083=2, srv1208079275=11, srv1583036613=30, srv621720230=74, srv1451062709=22, srv1617432811=33, srv951627724=97, srv1436361316=19, srv1494982340=24, srv1927735168=47} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-14 12:53:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-14 12:53:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-14 12:53:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-14 12:53:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-14 12:53:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-14 12:53:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-14 12:53:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-14 12:53:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-14 12:53:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-14 12:53:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-14 12:53:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-14 12:53:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-14 12:53:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-14 12:53:20,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-14 12:53:20,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-14 12:53:20,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-14 12:53:20,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-14 12:53:20,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-14 12:53:20,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-14 12:53:20,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-14 12:53:20,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-14 12:53:20,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-14 12:53:20,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-14 12:53:20,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-14 12:53:20,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-14 12:53:20,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-14 12:53:20,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-14 12:53:20,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-14 12:53:20,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-14 12:53:20,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-14 12:53:20,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-14 12:53:20,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-14 12:53:20,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-14 12:53:20,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-14 12:53:20,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-14 12:53:20,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-14 12:53:20,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-14 12:53:20,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-14 12:53:20,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-14 12:53:20,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-14 12:53:20,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-14 12:53:20,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-14 12:53:20,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-14 12:53:20,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-14 12:53:20,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-14 12:53:20,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-14 12:53:20,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-14 12:53:20,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-14 12:53:20,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-14 12:53:20,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-14 12:53:20,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-14 12:53:20,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-14 12:53:20,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-14 12:53:20,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-14 12:53:20,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-14 12:53:20,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-14 12:53:20,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-14 12:53:20,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-14 12:53:20,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-14 12:53:20,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-14 12:53:20,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-14 12:53:20,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-14 12:53:20,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-14 12:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-14 12:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-14 12:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-14 12:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-14 12:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-14 12:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-14 12:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-14 12:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-14 12:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-14 12:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-14 12:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-14 12:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-14 12:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-14 12:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-14 12:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-14 12:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-14 12:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-14 12:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-14 12:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-14 12:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-14 12:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-14 12:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-14 12:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-14 12:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-14 12:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-14 12:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-14 12:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-14 12:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-14 12:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-14 12:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-14 12:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-14 12:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-14 12:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-14 12:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-14 12:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-14 12:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-14 12:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-14 12:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-14 12:53:20,485 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03830131716560836, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05821898364036186, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03830131716560836, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1785, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24558333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-14 12:53:20,485 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-14 12:53:20,486 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18325516560437458, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03830131716560836, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05821898364036186, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03830131716560836, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1785, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24558333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-14 12:53:20,763 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 404 ms to try 5000 different iterations.  Found a solution that moves 2667 regions; Going from a computed imbalance of 0.18325516560437458 to a new imbalance of 0.013066701891459868. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019276086924557137); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03637934565276644, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29633333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019276086924557137); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04125, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-14 12:53:20,776 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1436361316:75 , srv507177496:76 , srv769762654:76 , srv841610357:78 , srv1617432811:79 , srv2072224834:80 , srv1135705250:81 , srv1571123625:81 , srv1487411533:82 , srv1494982340:82 , srv254110001:82 , srv59097105:82 , srv925836617:82 , srv951627724:82 , srv97365025:82 , srv1098814130:83 , srv1532004618:83 , srv1591574578:83 , srv174896352:83 , srv1801020605:83 , srv217165825:83 , srv44378023:83 , srv542986781:83 , srv634811507:83 , srv64655934:83 , srv834491649:83 , srv839061776:83 , srv927556371:83 , srv1166723891:84 , srv620716800:84 , srv714886462:84 , srv730089392:84 , srv1042863083:85 , srv1080173317:85 , srv1451062709:85 , srv1734494479:85 , srv1923927174:85 , srv2038428799:85 , srv2110486827:85 , srv658694445:85 , srv765551783:85 , srv887823969:85 , srv920232234:85 , srv988021445:85 , srv1359072074:86 , srv1440795719:86 , srv1581978829:86 , srv1583036613:86 , srv1590252523:86 , srv1737022961:86 , srv1738987469:86 , srv1927735168:86 , srv267839090:86 , srv425210355:86 , srv603697832:86 , srv621720230:86 , srv755368311:86 , srv756256194:86 , srv110516319:87 , srv1231889775:87 , srv1760363979:87 , srv1774753965:87 , srv1954068424:87 , srv1989899155:87 , srv2031413057:87 , srv589243199:87 , srv756793927:87 , srv862539780:87 , srv1040293972:88 , srv1126417795:88 , srv1185398299:88 , srv13184598:88 , srv144946619:88 , srv1674977367:88 , srv1836005714:88 , srv2019920118:88 , srv271683615:88 , srv290885374:88 , srv326757103:88 , srv844783841:88 , srv89107260:88 , srv1222154051:89 , srv1408769694:89 , srv1860685199:89 , srv2112341730:89 , srv317003199:89 , srv1010419103:90 , srv1512474112:90 , srv1516756773:90 , srv1891176072:90 , srv2057119221:90 , srv828866544:90 , srv1208079275:91 , srv408495778:91 , srv798425135:92 , srv1434815708:93 , srv1904365002:93 , srv397411768:93 , srv1241994342:94 , srv1086614602:515 }
2024-01-14 12:53:20,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-14 12:53:20,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv290885374=61, srv769762654=84, srv1126417795=7, srv1954068424=48, srv2031413057=51, srv1487411533=23, srv326757103=63, srv64655934=76, srv988021445=99, srv1923927174=46, srv828866544=86, srv887823969=92, srv2019920118=50, srv1086614602=4, srv839061776=88, srv13184598=15, srv110516319=6, srv1989899155=49, srv2072224834=54, srv730089392=79, srv1135705250=8, srv1185398299=10, srv765551783=83, srv1737022961=36, srv714886462=78, srv798425135=85, srv97365025=98, srv834491649=87, srv927556371=96, srv254110001=58, srv603697832=72, srv620716800=73, srv144946619=21, srv267839090=59, srv1440795719=20, srv1098814130=5, srv1591574578=32, srv920232234=94, srv862539780=91, srv174896352=38, srv1674977367=34, srv1010419103=0, srv1801020605=41, srv925836617=95, srv271683615=60, srv397411768=64, srv2110486827=55, srv756256194=81, srv1408769694=17, srv2112341730=56, srv755368311=80, srv1241994342=14, srv89107260=93, srv1860685199=43, srv756793927=82, srv59097105=71, srv634811507=75, srv1590252523=31, srv658694445=77, srv217165825=57, srv1080173317=3, srv1512474112=25, srv317003199=62, srv844783841=90, srv1760363979=39, srv2038428799=52, srv1581978829=29, srv44378023=67, srv1222154051=12, srv1532004618=27, srv1836005714=42, srv1571123625=28, srv2057119221=53, srv1231889775=13, srv841610357=89, srv1434815708=18, srv1738987469=37, srv507177496=68, srv1734494479=35, srv1774753965=40, srv408495778=65, srv1516756773=26, srv1040293972=1, srv1904365002=45, srv1166723891=9, srv1359072074=16, srv425210355=66, srv542986781=69, srv589243199=70, srv1891176072=44, srv1042863083=2, srv1208079275=11, srv1583036613=30, srv621720230=74, srv1451062709=22, srv1617432811=33, srv951627724=97, srv1436361316=19, srv1494982340=24, srv1927735168=47} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-14 12:53:20,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-14 12:53:20,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-14 12:53:20,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-14 12:53:20,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-14 12:53:20,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-14 12:53:20,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-14 12:53:20,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-14 12:53:20,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-14 12:53:20,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-14 12:53:20,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-14 12:53:20,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-14 12:53:20,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-14 12:53:20,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-14 12:53:20,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-14 12:53:20,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-14 12:53:20,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-14 12:53:20,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-14 12:53:20,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-14 12:53:20,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-14 12:53:20,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-14 12:53:20,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-14 12:53:20,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-14 12:53:20,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-14 12:53:20,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-14 12:53:20,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-14 12:53:20,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-14 12:53:20,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-14 12:53:20,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-14 12:53:20,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-14 12:53:20,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-14 12:53:20,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-14 12:53:20,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-14 12:53:20,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-14 12:53:20,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-14 12:53:20,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-14 12:53:20,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-14 12:53:20,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-14 12:53:20,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-14 12:53:20,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-14 12:53:20,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-14 12:53:20,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-14 12:53:20,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-14 12:53:20,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-14 12:53:20,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-14 12:53:20,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-14 12:53:20,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-14 12:53:20,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-14 12:53:20,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-14 12:53:20,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-14 12:53:20,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-14 12:53:20,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-14 12:53:20,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-14 12:53:20,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-14 12:53:20,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-14 12:53:20,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-14 12:53:20,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-14 12:53:20,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-14 12:53:20,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-14 12:53:20,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-14 12:53:20,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-14 12:53:20,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-14 12:53:20,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-14 12:53:20,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-14 12:53:20,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-14 12:53:20,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-14 12:53:20,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-14 12:53:20,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-14 12:53:20,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-14 12:53:20,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-14 12:53:20,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-14 12:53:20,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-14 12:53:20,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-14 12:53:20,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-14 12:53:20,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-14 12:53:20,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-14 12:53:20,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-14 12:53:20,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-14 12:53:20,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-14 12:53:20,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-14 12:53:20,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-14 12:53:20,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-14 12:53:20,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-14 12:53:20,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-14 12:53:20,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-14 12:53:20,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-14 12:53:20,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-14 12:53:20,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-14 12:53:20,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-14 12:53:20,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-14 12:53:20,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-14 12:53:20,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-14 12:53:20,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-14 12:53:20,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-14 12:53:20,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-14 12:53:20,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-14 12:53:20,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-14 12:53:20,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-14 12:53:20,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-14 12:53:20,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-14 12:53:20,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-14 12:53:20,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-14 12:53:20,918 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019276086924557137); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03637934565276644, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019276086924557137); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04125, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-14 12:53:20,918 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-14 12:53:20,918 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013048027158316794, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019276086924557137); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03637934565276644, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019276086924557137); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010083333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04125, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-14 12:53:21,257 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 439 ms to try 5000 different iterations.  Found a solution that moves 948 regions; Going from a computed imbalance of 0.013048027158316794 to a new imbalance of 9.804516949433655E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0065515691050215255); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.012659753562742059); MoveCostFunction : (multiplier=7.0, imbalance=0.10533333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0065515691050215255); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-14 12:53:21,259 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1241994342:79 , srv1891176072:81 , srv1904365002:81 , srv317003199:82 , srv1434815708:83 , srv1512474112:83 , srv1674977367:83 , srv1774753965:83 , srv1927735168:83 , srv1954068424:83 , srv1989899155:83 , srv2057119221:83 , srv397411768:83 , srv408495778:83 , srv862539780:83 , srv89107260:83 , srv1098814130:84 , srv1126417795:84 , srv1185398299:84 , srv1222154051:84 , srv13184598:84 , srv1617432811:84 , srv1860685199:84 , srv217165825:84 , srv271683615:84 , srv589243199:84 , srv64655934:84 , srv828866544:84 , srv839061776:84 , srv844783841:84 , srv920232234:84 , srv951627724:84 , srv1010419103:85 , srv1042863083:85 , srv1166723891:85 , srv1359072074:85 , srv1440795719:85 , srv144946619:85 , srv1451062709:85 , srv1494982340:85 , srv1532004618:85 , srv1581978829:85 , srv1591574578:85 , srv1760363979:85 , srv1836005714:85 , srv2112341730:85 , srv254110001:85 , srv267839090:85 , srv326757103:85 , srv542986781:85 , srv620716800:85 , srv658694445:85 , srv755368311:85 , srv765551783:85 , srv769762654:85 , srv798425135:85 , srv834491649:85 , srv841610357:85 , srv1040293972:86 , srv110516319:86 , srv1208079275:86 , srv1231889775:86 , srv1487411533:86 , srv1571123625:86 , srv1583036613:86 , srv1734494479:86 , srv1737022961:86 , srv1738987469:86 , srv174896352:86 , srv2019920118:86 , srv2072224834:86 , srv507177496:86 , srv603697832:86 , srv621720230:86 , srv714886462:86 , srv756793927:86 , srv1080173317:87 , srv1408769694:87 , srv1436361316:87 , srv1516756773:87 , srv1590252523:87 , srv1801020605:87 , srv1923927174:87 , srv2031413057:87 , srv2038428799:87 , srv290885374:87 , srv425210355:87 , srv44378023:87 , srv59097105:87 , srv730089392:87 , srv756256194:87 , srv927556371:87 , srv97365025:87 , srv1135705250:88 , srv2110486827:88 , srv887823969:88 , srv925836617:88 , srv988021445:88 , srv634811507:89 , srv1086614602:571 }
2024-01-14 12:53:21,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-14 12:53:21,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv290885374=61, srv769762654=84, srv1126417795=7, srv1954068424=48, srv2031413057=51, srv1487411533=23, srv326757103=63, srv64655934=76, srv988021445=99, srv1923927174=46, srv828866544=86, srv887823969=92, srv2019920118=50, srv1086614602=4, srv839061776=88, srv13184598=15, srv110516319=6, srv1989899155=49, srv2072224834=54, srv730089392=79, srv1135705250=8, srv1185398299=10, srv765551783=83, srv1737022961=36, srv714886462=78, srv798425135=85, srv97365025=98, srv834491649=87, srv927556371=96, srv254110001=58, srv603697832=72, srv620716800=73, srv144946619=21, srv267839090=59, srv1440795719=20, srv1098814130=5, srv1591574578=32, srv920232234=94, srv862539780=91, srv174896352=38, srv1674977367=34, srv1010419103=0, srv1801020605=41, srv925836617=95, srv271683615=60, srv397411768=64, srv2110486827=55, srv756256194=81, srv1408769694=17, srv2112341730=56, srv755368311=80, srv1241994342=14, srv89107260=93, srv1860685199=43, srv756793927=82, srv59097105=71, srv634811507=75, srv1590252523=31, srv658694445=77, srv217165825=57, srv1080173317=3, srv1512474112=25, srv317003199=62, srv844783841=90, srv1760363979=39, srv2038428799=52, srv1581978829=29, srv44378023=67, srv1222154051=12, srv1532004618=27, srv1836005714=42, srv1571123625=28, srv2057119221=53, srv1231889775=13, srv841610357=89, srv1434815708=18, srv1738987469=37, srv507177496=68, srv1734494479=35, srv1774753965=40, srv408495778=65, srv1516756773=26, srv1040293972=1, srv1904365002=45, srv1166723891=9, srv1359072074=16, srv425210355=66, srv542986781=69, srv589243199=70, srv1891176072=44, srv1042863083=2, srv1208079275=11, srv1583036613=30, srv621720230=74, srv1451062709=22, srv1617432811=33, srv951627724=97, srv1436361316=19, srv1494982340=24, srv1927735168=47} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-14 12:53:21,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-14 12:53:21,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-14 12:53:21,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-14 12:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-14 12:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-14 12:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-14 12:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-14 12:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-14 12:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-14 12:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-14 12:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-14 12:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-14 12:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-14 12:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-14 12:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-14 12:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-14 12:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-14 12:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-14 12:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-14 12:53:21,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-14 12:53:21,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-14 12:53:21,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-14 12:53:21,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-14 12:53:21,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-14 12:53:21,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-14 12:53:21,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-14 12:53:21,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-14 12:53:21,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-14 12:53:21,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-14 12:53:21,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-14 12:53:21,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-14 12:53:21,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-14 12:53:21,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-14 12:53:21,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-14 12:53:21,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-14 12:53:21,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-14 12:53:21,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-14 12:53:21,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-14 12:53:21,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-14 12:53:21,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-14 12:53:21,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-14 12:53:21,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-14 12:53:21,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-14 12:53:21,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-14 12:53:21,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-14 12:53:21,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-14 12:53:21,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-14 12:53:21,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-14 12:53:21,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-14 12:53:21,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-14 12:53:21,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-14 12:53:21,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-14 12:53:21,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-14 12:53:21,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-14 12:53:21,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-14 12:53:21,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-14 12:53:21,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-14 12:53:21,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-14 12:53:21,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-14 12:53:21,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-14 12:53:21,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-14 12:53:21,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-14 12:53:21,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-14 12:53:21,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-14 12:53:21,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-14 12:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-14 12:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-14 12:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-14 12:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-14 12:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-14 12:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-14 12:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-14 12:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-14 12:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-14 12:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-14 12:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-14 12:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-14 12:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-14 12:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-14 12:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-14 12:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-14 12:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-14 12:53:21,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-14 12:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-14 12:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-14 12:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-14 12:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-14 12:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-14 12:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-14 12:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-14 12:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-14 12:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-14 12:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-14 12:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-14 12:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-14 12:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-14 12:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-14 12:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-14 12:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-14 12:53:21,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-14 12:53:21,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-14 12:53:21,392 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0065515691050215255); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.012659753562742059); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0065515691050215255); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-14 12:53:21,393 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-14 12:53:21,393 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.738136570657371E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0065515691050215255); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.012659753562742059); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0065515691050215255); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-14 12:53:21,624 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 334 ms to try 5000 different iterations.  Found a solution that moves 306 regions; Going from a computed imbalance of 9.738136570657371E-4 to a new imbalance of 2.8836259548792116E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.034, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-14 12:53:21,625 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1010419103:83 , srv1737022961:83 , srv1836005714:83 , srv2038428799:83 , srv634811507:83 , srv988021445:83 , srv1135705250:84 , srv1185398299:84 , srv1208079275:84 , srv1231889775:84 , srv1241994342:84 , srv1359072074:84 , srv1408769694:84 , srv1434815708:84 , srv144946619:84 , srv1451062709:84 , srv1487411533:84 , srv1516756773:84 , srv1571123625:84 , srv1591574578:84 , srv174896352:84 , srv1760363979:84 , srv1774753965:84 , srv1860685199:84 , srv1923927174:84 , srv2019920118:84 , srv2031413057:84 , srv2110486827:84 , srv267839090:84 , srv290885374:84 , srv397411768:84 , srv408495778:84 , srv44378023:84 , srv59097105:84 , srv621720230:84 , srv730089392:84 , srv769762654:84 , srv798425135:84 , srv834491649:84 , srv862539780:84 , srv887823969:84 , srv925836617:84 , srv1040293972:85 , srv1042863083:85 , srv1080173317:85 , srv1098814130:85 , srv110516319:85 , srv1126417795:85 , srv1166723891:85 , srv1222154051:85 , srv13184598:85 , srv1440795719:85 , srv1581978829:85 , srv1583036613:85 , srv1590252523:85 , srv1617432811:85 , srv1674977367:85 , srv1734494479:85 , srv1738987469:85 , srv1801020605:85 , srv1904365002:85 , srv1927735168:85 , srv1954068424:85 , srv1989899155:85 , srv2057119221:85 , srv2072224834:85 , srv2112341730:85 , srv217165825:85 , srv254110001:85 , srv271683615:85 , srv317003199:85 , srv326757103:85 , srv425210355:85 , srv507177496:85 , srv542986781:85 , srv589243199:85 , srv620716800:85 , srv64655934:85 , srv658694445:85 , srv714886462:85 , srv755368311:85 , srv756256194:85 , srv756793927:85 , srv765551783:85 , srv839061776:85 , srv844783841:85 , srv89107260:85 , srv920232234:85 , srv927556371:85 , srv951627724:85 , srv97365025:85 , srv1436361316:86 , srv1494982340:86 , srv1512474112:86 , srv1532004618:86 , srv1891176072:86 , srv603697832:86 , srv828866544:86 , srv841610357:86 , srv1086614602:625 }
2024-01-14 12:53:21,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-14 12:53:21,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv290885374=61, srv769762654=84, srv1126417795=7, srv1954068424=48, srv2031413057=51, srv1487411533=23, srv326757103=63, srv64655934=76, srv988021445=99, srv1923927174=46, srv828866544=86, srv887823969=92, srv2019920118=50, srv1086614602=4, srv839061776=88, srv13184598=15, srv110516319=6, srv1989899155=49, srv2072224834=54, srv730089392=79, srv1135705250=8, srv1185398299=10, srv765551783=83, srv1737022961=36, srv714886462=78, srv798425135=85, srv97365025=98, srv834491649=87, srv927556371=96, srv254110001=58, srv603697832=72, srv620716800=73, srv144946619=21, srv267839090=59, srv1440795719=20, srv1098814130=5, srv1591574578=32, srv920232234=94, srv862539780=91, srv174896352=38, srv1674977367=34, srv1010419103=0, srv1801020605=41, srv925836617=95, srv271683615=60, srv397411768=64, srv2110486827=55, srv756256194=81, srv1408769694=17, srv2112341730=56, srv755368311=80, srv1241994342=14, srv89107260=93, srv1860685199=43, srv756793927=82, srv59097105=71, srv634811507=75, srv1590252523=31, srv658694445=77, srv217165825=57, srv1080173317=3, srv1512474112=25, srv317003199=62, srv844783841=90, srv1760363979=39, srv2038428799=52, srv1581978829=29, srv44378023=67, srv1222154051=12, srv1532004618=27, srv1836005714=42, srv1571123625=28, srv2057119221=53, srv1231889775=13, srv841610357=89, srv1434815708=18, srv1738987469=37, srv507177496=68, srv1734494479=35, srv1774753965=40, srv408495778=65, srv1516756773=26, srv1040293972=1, srv1904365002=45, srv1166723891=9, srv1359072074=16, srv425210355=66, srv542986781=69, srv589243199=70, srv1891176072=44, srv1042863083=2, srv1208079275=11, srv1583036613=30, srv621720230=74, srv1451062709=22, srv1617432811=33, srv951627724=97, srv1436361316=19, srv1494982340=24, srv1927735168=47} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-14 12:53:21,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-14 12:53:21,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-14 12:53:21,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-14 12:53:21,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-14 12:53:21,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-14 12:53:21,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-14 12:53:21,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-14 12:53:21,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-14 12:53:21,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-14 12:53:21,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-14 12:53:21,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-14 12:53:21,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-14 12:53:21,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-14 12:53:21,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-14 12:53:21,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-14 12:53:21,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-14 12:53:21,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-14 12:53:21,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-14 12:53:21,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-14 12:53:21,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-14 12:53:21,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-14 12:53:21,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-14 12:53:21,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-14 12:53:21,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-14 12:53:21,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-14 12:53:21,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-14 12:53:21,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-14 12:53:21,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-14 12:53:21,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-14 12:53:21,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-14 12:53:21,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-14 12:53:21,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-14 12:53:21,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-14 12:53:21,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-14 12:53:21,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-14 12:53:21,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-14 12:53:21,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-14 12:53:21,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-14 12:53:21,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-14 12:53:21,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-14 12:53:21,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-14 12:53:21,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-14 12:53:21,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-14 12:53:21,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-14 12:53:21,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-14 12:53:21,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-14 12:53:21,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-14 12:53:21,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-14 12:53:21,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-14 12:53:21,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-14 12:53:21,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-14 12:53:21,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-14 12:53:21,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-14 12:53:21,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-14 12:53:21,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-14 12:53:21,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-14 12:53:21,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-14 12:53:21,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-14 12:53:21,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-14 12:53:21,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-14 12:53:21,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-14 12:53:21,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-14 12:53:21,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-14 12:53:21,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-14 12:53:21,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-14 12:53:21,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-14 12:53:21,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-14 12:53:21,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-14 12:53:21,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-14 12:53:21,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-14 12:53:21,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-14 12:53:21,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-14 12:53:21,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-14 12:53:21,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-14 12:53:21,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-14 12:53:21,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-14 12:53:21,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-14 12:53:21,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-14 12:53:21,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-14 12:53:21,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-14 12:53:21,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-14 12:53:21,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-14 12:53:21,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-14 12:53:21,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-14 12:53:21,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-14 12:53:21,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-14 12:53:21,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-14 12:53:21,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-14 12:53:21,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-14 12:53:21,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-14 12:53:21,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-14 12:53:21,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-14 12:53:21,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-14 12:53:21,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-14 12:53:21,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-14 12:53:21,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-14 12:53:21,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-14 12:53:21,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-14 12:53:21,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-14 12:53:21,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-14 12:53:21,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-14 12:53:21,753 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-14 12:53:21,753 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-14 12:53:21,753 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.6693601752848763E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-14 12:53:21,981 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 327 ms to try 5000 different iterations.  Found a solution that moves 24 regions; Going from a computed imbalance of 2.6693601752848763E-5 to a new imbalance of 8.147741708879284E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0026666666666666666); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-14 12:53:21,981 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv110516319:83 , srv13184598:83 , srv1436361316:83 , srv1532004618:83 , srv1590252523:83 , srv2057119221:83 , srv603697832:83 , srv828866544:83 , srv1040293972:84 , srv1042863083:84 , srv1080173317:84 , srv1098814130:84 , srv1126417795:84 , srv1135705250:84 , srv1166723891:84 , srv1185398299:84 , srv1208079275:84 , srv1222154051:84 , srv1231889775:84 , srv1241994342:84 , srv1359072074:84 , srv1408769694:84 , srv1434815708:84 , srv1440795719:84 , srv144946619:84 , srv1451062709:84 , srv1487411533:84 , srv1494982340:84 , srv1512474112:84 , srv1516756773:84 , srv1581978829:84 , srv1583036613:84 , srv1617432811:84 , srv1674977367:84 , srv1734494479:84 , srv1738987469:84 , srv174896352:84 , srv1760363979:84 , srv1774753965:84 , srv1801020605:84 , srv1836005714:84 , srv1860685199:84 , srv1891176072:84 , srv1904365002:84 , srv1923927174:84 , srv1927735168:84 , srv1954068424:84 , srv1989899155:84 , srv2019920118:84 , srv2031413057:84 , srv2038428799:84 , srv2072224834:84 , srv2110486827:84 , srv2112341730:84 , srv217165825:84 , srv254110001:84 , srv267839090:84 , srv271683615:84 , srv290885374:84 , srv317003199:84 , srv326757103:84 , srv397411768:84 , srv408495778:84 , srv425210355:84 , srv44378023:84 , srv507177496:84 , srv542986781:84 , srv589243199:84 , srv59097105:84 , srv620716800:84 , srv621720230:84 , srv634811507:84 , srv64655934:84 , srv658694445:84 , srv714886462:84 , srv730089392:84 , srv755368311:84 , srv756256194:84 , srv765551783:84 , srv798425135:84 , srv834491649:84 , srv841610357:84 , srv844783841:84 , srv862539780:84 , srv887823969:84 , srv89107260:84 , srv920232234:84 , srv925836617:84 , srv927556371:84 , srv951627724:84 , srv97365025:84 , srv988021445:84 , srv1010419103:85 , srv1571123625:85 , srv1591574578:85 , srv1737022961:85 , srv756793927:85 , srv769762654:85 , srv839061776:85 , srv1086614602:685 }
2024-01-14 12:53:21,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-14 12:53:21,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv290885374=61, srv769762654=84, srv1126417795=7, srv1954068424=48, srv2031413057=51, srv1487411533=23, srv326757103=63, srv64655934=76, srv988021445=99, srv1923927174=46, srv828866544=86, srv887823969=92, srv2019920118=50, srv1086614602=4, srv839061776=88, srv13184598=15, srv110516319=6, srv1989899155=49, srv2072224834=54, srv730089392=79, srv1135705250=8, srv1185398299=10, srv765551783=83, srv1737022961=36, srv714886462=78, srv798425135=85, srv97365025=98, srv834491649=87, srv927556371=96, srv254110001=58, srv603697832=72, srv620716800=73, srv144946619=21, srv267839090=59, srv1440795719=20, srv1098814130=5, srv1591574578=32, srv920232234=94, srv862539780=91, srv174896352=38, srv1674977367=34, srv1010419103=0, srv1801020605=41, srv925836617=95, srv271683615=60, srv397411768=64, srv2110486827=55, srv756256194=81, srv1408769694=17, srv2112341730=56, srv755368311=80, srv1241994342=14, srv89107260=93, srv1860685199=43, srv756793927=82, srv59097105=71, srv634811507=75, srv1590252523=31, srv658694445=77, srv217165825=57, srv1080173317=3, srv1512474112=25, srv317003199=62, srv844783841=90, srv1760363979=39, srv2038428799=52, srv1581978829=29, srv44378023=67, srv1222154051=12, srv1532004618=27, srv1836005714=42, srv1571123625=28, srv2057119221=53, srv1231889775=13, srv841610357=89, srv1434815708=18, srv1738987469=37, srv507177496=68, srv1734494479=35, srv1774753965=40, srv408495778=65, srv1516756773=26, srv1040293972=1, srv1904365002=45, srv1166723891=9, srv1359072074=16, srv425210355=66, srv542986781=69, srv589243199=70, srv1891176072=44, srv1042863083=2, srv1208079275=11, srv1583036613=30, srv621720230=74, srv1451062709=22, srv1617432811=33, srv951627724=97, srv1436361316=19, srv1494982340=24, srv1927735168=47} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-14 12:53:22,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-14 12:53:22,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-14 12:53:22,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-14 12:53:22,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-14 12:53:22,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-14 12:53:22,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-14 12:53:22,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-14 12:53:22,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-14 12:53:22,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-14 12:53:22,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-14 12:53:22,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-14 12:53:22,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-14 12:53:22,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-14 12:53:22,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-14 12:53:22,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-14 12:53:22,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-14 12:53:22,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-14 12:53:22,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-14 12:53:22,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-14 12:53:22,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-14 12:53:22,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-14 12:53:22,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-14 12:53:22,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-14 12:53:22,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-14 12:53:22,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-14 12:53:22,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-14 12:53:22,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-14 12:53:22,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-14 12:53:22,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-14 12:53:22,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-14 12:53:22,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-14 12:53:22,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-14 12:53:22,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-14 12:53:22,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-14 12:53:22,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-14 12:53:22,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-14 12:53:22,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-14 12:53:22,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-14 12:53:22,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-14 12:53:22,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-14 12:53:22,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-14 12:53:22,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-14 12:53:22,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-14 12:53:22,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-14 12:53:22,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-14 12:53:22,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-14 12:53:22,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-14 12:53:22,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-14 12:53:22,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-14 12:53:22,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-14 12:53:22,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-14 12:53:22,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-14 12:53:22,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-14 12:53:22,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-14 12:53:22,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-14 12:53:22,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-14 12:53:22,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-14 12:53:22,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-14 12:53:22,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-14 12:53:22,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-14 12:53:22,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-14 12:53:22,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-14 12:53:22,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-14 12:53:22,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-14 12:53:22,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-14 12:53:22,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-14 12:53:22,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-14 12:53:22,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-14 12:53:22,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-14 12:53:22,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-14 12:53:22,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-14 12:53:22,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-14 12:53:22,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-14 12:53:22,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-14 12:53:22,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-14 12:53:22,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-14 12:53:22,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-14 12:53:22,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-14 12:53:22,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-14 12:53:22,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-14 12:53:22,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-14 12:53:22,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-14 12:53:22,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-14 12:53:22,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-14 12:53:22,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-14 12:53:22,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-14 12:53:22,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-14 12:53:22,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-14 12:53:22,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-14 12:53:22,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-14 12:53:22,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-14 12:53:22,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-14 12:53:22,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-14 12:53:22,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-14 12:53:22,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-14 12:53:22,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-14 12:53:22,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-14 12:53:22,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-14 12:53:22,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-14 12:53:22,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-14 12:53:22,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-14 12:53:22,117 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.979690117040589E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-14 12:53:22,117 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv110516319:83 , srv13184598:83 , srv1436361316:83 , srv1532004618:83 , srv1590252523:83 , srv2057119221:83 , srv603697832:83 , srv828866544:83 , srv1040293972:84 , srv1042863083:84 , srv1080173317:84 , srv1098814130:84 , srv1126417795:84 , srv1135705250:84 , srv1166723891:84 , srv1185398299:84 , srv1208079275:84 , srv1222154051:84 , srv1231889775:84 , srv1241994342:84 , srv1359072074:84 , srv1408769694:84 , srv1434815708:84 , srv1440795719:84 , srv144946619:84 , srv1451062709:84 , srv1487411533:84 , srv1494982340:84 , srv1512474112:84 , srv1516756773:84 , srv1581978829:84 , srv1583036613:84 , srv1617432811:84 , srv1674977367:84 , srv1734494479:84 , srv1738987469:84 , srv174896352:84 , srv1760363979:84 , srv1774753965:84 , srv1801020605:84 , srv1836005714:84 , srv1860685199:84 , srv1891176072:84 , srv1904365002:84 , srv1923927174:84 , srv1927735168:84 , srv1954068424:84 , srv1989899155:84 , srv2019920118:84 , srv2031413057:84 , srv2038428799:84 , srv2072224834:84 , srv2110486827:84 , srv2112341730:84 , srv217165825:84 , srv254110001:84 , srv267839090:84 , srv271683615:84 , srv290885374:84 , srv317003199:84 , srv326757103:84 , srv397411768:84 , srv408495778:84 , srv425210355:84 , srv44378023:84 , srv507177496:84 , srv542986781:84 , srv589243199:84 , srv59097105:84 , srv620716800:84 , srv621720230:84 , srv634811507:84 , srv64655934:84 , srv658694445:84 , srv714886462:84 , srv730089392:84 , srv755368311:84 , srv756256194:84 , srv765551783:84 , srv798425135:84 , srv834491649:84 , srv841610357:84 , srv844783841:84 , srv862539780:84 , srv887823969:84 , srv89107260:84 , srv920232234:84 , srv925836617:84 , srv927556371:84 , srv951627724:84 , srv97365025:84 , srv988021445:84 , srv1010419103:85 , srv1571123625:85 , srv1591574578:85 , srv1737022961:85 , srv756793927:85 , srv769762654:85 , srv839061776:85 , srv1086614602:685 }
2024-01-14 12:53:22,150 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=281 (was 270) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=15025 (was 15655)
2024-01-14 12:53:22,158 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=281, ProcessCount=167, AvailableMemoryMB=15025
2024-01-14 12:53:22,158 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-14 12:53:22,159 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-01-14 12:53:22,175 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1139422152:84 , srv1150567592:84 , srv1177950026:84 , srv1209168198:84 , srv1234220119:84 , srv1250267882:84 , srv1286393996:84 , srv1485744913:84 , srv1565271292:84 , srv1698001589:84 , srv1710524888:84 , srv1782548679:84 , srv1787215206:84 , srv1809507433:84 , srv1890463876:84 , srv1952704138:84 , srv1969700630:84 , srv2006380546:84 , srv2061107914:84 , srv29352229:84 , srv29940592:84 , srv390333331:84 , srv561616968:84 , srv597674115:84 , srv637354131:84 , srv6695947:84 , srv866268371:84 , srv883324476:84 , srv89112498:84 , srv892355969:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-01-14 12:53:22,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-14 12:53:22,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1787215206=12, srv29940592=20, srv1485744913=7, srv637354131=24, srv390333331=21, srv89112498=28, srv1782548679=11, srv597674115=23, srv866268371=26, srv1969700630=16, srv1150567592=1, srv1286393996=6, srv892355969=29, srv1209168198=3, srv1698001589=9, srv29352229=19, srv1139422152=0, srv1565271292=8, srv1809507433=13, srv1952704138=15, srv2006380546=17, srv883324476=27, srv1250267882=5, srv1710524888=10, srv2061107914=18, srv1177950026=2, srv1234220119=4, srv561616968=22, srv1890463876=14, srv6695947=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-14 12:53:22,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-14 12:53:22,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-14 12:53:22,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-14 12:53:22,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-14 12:53:22,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-14 12:53:22,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-14 12:53:22,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-14 12:53:22,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-14 12:53:22,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-14 12:53:22,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-14 12:53:22,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-14 12:53:22,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-14 12:53:22,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-14 12:53:22,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-14 12:53:22,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-14 12:53:22,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-14 12:53:22,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-14 12:53:22,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-14 12:53:22,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-14 12:53:22,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-14 12:53:22,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-14 12:53:22,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-14 12:53:22,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-14 12:53:22,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-14 12:53:22,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-14 12:53:22,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-14 12:53:22,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-14 12:53:22,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-14 12:53:22,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-14 12:53:22,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-14 12:53:22,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-01-14 12:53:22,228 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.120067389074546, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-14 12:53:22,229 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912571371686693, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.120067389074546, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-01-14 12:55:22,203 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5940089 different iterations.  Found a solution that moves 1862 regions; Going from a computed imbalance of 0.9912571371686693 to a new imbalance of 4.346000888939571E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6896296296296296, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-14 12:55:22,211 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1139422152:90 , srv1150567592:90 , srv1177950026:90 , srv1209168198:90 , srv1234220119:90 , srv1250267882:90 , srv1286393996:90 , srv1485744913:90 , srv1565271292:90 , srv1698001589:90 , srv1710524888:90 , srv1782548679:90 , srv1787215206:90 , srv1809507433:90 , srv1890463876:90 , srv1952704138:90 , srv1969700630:90 , srv2006380546:90 , srv2061107914:90 , srv29352229:90 , srv29940592:90 , srv390333331:90 , srv561616968:90 , srv597674115:90 , srv637354131:90 , srv6695947:90 , srv866268371:90 , srv883324476:90 , srv89112498:90 , srv892355969:90 }
2024-01-14 12:55:22,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-14 12:55:22,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1787215206=12, srv29940592=20, srv1485744913=7, srv637354131=24, srv390333331=21, srv89112498=28, srv1782548679=11, srv597674115=23, srv866268371=26, srv1969700630=16, srv1150567592=1, srv1286393996=6, srv892355969=29, srv1209168198=3, srv1698001589=9, srv29352229=19, srv1139422152=0, srv1565271292=8, srv1809507433=13, srv1952704138=15, srv2006380546=17, srv883324476=27, srv1250267882=5, srv1710524888=10, srv2061107914=18, srv1177950026=2, srv1234220119=4, srv561616968=22, srv1890463876=14, srv6695947=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-14 12:55:22,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-14 12:55:22,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-14 12:55:22,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-14 12:55:22,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-14 12:55:22,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-14 12:55:22,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-14 12:55:22,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-14 12:55:22,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-14 12:55:22,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-14 12:55:22,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-14 12:55:22,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-14 12:55:22,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-14 12:55:22,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-14 12:55:22,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-14 12:55:22,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-14 12:55:22,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-14 12:55:22,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-14 12:55:22,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-14 12:55:22,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-14 12:55:22,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-14 12:55:22,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-14 12:55:22,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-14 12:55:22,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-14 12:55:22,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-14 12:55:22,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-14 12:55:22,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-14 12:55:22,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-14 12:55:22,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-14 12:55:22,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-14 12:55:22,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-14 12:55:22,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-01-14 12:55:22,229 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-14 12:55:22,229 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1139422152:90 , srv1150567592:90 , srv1177950026:90 , srv1209168198:90 , srv1234220119:90 , srv1250267882:90 , srv1286393996:90 , srv1485744913:90 , srv1565271292:90 , srv1698001589:90 , srv1710524888:90 , srv1782548679:90 , srv1787215206:90 , srv1809507433:90 , srv1890463876:90 , srv1952704138:90 , srv1969700630:90 , srv2006380546:90 , srv2061107914:90 , srv29352229:90 , srv29940592:90 , srv390333331:90 , srv561616968:90 , srv597674115:90 , srv637354131:90 , srv6695947:90 , srv866268371:90 , srv883324476:90 , srv89112498:90 , srv892355969:90 }
2024-01-14 12:55:22,245 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=454 (was 281) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 167) - ProcessCount LEAK? -, AvailableMemoryMB=14069 (was 15025)