2023-11-18 00:55:13,892 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-11-18 00:55:14,165 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-11-18 00:55:14,262 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-11-18 00:55:14,328 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-18 00:55:14,335 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-18 00:55:14,348 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=130, ProcessCount=167, AvailableMemoryMB=10226
2023-11-18 00:55:14,349 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-18 00:55:14,350 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-18 00:55:14,482 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1037299347:84 , srv104947518:84 , srv108586115:84 , srv1108926886:84 , srv115575050:84 , srv1181686098:84 , srv1194454453:84 , srv1211151865:84 , srv122662167:84 , srv1230369895:84 , srv1244348113:84 , srv1258476300:84 , srv1316659437:84 , srv1328598336:84 , srv1334452456:84 , srv1366191115:84 , srv1393219469:84 , srv1412888676:84 , srv1414810013:84 , srv1497375767:84 , srv1499072680:84 , srv1535667581:84 , srv1633192690:84 , srv1639311722:84 , srv1663468531:84 , srv1668368127:84 , srv1706288343:84 , srv1736012805:84 , srv1739877607:84 , srv1743375114:84 , srv1763732571:84 , srv1770984919:84 , srv1783651661:84 , srv1784940698:84 , srv1823792578:84 , srv1831236819:84 , srv1848638808:84 , srv1855919477:84 , srv1887524330:84 , srv1890378508:84 , srv1905999479:84 , srv1949128609:84 , srv1950363818:84 , srv1958647553:84 , srv196217858:84 , srv1962377826:84 , srv1972511279:84 , srv1972710915:84 , srv197786008:84 , srv1980427842:84 , srv2022295039:84 , srv2024633220:84 , srv2027343538:84 , srv2035644295:84 , srv2063455666:84 , srv2076974146:84 , srv2078403920:84 , srv2111368570:84 , srv2115346705:84 , srv2130655033:84 , srv236699962:84 , srv241115300:84 , srv241690310:84 , srv263185745:84 , srv322600707:84 , srv326535906:84 , srv342612672:84 , srv349656606:84 , srv40529009:84 , srv444353487:84 , srv476671664:84 , srv523178050:84 , srv585864537:84 , srv588525610:84 , srv616935826:84 , srv646323014:84 , srv650722868:84 , srv682018230:84 , srv737595659:84 , srv749597331:84 , srv750476644:84 , srv758419311:84 , srv761051826:84 , srv785751649:84 , srv794450782:84 , srv82226081:84 , srv852804053:84 , srv87687647:84 , srv904688275:84 , srv917057313:84 , srv924252809:84 , srv935363611:84 , srv938300776:84 , srv946322211:84 , srv952725758:84 , srv963644934:84 , srv965610976:84 , srv971975363:84 , srv976920049:84 , srv1595011882:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-11-18 00:55:14,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-18 00:55:14,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2035644295=54, srv616935826=75, srv1108926886=3, srv241115300=62, srv2024633220=52, srv1535667581=21, srv196217858=45, srv2076974146=56, srv1905999479=41, srv87687647=88, srv2111368570=58, srv976920049=99, srv523178050=72, srv952725758=95, srv1739877607=29, srv737595659=79, srv749597331=80, srv1497375767=19, srv1962377826=46, srv588525610=74, srv682018230=78, srv1334452456=14, srv650722868=77, srv476671664=71, srv1887524330=39, srv1412888676=17, srv2063455666=55, srv1595011882=22, srv1980427842=50, srv1639311722=24, srv852804053=87, srv1633192690=23, srv904688275=89, srv965610976=97, srv1316659437=12, srv1414810013=18, srv1743375114=30, srv1211151865=7, srv646323014=76, srv785751649=84, srv1194454453=6, srv197786008=49, srv761051826=83, srv1972511279=47, srv750476644=81, srv1950363818=43, srv263185745=64, srv1972710915=48, srv1949128609=42, srv917057313=90, srv1823792578=35, srv241690310=63, srv1328598336=13, srv938300776=93, srv1366191115=15, srv1890378508=40, srv82226081=86, srv946322211=94, srv1706288343=27, srv758419311=82, srv2130655033=60, srv1736012805=28, srv322600707=65, srv1855919477=38, srv2115346705=59, srv1181686098=5, srv122662167=8, srv1783651661=33, srv1258476300=11, srv1770984919=32, srv1230369895=9, srv108586115=2, srv1668368127=26, srv326535906=66, srv349656606=68, srv963644934=96, srv342612672=67, srv2022295039=51, srv1037299347=0, srv1848638808=37, srv1958647553=44, srv924252809=91, srv115575050=4, srv40529009=69, srv585864537=73, srv1763732571=31, srv1244348113=10, srv2027343538=53, srv1393219469=16, srv971975363=98, srv794450782=85, srv444353487=70, srv935363611=92, srv104947518=1, srv236699962=61, srv2078403920=57, srv1831236819=36, srv1784940698=34, srv1499072680=20, srv1663468531=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-18 00:55:14,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-18 00:55:14,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-18 00:55:14,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-18 00:55:14,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-18 00:55:14,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-18 00:55:14,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-18 00:55:14,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-18 00:55:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-18 00:55:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-18 00:55:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-18 00:55:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-18 00:55:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-18 00:55:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-18 00:55:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-18 00:55:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-18 00:55:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-18 00:55:14,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-18 00:55:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-18 00:55:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-18 00:55:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-18 00:55:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-18 00:55:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-18 00:55:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-18 00:55:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-18 00:55:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-18 00:55:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-18 00:55:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-18 00:55:14,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-18 00:55:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-18 00:55:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-18 00:55:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-18 00:55:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-18 00:55:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-18 00:55:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-18 00:55:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-18 00:55:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-18 00:55:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-18 00:55:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-18 00:55:14,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-18 00:55:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-18 00:55:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-18 00:55:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-18 00:55:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-18 00:55:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-18 00:55:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-18 00:55:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-18 00:55:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-18 00:55:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-18 00:55:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-18 00:55:14,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-18 00:55:14,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-18 00:55:14,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-18 00:55:14,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-18 00:55:14,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-18 00:55:14,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-18 00:55:14,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-18 00:55:14,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-18 00:55:14,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-18 00:55:14,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-18 00:55:14,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-18 00:55:14,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-18 00:55:14,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-18 00:55:14,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-18 00:55:14,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-18 00:55:14,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-18 00:55:14,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-18 00:55:14,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-18 00:55:14,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-18 00:55:14,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-18 00:55:14,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-18 00:55:14,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-18 00:55:14,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-18 00:55:14,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-18 00:55:14,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-18 00:55:14,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-18 00:55:14,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-18 00:55:14,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-18 00:55:14,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-18 00:55:14,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-18 00:55:14,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-18 00:55:14,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-18 00:55:14,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-18 00:55:14,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-18 00:55:14,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-18 00:55:14,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-18 00:55:14,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-18 00:55:14,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-18 00:55:14,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-18 00:55:14,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-18 00:55:14,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-18 00:55:14,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-18 00:55:14,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-18 00:55:14,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-18 00:55:14,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-18 00:55:14,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-18 00:55:14,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-18 00:55:14,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-18 00:55:14,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-18 00:55:14,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-18 00:55:14,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-18 00:55:14,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-18 00:55:14,763 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-18 00:55:14,763 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-18 00:55:14,764 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-18 00:55:15,134 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 523 ms to try 5000 different iterations.  Found a solution that moves 4715 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17658853055764437. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03924369172322272, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05552322291463531, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5238888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03924369172322272, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.172, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23625, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-18 00:55:15,151 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2027343538:81 , srv1905999479:82 , srv349656606:82 , srv1037299347:83 , srv1770984919:83 , srv342612672:83 , srv122662167:84 , srv1244348113:84 , srv1663468531:84 , srv1831236819:84 , srv1958647553:84 , srv241690310:84 , srv322600707:84 , srv750476644:84 , srv938300776:84 , srv1108926886:85 , srv115575050:85 , srv1497375767:85 , srv1535667581:85 , srv1706288343:85 , srv1736012805:85 , srv1739877607:85 , srv1823792578:85 , srv1848638808:85 , srv1949128609:85 , srv2024633220:85 , srv2063455666:85 , srv2130655033:85 , srv236699962:85 , srv263185745:85 , srv646323014:85 , srv737595659:85 , srv935363611:85 , srv963644934:85 , srv1258476300:86 , srv1328598336:86 , srv1334452456:86 , srv1412888676:86 , srv1639311722:86 , srv1743375114:86 , srv1763732571:86 , srv1887524330:86 , srv1950363818:86 , srv1972511279:86 , srv1980427842:86 , srv2035644295:86 , srv2078403920:86 , srv326535906:86 , srv616935826:86 , srv682018230:86 , srv749597331:86 , srv917057313:86 , srv946322211:86 , srv952725758:86 , srv1194454453:87 , srv1366191115:87 , srv1633192690:87 , srv1783651661:87 , srv1784940698:87 , srv196217858:87 , srv1962377826:87 , srv197786008:87 , srv2022295039:87 , srv2111368570:87 , srv476671664:87 , srv588525610:87 , srv650722868:87 , srv758419311:87 , srv761051826:87 , srv785751649:87 , srv904688275:87 , srv108586115:88 , srv1211151865:88 , srv1316659437:88 , srv444353487:88 , srv585864537:88 , srv852804053:88 , srv87687647:88 , srv924252809:88 , srv965610976:88 , srv971975363:88 , srv976920049:88 , srv104947518:89 , srv1181686098:89 , srv1393219469:89 , srv1855919477:89 , srv1890378508:89 , srv2115346705:89 , srv241115300:89 , srv794450782:89 , srv82226081:89 , srv1414810013:90 , srv1972710915:90 , srv40529009:90 , srv1499072680:91 , srv1668368127:91 , srv523178050:91 , srv2076974146:92 , srv1230369895:93 , srv1595011882:439 }
2023-11-18 00:55:15,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-18 00:55:15,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2035644295=54, srv616935826=75, srv1108926886=3, srv241115300=62, srv2024633220=52, srv1535667581=21, srv196217858=45, srv2076974146=56, srv1905999479=41, srv87687647=88, srv2111368570=58, srv976920049=99, srv523178050=72, srv952725758=95, srv1739877607=29, srv737595659=79, srv749597331=80, srv1497375767=19, srv1962377826=46, srv588525610=74, srv682018230=78, srv1334452456=14, srv650722868=77, srv476671664=71, srv1887524330=39, srv1412888676=17, srv2063455666=55, srv1595011882=22, srv1980427842=50, srv1639311722=24, srv852804053=87, srv1633192690=23, srv904688275=89, srv965610976=97, srv1316659437=12, srv1414810013=18, srv1743375114=30, srv1211151865=7, srv646323014=76, srv785751649=84, srv1194454453=6, srv197786008=49, srv761051826=83, srv1972511279=47, srv750476644=81, srv1950363818=43, srv263185745=64, srv1972710915=48, srv1949128609=42, srv917057313=90, srv1823792578=35, srv241690310=63, srv1328598336=13, srv938300776=93, srv1366191115=15, srv1890378508=40, srv82226081=86, srv946322211=94, srv1706288343=27, srv758419311=82, srv2130655033=60, srv1736012805=28, srv322600707=65, srv1855919477=38, srv2115346705=59, srv1181686098=5, srv122662167=8, srv1783651661=33, srv1258476300=11, srv1770984919=32, srv1230369895=9, srv108586115=2, srv1668368127=26, srv326535906=66, srv349656606=68, srv963644934=96, srv342612672=67, srv2022295039=51, srv1037299347=0, srv1848638808=37, srv1958647553=44, srv924252809=91, srv115575050=4, srv40529009=69, srv585864537=73, srv1763732571=31, srv1244348113=10, srv2027343538=53, srv1393219469=16, srv971975363=98, srv794450782=85, srv444353487=70, srv935363611=92, srv104947518=1, srv236699962=61, srv2078403920=57, srv1831236819=36, srv1784940698=34, srv1499072680=20, srv1663468531=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-18 00:55:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-18 00:55:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-18 00:55:15,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-18 00:55:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-18 00:55:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-18 00:55:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-18 00:55:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-18 00:55:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-18 00:55:15,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-18 00:55:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-18 00:55:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-18 00:55:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-18 00:55:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-18 00:55:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-18 00:55:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-18 00:55:15,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-18 00:55:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-18 00:55:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-18 00:55:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-18 00:55:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-18 00:55:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-18 00:55:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-18 00:55:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-18 00:55:15,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-18 00:55:15,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-18 00:55:15,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-18 00:55:15,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-18 00:55:15,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-18 00:55:15,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-18 00:55:15,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-18 00:55:15,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-18 00:55:15,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-18 00:55:15,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-18 00:55:15,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-18 00:55:15,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-18 00:55:15,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-18 00:55:15,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-18 00:55:15,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-18 00:55:15,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-18 00:55:15,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-18 00:55:15,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-18 00:55:15,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-18 00:55:15,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-18 00:55:15,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-18 00:55:15,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-18 00:55:15,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-18 00:55:15,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-18 00:55:15,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-18 00:55:15,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-18 00:55:15,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-18 00:55:15,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-18 00:55:15,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-18 00:55:15,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-18 00:55:15,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-18 00:55:15,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-18 00:55:15,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-18 00:55:15,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-18 00:55:15,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-18 00:55:15,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-18 00:55:15,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-18 00:55:15,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-18 00:55:15,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-18 00:55:15,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-18 00:55:15,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-18 00:55:15,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-18 00:55:15,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-18 00:55:15,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-18 00:55:15,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-18 00:55:15,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-18 00:55:15,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-18 00:55:15,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-18 00:55:15,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-18 00:55:15,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-18 00:55:15,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-18 00:55:15,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-18 00:55:15,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-18 00:55:15,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-18 00:55:15,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-18 00:55:15,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-18 00:55:15,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-18 00:55:15,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-18 00:55:15,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-18 00:55:15,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-18 00:55:15,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-18 00:55:15,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-18 00:55:15,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-18 00:55:15,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-18 00:55:15,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-18 00:55:15,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-18 00:55:15,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-18 00:55:15,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-18 00:55:15,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-18 00:55:15,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-18 00:55:15,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-18 00:55:15,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-18 00:55:15,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-18 00:55:15,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-18 00:55:15,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-18 00:55:15,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-18 00:55:15,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-18 00:55:15,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-18 00:55:15,333 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03924369172322272, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05552322291463531, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03924369172322272, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.172, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23625, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-18 00:55:15,334 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-18 00:55:15,334 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17655551542199774, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03924369172322272, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05552322291463531, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03924369172322272, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.172, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23625, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-18 00:55:15,552 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 331 ms to try 5000 different iterations.  Found a solution that moves 2589 regions; Going from a computed imbalance of 0.17655551542199774 to a new imbalance of 0.015383613091299738. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020255876584992862); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03683006140532608, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2876666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020255876584992862); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04525, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-18 00:55:15,561 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1230369895:78 , srv1668368127:80 , srv2076974146:80 , srv1499072680:81 , srv1855919477:81 , srv523178050:81 , srv1414810013:82 , srv40529009:82 , srv761051826:82 , srv852804053:82 , srv976920049:82 , srv1181686098:83 , srv1194454453:83 , srv1890378508:83 , srv241115300:83 , srv108586115:84 , srv1258476300:84 , srv1316659437:84 , srv1393219469:84 , srv1412888676:84 , srv1783651661:84 , srv196217858:84 , srv2078403920:84 , srv2111368570:84 , srv2115346705:84 , srv444353487:84 , srv650722868:84 , srv794450782:84 , srv82226081:84 , srv87687647:84 , srv904688275:84 , srv924252809:84 , srv952725758:84 , srv971975363:84 , srv1211151865:85 , srv1950363818:85 , srv1962377826:85 , srv1972511279:85 , srv1972710915:85 , srv2022295039:85 , srv2035644295:85 , srv2063455666:85 , srv585864537:85 , srv588525610:85 , srv965610976:85 , srv104947518:86 , srv122662167:86 , srv1639311722:86 , srv1743375114:86 , srv1763732571:86 , srv1784940698:86 , srv1887524330:86 , srv1949128609:86 , srv236699962:86 , srv263185745:86 , srv476671664:86 , srv758419311:86 , srv917057313:86 , srv115575050:87 , srv1244348113:87 , srv1328598336:87 , srv1334452456:87 , srv1366191115:87 , srv1497375767:87 , srv1633192690:87 , srv1663468531:87 , srv1848638808:87 , srv1958647553:87 , srv197786008:87 , srv2024633220:87 , srv2130655033:87 , srv326535906:87 , srv616935826:87 , srv682018230:87 , srv749597331:87 , srv935363611:87 , srv1535667581:88 , srv1736012805:88 , srv1739877607:88 , srv1823792578:88 , srv1980427842:88 , srv241690310:88 , srv737595659:88 , srv750476644:88 , srv785751649:88 , srv938300776:88 , srv946322211:88 , srv1037299347:89 , srv1770984919:89 , srv342612672:89 , srv1108926886:90 , srv1706288343:90 , srv1831236819:90 , srv1905999479:90 , srv322600707:90 , srv646323014:90 , srv963644934:90 , srv2027343538:91 , srv349656606:91 , srv1595011882:515 }
2023-11-18 00:55:15,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-18 00:55:15,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2035644295=54, srv616935826=75, srv1108926886=3, srv241115300=62, srv2024633220=52, srv1535667581=21, srv196217858=45, srv2076974146=56, srv1905999479=41, srv87687647=88, srv2111368570=58, srv976920049=99, srv523178050=72, srv952725758=95, srv1739877607=29, srv737595659=79, srv749597331=80, srv1497375767=19, srv1962377826=46, srv588525610=74, srv682018230=78, srv1334452456=14, srv650722868=77, srv476671664=71, srv1887524330=39, srv1412888676=17, srv2063455666=55, srv1595011882=22, srv1980427842=50, srv1639311722=24, srv852804053=87, srv1633192690=23, srv904688275=89, srv965610976=97, srv1316659437=12, srv1414810013=18, srv1743375114=30, srv1211151865=7, srv646323014=76, srv785751649=84, srv1194454453=6, srv197786008=49, srv761051826=83, srv1972511279=47, srv750476644=81, srv1950363818=43, srv263185745=64, srv1972710915=48, srv1949128609=42, srv917057313=90, srv1823792578=35, srv241690310=63, srv1328598336=13, srv938300776=93, srv1366191115=15, srv1890378508=40, srv82226081=86, srv946322211=94, srv1706288343=27, srv758419311=82, srv2130655033=60, srv1736012805=28, srv322600707=65, srv1855919477=38, srv2115346705=59, srv1181686098=5, srv122662167=8, srv1783651661=33, srv1258476300=11, srv1770984919=32, srv1230369895=9, srv108586115=2, srv1668368127=26, srv326535906=66, srv349656606=68, srv963644934=96, srv342612672=67, srv2022295039=51, srv1037299347=0, srv1848638808=37, srv1958647553=44, srv924252809=91, srv115575050=4, srv40529009=69, srv585864537=73, srv1763732571=31, srv1244348113=10, srv2027343538=53, srv1393219469=16, srv971975363=98, srv794450782=85, srv444353487=70, srv935363611=92, srv104947518=1, srv236699962=61, srv2078403920=57, srv1831236819=36, srv1784940698=34, srv1499072680=20, srv1663468531=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-18 00:55:15,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-18 00:55:15,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-18 00:55:15,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-18 00:55:15,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-18 00:55:15,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-18 00:55:15,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-18 00:55:15,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-18 00:55:15,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-18 00:55:15,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-18 00:55:15,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-18 00:55:15,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-18 00:55:15,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-18 00:55:15,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-18 00:55:15,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-18 00:55:15,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-18 00:55:15,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-18 00:55:15,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-18 00:55:15,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-18 00:55:15,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-18 00:55:15,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-18 00:55:15,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-18 00:55:15,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-18 00:55:15,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-18 00:55:15,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-18 00:55:15,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-18 00:55:15,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-18 00:55:15,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-18 00:55:15,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-18 00:55:15,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-18 00:55:15,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-18 00:55:15,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-18 00:55:15,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-18 00:55:15,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-18 00:55:15,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-18 00:55:15,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-18 00:55:15,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-18 00:55:15,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-18 00:55:15,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-18 00:55:15,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-18 00:55:15,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-18 00:55:15,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-18 00:55:15,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-18 00:55:15,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-18 00:55:15,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-18 00:55:15,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-18 00:55:15,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-18 00:55:15,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-18 00:55:15,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-18 00:55:15,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-18 00:55:15,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-18 00:55:15,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-18 00:55:15,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-18 00:55:15,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-18 00:55:15,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-18 00:55:15,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-18 00:55:15,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-18 00:55:15,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-18 00:55:15,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-18 00:55:15,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-18 00:55:15,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-18 00:55:15,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-18 00:55:15,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-18 00:55:15,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-18 00:55:15,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-18 00:55:15,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-18 00:55:15,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-18 00:55:15,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-18 00:55:15,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-18 00:55:15,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-18 00:55:15,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-18 00:55:15,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-18 00:55:15,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-18 00:55:15,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-18 00:55:15,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-18 00:55:15,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-18 00:55:15,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-18 00:55:15,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-18 00:55:15,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-18 00:55:15,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-18 00:55:15,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-18 00:55:15,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-18 00:55:15,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-18 00:55:15,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-18 00:55:15,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-18 00:55:15,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-18 00:55:15,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-18 00:55:15,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-18 00:55:15,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-18 00:55:15,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-18 00:55:15,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-18 00:55:15,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-18 00:55:15,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-18 00:55:15,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-18 00:55:15,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-18 00:55:15,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-18 00:55:15,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-18 00:55:15,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-18 00:55:15,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-18 00:55:15,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-18 00:55:15,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-18 00:55:15,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-18 00:55:15,770 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020255876584992862); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03683006140532608, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020255876584992862); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04525, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-18 00:55:15,771 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-18 00:55:15,771 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015365484525830137, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020255876584992862); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03683006140532608, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020255876584992862); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04525, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-18 00:55:16,007 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 388 ms to try 5000 different iterations.  Found a solution that moves 970 regions; Going from a computed imbalance of 0.015365484525830137 to a new imbalance of 6.912344041760062E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005445601013231794); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009558139185602919); MoveCostFunction : (multiplier=7.0, imbalance=0.10777777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005445601013231794); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-18 00:55:16,008 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv785751649:81 , srv646323014:82 , srv963644934:82 , srv1706288343:83 , srv1972710915:83 , srv322600707:83 , srv585864537:83 , srv946322211:83 , srv104947518:84 , srv1108926886:84 , srv1328598336:84 , srv1366191115:84 , srv1393219469:84 , srv1535667581:84 , srv1633192690:84 , srv1639311722:84 , srv1739877607:84 , srv1784940698:84 , srv1890378508:84 , srv197786008:84 , srv1980427842:84 , srv326535906:84 , srv349656606:84 , srv476671664:84 , srv616935826:84 , srv682018230:84 , srv749597331:84 , srv758419311:84 , srv82226081:84 , srv965610976:84 , srv1037299347:85 , srv108586115:85 , srv1211151865:85 , srv1334452456:85 , srv1736012805:85 , srv1743375114:85 , srv1770984919:85 , srv1823792578:85 , srv1831236819:85 , srv1848638808:85 , srv1887524330:85 , srv1905999479:85 , srv1949128609:85 , srv2027343538:85 , srv2076974146:85 , srv342612672:85 , srv444353487:85 , srv523178050:85 , srv737595659:85 , srv938300776:85 , srv115575050:86 , srv1181686098:86 , srv1230369895:86 , srv1316659437:86 , srv1414810013:86 , srv1499072680:86 , srv1668368127:86 , srv1763732571:86 , srv1855919477:86 , srv1958647553:86 , srv196217858:86 , srv1962377826:86 , srv1972511279:86 , srv2022295039:86 , srv2024633220:86 , srv2035644295:86 , srv2115346705:86 , srv2130655033:86 , srv241115300:86 , srv241690310:86 , srv263185745:86 , srv588525610:86 , srv750476644:86 , srv794450782:86 , srv87687647:86 , srv904688275:86 , srv917057313:86 , srv924252809:86 , srv935363611:86 , srv971975363:86 , srv1244348113:87 , srv1258476300:87 , srv1412888676:87 , srv1497375767:87 , srv1663468531:87 , srv1783651661:87 , srv1950363818:87 , srv2078403920:87 , srv2111368570:87 , srv236699962:87 , srv40529009:87 , srv650722868:87 , srv976920049:87 , srv1194454453:88 , srv122662167:88 , srv2063455666:88 , srv852804053:88 , srv952725758:88 , srv761051826:89 , srv1595011882:552 }
2023-11-18 00:55:16,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-18 00:55:16,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2035644295=54, srv616935826=75, srv1108926886=3, srv241115300=62, srv2024633220=52, srv1535667581=21, srv196217858=45, srv2076974146=56, srv1905999479=41, srv87687647=88, srv2111368570=58, srv976920049=99, srv523178050=72, srv952725758=95, srv1739877607=29, srv737595659=79, srv749597331=80, srv1497375767=19, srv1962377826=46, srv588525610=74, srv682018230=78, srv1334452456=14, srv650722868=77, srv476671664=71, srv1887524330=39, srv1412888676=17, srv2063455666=55, srv1595011882=22, srv1980427842=50, srv1639311722=24, srv852804053=87, srv1633192690=23, srv904688275=89, srv965610976=97, srv1316659437=12, srv1414810013=18, srv1743375114=30, srv1211151865=7, srv646323014=76, srv785751649=84, srv1194454453=6, srv197786008=49, srv761051826=83, srv1972511279=47, srv750476644=81, srv1950363818=43, srv263185745=64, srv1972710915=48, srv1949128609=42, srv917057313=90, srv1823792578=35, srv241690310=63, srv1328598336=13, srv938300776=93, srv1366191115=15, srv1890378508=40, srv82226081=86, srv946322211=94, srv1706288343=27, srv758419311=82, srv2130655033=60, srv1736012805=28, srv322600707=65, srv1855919477=38, srv2115346705=59, srv1181686098=5, srv122662167=8, srv1783651661=33, srv1258476300=11, srv1770984919=32, srv1230369895=9, srv108586115=2, srv1668368127=26, srv326535906=66, srv349656606=68, srv963644934=96, srv342612672=67, srv2022295039=51, srv1037299347=0, srv1848638808=37, srv1958647553=44, srv924252809=91, srv115575050=4, srv40529009=69, srv585864537=73, srv1763732571=31, srv1244348113=10, srv2027343538=53, srv1393219469=16, srv971975363=98, srv794450782=85, srv444353487=70, srv935363611=92, srv104947518=1, srv236699962=61, srv2078403920=57, srv1831236819=36, srv1784940698=34, srv1499072680=20, srv1663468531=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-18 00:55:16,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-18 00:55:16,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-18 00:55:16,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-18 00:55:16,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-18 00:55:16,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-18 00:55:16,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-18 00:55:16,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-18 00:55:16,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-18 00:55:16,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-18 00:55:16,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-18 00:55:16,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-18 00:55:16,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-18 00:55:16,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-18 00:55:16,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-18 00:55:16,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-18 00:55:16,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-18 00:55:16,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-18 00:55:16,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-18 00:55:16,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-18 00:55:16,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-18 00:55:16,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-18 00:55:16,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-18 00:55:16,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-18 00:55:16,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-18 00:55:16,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-18 00:55:16,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-18 00:55:16,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-18 00:55:16,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-18 00:55:16,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-18 00:55:16,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-18 00:55:16,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-18 00:55:16,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-18 00:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-18 00:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-18 00:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-18 00:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-18 00:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-18 00:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-18 00:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-18 00:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-18 00:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-18 00:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-18 00:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-18 00:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-18 00:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-18 00:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-18 00:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-18 00:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-18 00:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-18 00:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-18 00:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-18 00:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-18 00:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-18 00:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-18 00:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-18 00:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-18 00:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-18 00:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-18 00:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-18 00:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-18 00:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-18 00:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-18 00:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-18 00:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-18 00:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-18 00:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-18 00:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-18 00:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-18 00:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-18 00:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-18 00:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-18 00:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-18 00:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-18 00:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-18 00:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-18 00:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-18 00:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-18 00:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-18 00:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-18 00:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-18 00:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-18 00:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-18 00:55:16,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-18 00:55:16,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-18 00:55:16,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-18 00:55:16,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-18 00:55:16,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-18 00:55:16,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-18 00:55:16,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-18 00:55:16,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-18 00:55:16,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-18 00:55:16,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-18 00:55:16,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-18 00:55:16,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-18 00:55:16,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-18 00:55:16,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-18 00:55:16,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-18 00:55:16,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-18 00:55:16,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-18 00:55:16,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-18 00:55:16,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-18 00:55:16,148 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005445601013231794); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009558139185602919); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005445601013231794); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-18 00:55:16,149 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-18 00:55:16,149 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.844423190058589E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005445601013231794); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009558139185602919); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005445601013231794); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-18 00:55:16,552 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 511 ms to try 5000 different iterations.  Found a solution that moves 234 regions; Going from a computed imbalance of 6.844423190058589E-4 to a new imbalance of 1.9132985512243616E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.026, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-18 00:55:16,552 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv104947518:83 , srv1108926886:83 , srv1194454453:83 , srv1497375767:83 , srv1831236819:83 , srv2115346705:83 , srv241115300:83 , srv40529009:83 , srv650722868:83 , srv917057313:83 , srv115575050:84 , srv1181686098:84 , srv122662167:84 , srv1244348113:84 , srv1316659437:84 , srv1334452456:84 , srv1366191115:84 , srv1414810013:84 , srv1499072680:84 , srv1633192690:84 , srv1663468531:84 , srv1706288343:84 , srv1736012805:84 , srv1763732571:84 , srv1783651661:84 , srv1823792578:84 , srv1950363818:84 , srv1958647553:84 , srv1962377826:84 , srv1972710915:84 , srv1980427842:84 , srv2022295039:84 , srv2024633220:84 , srv2035644295:84 , srv2063455666:84 , srv2111368570:84 , srv2130655033:84 , srv236699962:84 , srv241690310:84 , srv588525610:84 , srv737595659:84 , srv750476644:84 , srv761051826:84 , srv794450782:84 , srv852804053:84 , srv87687647:84 , srv935363611:84 , srv952725758:84 , srv971975363:84 , srv1037299347:85 , srv108586115:85 , srv1211151865:85 , srv1230369895:85 , srv1258476300:85 , srv1328598336:85 , srv1393219469:85 , srv1412888676:85 , srv1535667581:85 , srv1668368127:85 , srv1739877607:85 , srv1743375114:85 , srv1770984919:85 , srv1784940698:85 , srv1848638808:85 , srv1855919477:85 , srv1887524330:85 , srv1905999479:85 , srv196217858:85 , srv197786008:85 , srv2027343538:85 , srv2076974146:85 , srv2078403920:85 , srv263185745:85 , srv322600707:85 , srv326535906:85 , srv342612672:85 , srv349656606:85 , srv444353487:85 , srv476671664:85 , srv523178050:85 , srv682018230:85 , srv749597331:85 , srv758419311:85 , srv82226081:85 , srv904688275:85 , srv924252809:85 , srv938300776:85 , srv946322211:85 , srv963644934:85 , srv965610976:85 , srv976920049:85 , srv1639311722:86 , srv1890378508:86 , srv1949128609:86 , srv1972511279:86 , srv585864537:86 , srv616935826:86 , srv646323014:86 , srv785751649:86 , srv1595011882:636 }
2023-11-18 00:55:16,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-18 00:55:16,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2035644295=54, srv616935826=75, srv1108926886=3, srv241115300=62, srv2024633220=52, srv1535667581=21, srv196217858=45, srv2076974146=56, srv1905999479=41, srv87687647=88, srv2111368570=58, srv976920049=99, srv523178050=72, srv952725758=95, srv1739877607=29, srv737595659=79, srv749597331=80, srv1497375767=19, srv1962377826=46, srv588525610=74, srv682018230=78, srv1334452456=14, srv650722868=77, srv476671664=71, srv1887524330=39, srv1412888676=17, srv2063455666=55, srv1595011882=22, srv1980427842=50, srv1639311722=24, srv852804053=87, srv1633192690=23, srv904688275=89, srv965610976=97, srv1316659437=12, srv1414810013=18, srv1743375114=30, srv1211151865=7, srv646323014=76, srv785751649=84, srv1194454453=6, srv197786008=49, srv761051826=83, srv1972511279=47, srv750476644=81, srv1950363818=43, srv263185745=64, srv1972710915=48, srv1949128609=42, srv917057313=90, srv1823792578=35, srv241690310=63, srv1328598336=13, srv938300776=93, srv1366191115=15, srv1890378508=40, srv82226081=86, srv946322211=94, srv1706288343=27, srv758419311=82, srv2130655033=60, srv1736012805=28, srv322600707=65, srv1855919477=38, srv2115346705=59, srv1181686098=5, srv122662167=8, srv1783651661=33, srv1258476300=11, srv1770984919=32, srv1230369895=9, srv108586115=2, srv1668368127=26, srv326535906=66, srv349656606=68, srv963644934=96, srv342612672=67, srv2022295039=51, srv1037299347=0, srv1848638808=37, srv1958647553=44, srv924252809=91, srv115575050=4, srv40529009=69, srv585864537=73, srv1763732571=31, srv1244348113=10, srv2027343538=53, srv1393219469=16, srv971975363=98, srv794450782=85, srv444353487=70, srv935363611=92, srv104947518=1, srv236699962=61, srv2078403920=57, srv1831236819=36, srv1784940698=34, srv1499072680=20, srv1663468531=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-18 00:55:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-18 00:55:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-18 00:55:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-18 00:55:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-18 00:55:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-18 00:55:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-18 00:55:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-18 00:55:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-18 00:55:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-18 00:55:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-18 00:55:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-18 00:55:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-18 00:55:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-18 00:55:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-18 00:55:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-18 00:55:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-18 00:55:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-18 00:55:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-18 00:55:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-18 00:55:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-18 00:55:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-18 00:55:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-18 00:55:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-18 00:55:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-18 00:55:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-18 00:55:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-18 00:55:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-18 00:55:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-18 00:55:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-18 00:55:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-18 00:55:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-18 00:55:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-18 00:55:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-18 00:55:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-18 00:55:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-18 00:55:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-18 00:55:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-18 00:55:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-18 00:55:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-18 00:55:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-18 00:55:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-18 00:55:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-18 00:55:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-18 00:55:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-18 00:55:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-18 00:55:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-18 00:55:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-18 00:55:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-18 00:55:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-18 00:55:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-18 00:55:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-18 00:55:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-18 00:55:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-18 00:55:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-18 00:55:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-18 00:55:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-18 00:55:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-18 00:55:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-18 00:55:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-18 00:55:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-18 00:55:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-18 00:55:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-18 00:55:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-18 00:55:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-18 00:55:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-18 00:55:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-18 00:55:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-18 00:55:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-18 00:55:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-18 00:55:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-18 00:55:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-18 00:55:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-18 00:55:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-18 00:55:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-18 00:55:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-18 00:55:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-18 00:55:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-18 00:55:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-18 00:55:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-18 00:55:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-18 00:55:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-18 00:55:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-18 00:55:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-18 00:55:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-18 00:55:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-18 00:55:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-18 00:55:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-18 00:55:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-18 00:55:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-18 00:55:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-18 00:55:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-18 00:55:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-18 00:55:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-18 00:55:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-18 00:55:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-18 00:55:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-18 00:55:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-18 00:55:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-18 00:55:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-18 00:55:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-18 00:55:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-18 00:55:16,687 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.7494482491816342E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-18 00:55:16,688 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv104947518:83 , srv1108926886:83 , srv1194454453:83 , srv1497375767:83 , srv1831236819:83 , srv2115346705:83 , srv241115300:83 , srv40529009:83 , srv650722868:83 , srv917057313:83 , srv115575050:84 , srv1181686098:84 , srv122662167:84 , srv1244348113:84 , srv1316659437:84 , srv1334452456:84 , srv1366191115:84 , srv1414810013:84 , srv1499072680:84 , srv1633192690:84 , srv1663468531:84 , srv1706288343:84 , srv1736012805:84 , srv1763732571:84 , srv1783651661:84 , srv1823792578:84 , srv1950363818:84 , srv1958647553:84 , srv1962377826:84 , srv1972710915:84 , srv1980427842:84 , srv2022295039:84 , srv2024633220:84 , srv2035644295:84 , srv2063455666:84 , srv2111368570:84 , srv2130655033:84 , srv236699962:84 , srv241690310:84 , srv588525610:84 , srv737595659:84 , srv750476644:84 , srv761051826:84 , srv794450782:84 , srv852804053:84 , srv87687647:84 , srv935363611:84 , srv952725758:84 , srv971975363:84 , srv1037299347:85 , srv108586115:85 , srv1211151865:85 , srv1230369895:85 , srv1258476300:85 , srv1328598336:85 , srv1393219469:85 , srv1412888676:85 , srv1535667581:85 , srv1668368127:85 , srv1739877607:85 , srv1743375114:85 , srv1770984919:85 , srv1784940698:85 , srv1848638808:85 , srv1855919477:85 , srv1887524330:85 , srv1905999479:85 , srv196217858:85 , srv197786008:85 , srv2027343538:85 , srv2076974146:85 , srv2078403920:85 , srv263185745:85 , srv322600707:85 , srv326535906:85 , srv342612672:85 , srv349656606:85 , srv444353487:85 , srv476671664:85 , srv523178050:85 , srv682018230:85 , srv749597331:85 , srv758419311:85 , srv82226081:85 , srv904688275:85 , srv924252809:85 , srv938300776:85 , srv946322211:85 , srv963644934:85 , srv965610976:85 , srv976920049:85 , srv1639311722:86 , srv1890378508:86 , srv1949128609:86 , srv1972511279:86 , srv585864537:86 , srv616935826:86 , srv646323014:86 , srv785751649:86 , srv1595011882:636 }
2023-11-18 00:55:16,717 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=152 (was 130) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=9725 (was 10226)
2023-11-18 00:55:16,727 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=152, ProcessCount=167, AvailableMemoryMB=9725
2023-11-18 00:55:16,727 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-18 00:55:16,728 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-11-18 00:55:16,737 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1015204210:84 , srv1098869330:84 , srv1106239475:84 , srv1177827259:84 , srv1232335188:84 , srv1338498877:84 , srv1367052962:84 , srv1419991352:84 , srv1430734475:84 , srv1447833914:84 , srv1514340632:84 , srv1518912069:84 , srv1521268840:84 , srv1721589751:84 , srv1849655699:84 , srv1861121401:84 , srv2090600449:84 , srv228637585:84 , srv391716519:84 , srv411794862:84 , srv459647374:84 , srv463384675:84 , srv568281483:84 , srv589879982:84 , srv803258052:84 , srv804797866:84 , srv8488861:84 , srv970658960:84 , srv99641044:84 , srv1905814124:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-11-18 00:55:16,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-18 00:55:16,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv568281483=23, srv1098869330=1, srv970658960=28, srv1015204210=0, srv391716519=19, srv99641044=29, srv1367052962=6, srv1419991352=7, srv1521268840=12, srv589879982=24, srv803258052=25, srv1106239475=2, srv459647374=21, srv1447833914=9, srv1232335188=4, srv804797866=26, srv1338498877=5, srv1721589751=13, srv463384675=22, srv8488861=27, srv1430734475=8, srv1861121401=15, srv1849655699=14, srv2090600449=17, srv1905814124=16, srv1514340632=10, srv411794862=20, srv1177827259=3, srv1518912069=11, srv228637585=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-18 00:55:16,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-18 00:55:16,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-18 00:55:16,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-18 00:55:16,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-18 00:55:16,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-18 00:55:16,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-18 00:55:16,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-18 00:55:16,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-18 00:55:16,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-18 00:55:16,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-18 00:55:16,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-18 00:55:16,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-18 00:55:16,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-18 00:55:16,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-18 00:55:16,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-18 00:55:16,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-18 00:55:16,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-18 00:55:16,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-18 00:55:16,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-18 00:55:16,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-18 00:55:16,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-18 00:55:16,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-18 00:55:16,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-18 00:55:16,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-18 00:55:16,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-18 00:55:16,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-18 00:55:16,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-18 00:55:16,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-18 00:55:16,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-18 00:55:16,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-18 00:55:16,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-18 00:55:16,763 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1551140979953134, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-18 00:55:16,763 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912681802722121, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1551140979953134, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-11-18 00:57:16,751 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6128742 different iterations.  Found a solution that moves 1853 regions; Going from a computed imbalance of 0.9912681802722121 to a new imbalance of 4.324994439959734E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6862962962962963, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-18 00:57:16,758 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1015204210:90 , srv1098869330:90 , srv1106239475:90 , srv1177827259:90 , srv1232335188:90 , srv1338498877:90 , srv1367052962:90 , srv1419991352:90 , srv1430734475:90 , srv1447833914:90 , srv1514340632:90 , srv1518912069:90 , srv1521268840:90 , srv1721589751:90 , srv1849655699:90 , srv1861121401:90 , srv1905814124:90 , srv2090600449:90 , srv228637585:90 , srv391716519:90 , srv411794862:90 , srv459647374:90 , srv463384675:90 , srv568281483:90 , srv589879982:90 , srv803258052:90 , srv804797866:90 , srv8488861:90 , srv970658960:90 , srv99641044:90 }
2023-11-18 00:57:16,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-18 00:57:16,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv568281483=23, srv1098869330=1, srv970658960=28, srv1015204210=0, srv391716519=19, srv99641044=29, srv1367052962=6, srv1419991352=7, srv1521268840=12, srv589879982=24, srv803258052=25, srv1106239475=2, srv459647374=21, srv1447833914=9, srv1232335188=4, srv804797866=26, srv1338498877=5, srv1721589751=13, srv463384675=22, srv8488861=27, srv1430734475=8, srv1861121401=15, srv1849655699=14, srv2090600449=17, srv1905814124=16, srv1514340632=10, srv411794862=20, srv1177827259=3, srv1518912069=11, srv228637585=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-18 00:57:16,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-18 00:57:16,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-18 00:57:16,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-18 00:57:16,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-18 00:57:16,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-18 00:57:16,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-18 00:57:16,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-18 00:57:16,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-18 00:57:16,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-18 00:57:16,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-18 00:57:16,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-18 00:57:16,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-18 00:57:16,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-18 00:57:16,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-18 00:57:16,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-18 00:57:16,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-18 00:57:16,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-18 00:57:16,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-18 00:57:16,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-18 00:57:16,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-18 00:57:16,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-18 00:57:16,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-18 00:57:16,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-18 00:57:16,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-18 00:57:16,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-18 00:57:16,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-18 00:57:16,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-18 00:57:16,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-18 00:57:16,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-18 00:57:16,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-18 00:57:16,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-18 00:57:16,780 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-18 00:57:16,781 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1015204210:90 , srv1098869330:90 , srv1106239475:90 , srv1177827259:90 , srv1232335188:90 , srv1338498877:90 , srv1367052962:90 , srv1419991352:90 , srv1430734475:90 , srv1447833914:90 , srv1514340632:90 , srv1518912069:90 , srv1521268840:90 , srv1721589751:90 , srv1849655699:90 , srv1861121401:90 , srv1905814124:90 , srv2090600449:90 , srv228637585:90 , srv391716519:90 , srv411794862:90 , srv459647374:90 , srv463384675:90 , srv568281483:90 , srv589879982:90 , srv803258052:90 , srv804797866:90 , srv8488861:90 , srv970658960:90 , srv99641044:90 }
2023-11-18 00:57:16,793 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=204 (was 152) - SystemLoadAverage LEAK? -, ProcessCount=173 (was 167) - ProcessCount LEAK? -, AvailableMemoryMB=9668 (was 9725)