2023-11-29 02:53:45,892 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-11-29 02:53:46,222 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-11-29 02:53:46,342 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-11-29 02:53:46,414 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-29 02:53:46,422 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-29 02:53:46,435 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=310, ProcessCount=171, AvailableMemoryMB=10902
2023-11-29 02:53:46,437 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-29 02:53:46,437 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-29 02:53:46,576 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1038186288:84 , srv1056804226:84 , srv1083786200:84 , srv1084473663:84 , srv1173590814:84 , srv1181780157:84 , srv1211298562:84 , srv1252357142:84 , srv12607558:84 , srv1291358295:84 , srv1301135483:84 , srv1316612339:84 , srv1357387009:84 , srv1360192701:84 , srv1362232575:84 , srv1381400068:84 , srv1383601966:84 , srv1410641729:84 , srv1417673539:84 , srv1453321812:84 , srv1472834402:84 , srv1504133808:84 , srv1512310625:84 , srv1528894511:84 , srv1530272425:84 , srv154504776:84 , srv1557583544:84 , srv1570375596:84 , srv1605583587:84 , srv1609689979:84 , srv1648324776:84 , srv1665950797:84 , srv166660718:84 , srv1696749078:84 , srv1709085905:84 , srv1718197167:84 , srv1746812611:84 , srv1772557584:84 , srv1813019142:84 , srv1870159289:84 , srv1912368617:84 , srv1912691226:84 , srv1933740486:84 , srv1944845973:84 , srv1949745280:84 , srv2000441483:84 , srv2020972211:84 , srv2071624044:84 , srv2078891279:84 , srv208905869:84 , srv2123101866:84 , srv2123159871:84 , srv2130038467:84 , srv2139687630:84 , srv222048042:84 , srv247256714:84 , srv251550218:84 , srv2699086:84 , srv316514595:84 , srv372177068:84 , srv376466799:84 , srv377219443:84 , srv378491318:84 , srv394793467:84 , srv405526359:84 , srv427239674:84 , srv445983922:84 , srv45306137:84 , srv454312366:84 , srv492014377:84 , srv498909885:84 , srv526791786:84 , srv526967456:84 , srv527378034:84 , srv527712537:84 , srv539641945:84 , srv546667648:84 , srv575348855:84 , srv622695514:84 , srv648196981:84 , srv657486363:84 , srv671300967:84 , srv705186133:84 , srv734985817:84 , srv757031419:84 , srv804990028:84 , srv80959634:84 , srv850660886:84 , srv858105288:84 , srv874201700:84 , srv875481994:84 , srv896841697:84 , srv920171789:84 , srv938184466:84 , srv947869794:84 , srv948387237:84 , srv952022054:84 , srv995497987:84 , srv998548958:84 , srv250824897:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-11-29 02:53:46,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-29 02:53:46,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1252357142=7, srv1772557584=37, srv1813019142=38, srv948387237=96, srv454312366=69, srv498909885=71, srv1381400068=15, srv372177068=60, srv427239674=66, srv938184466=94, srv657486363=81, srv1665950797=31, srv208905869=49, srv1038186288=0, srv2123159871=51, srv250824897=56, srv154504776=25, srv527378034=74, srv648196981=80, srv995497987=98, srv622695514=79, srv998548958=99, srv1933740486=42, srv1696749078=33, srv734985817=84, srv875481994=91, srv920171789=93, srv1211298562=6, srv405526359=65, srv850660886=88, srv1912691226=41, srv376466799=61, srv1083786200=2, srv757031419=85, srv1557583544=26, srv952022054=97, srv858105288=89, srv575348855=78, srv1410641729=17, srv1360192701=13, srv1357387009=12, srv1291358295=9, srv1453321812=19, srv1173590814=4, srv1301135483=10, srv2071624044=47, srv804990028=86, srv222048042=54, srv316514595=59, srv705186133=83, srv539641945=76, srv874201700=90, srv377219443=62, srv947869794=95, srv1084473663=3, srv445983922=67, srv1512310625=22, srv1912368617=40, srv247256714=55, srv526791786=72, srv527712537=75, srv1383601966=16, srv1417673539=18, srv492014377=70, srv2020972211=46, srv394793467=64, srv1949745280=44, srv1056804226=1, srv526967456=73, srv1504133808=21, srv546667648=77, srv1181780157=5, srv2123101866=50, srv1648324776=30, srv1528894511=23, srv671300967=82, srv1605583587=28, srv1316612339=11, srv2078891279=48, srv1530272425=24, srv896841697=92, srv166660718=32, srv1472834402=20, srv1609689979=29, srv1570375596=27, srv45306137=68, srv2139687630=53, srv378491318=63, srv1709085905=34, srv1870159289=39, srv12607558=8, srv1944845973=43, srv2699086=58, srv2000441483=45, srv1718197167=35, srv1362232575=14, srv1746812611=36, srv2130038467=52, srv80959634=87, srv251550218=57} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-29 02:53:46,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-29 02:53:46,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-29 02:53:46,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-29 02:53:46,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-29 02:53:46,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-29 02:53:46,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-29 02:53:46,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-29 02:53:46,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-29 02:53:46,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-29 02:53:46,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-29 02:53:46,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-29 02:53:46,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-29 02:53:46,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-29 02:53:46,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-29 02:53:46,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-29 02:53:46,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-29 02:53:46,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-29 02:53:46,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-29 02:53:46,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-29 02:53:46,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-29 02:53:46,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-29 02:53:46,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-29 02:53:46,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-29 02:53:46,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-29 02:53:46,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-29 02:53:46,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-29 02:53:46,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-29 02:53:46,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-29 02:53:46,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-29 02:53:46,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-29 02:53:46,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-29 02:53:46,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-29 02:53:46,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-29 02:53:46,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-29 02:53:46,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-29 02:53:46,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-29 02:53:46,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-29 02:53:46,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-29 02:53:46,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-29 02:53:46,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-29 02:53:46,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-29 02:53:46,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-29 02:53:46,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-29 02:53:46,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-29 02:53:46,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-29 02:53:46,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-29 02:53:46,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-29 02:53:46,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-29 02:53:46,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-29 02:53:46,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-29 02:53:46,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-29 02:53:46,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-29 02:53:46,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-29 02:53:46,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-29 02:53:46,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-29 02:53:46,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-29 02:53:46,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-29 02:53:46,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-29 02:53:46,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-29 02:53:46,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-29 02:53:46,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-29 02:53:46,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-29 02:53:46,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-29 02:53:46,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-29 02:53:46,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-29 02:53:46,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-29 02:53:46,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-29 02:53:46,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-29 02:53:46,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-29 02:53:46,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-29 02:53:46,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-29 02:53:46,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-29 02:53:46,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-29 02:53:46,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-29 02:53:46,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-29 02:53:46,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-29 02:53:46,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-29 02:53:46,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-29 02:53:46,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-29 02:53:46,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-29 02:53:46,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-29 02:53:46,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-29 02:53:46,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-29 02:53:46,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-29 02:53:46,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-29 02:53:46,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-29 02:53:46,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-29 02:53:46,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-29 02:53:46,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-29 02:53:46,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-29 02:53:46,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-29 02:53:46,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-29 02:53:46,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-29 02:53:46,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-29 02:53:46,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-29 02:53:46,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-29 02:53:46,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-29 02:53:46,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-29 02:53:46,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-29 02:53:46,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-29 02:53:46,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-29 02:53:46,906 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 02:53:46,906 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-29 02:53:46,906 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-29 02:53:47,331 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 622 ms to try 5000 different iterations.  Found a solution that moves 4641 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18436830656501818. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041385470900378675, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.051654989448305974, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5156666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041385470900378675, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18016666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24108333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 02:53:47,361 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv526967456:82 , srv757031419:82 , srv1291358295:83 , srv1357387009:83 , srv2123101866:83 , srv222048042:83 , srv427239674:83 , srv527378034:83 , srv80959634:83 , srv938184466:83 , srv1211298562:84 , srv1410641729:84 , srv1453321812:84 , srv1570375596:84 , srv1696749078:84 , srv1709085905:84 , srv1718197167:84 , srv1870159289:84 , srv1912691226:84 , srv2000441483:84 , srv2123159871:84 , srv251550218:84 , srv394793467:84 , srv498909885:84 , srv657486363:84 , srv896841697:84 , srv947869794:84 , srv1038186288:85 , srv1316612339:85 , srv1383601966:85 , srv1512310625:85 , srv166660718:85 , srv247256714:85 , srv316514595:85 , srv526791786:85 , srv671300967:85 , srv874201700:85 , srv1084473663:86 , srv1417673539:86 , srv1504133808:86 , srv1609689979:86 , srv1772557584:86 , srv1813019142:86 , srv2139687630:86 , srv2699086:86 , srv376466799:86 , srv377219443:86 , srv454312366:86 , srv492014377:86 , srv804990028:86 , srv858105288:86 , srv1173590814:87 , srv1252357142:87 , srv1381400068:87 , srv1472834402:87 , srv1528894511:87 , srv154504776:87 , srv1557583544:87 , srv1665950797:87 , srv1912368617:87 , srv1933740486:87 , srv1944845973:87 , srv2020972211:87 , srv2078891279:87 , srv208905869:87 , srv2130038467:87 , srv372177068:87 , srv378491318:87 , srv45306137:87 , srv575348855:87 , srv622695514:87 , srv648196981:87 , srv705186133:87 , srv850660886:87 , srv948387237:87 , srv952022054:87 , srv1083786200:88 , srv1181780157:88 , srv1360192701:88 , srv1362232575:88 , srv1530272425:88 , srv527712537:88 , srv546667648:88 , srv875481994:88 , srv998548958:88 , srv12607558:89 , srv1301135483:89 , srv1605583587:89 , srv1648324776:89 , srv1949745280:89 , srv2071624044:89 , srv445983922:89 , srv539641945:89 , srv734985817:89 , srv1056804226:92 , srv1746812611:92 , srv920171789:92 , srv405526359:93 , srv995497987:95 , srv250824897:458 }
2023-11-29 02:53:47,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-29 02:53:47,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1252357142=7, srv1772557584=37, srv1813019142=38, srv948387237=96, srv454312366=69, srv498909885=71, srv1381400068=15, srv372177068=60, srv427239674=66, srv938184466=94, srv657486363=81, srv1665950797=31, srv208905869=49, srv1038186288=0, srv2123159871=51, srv250824897=56, srv154504776=25, srv527378034=74, srv648196981=80, srv995497987=98, srv622695514=79, srv998548958=99, srv1933740486=42, srv1696749078=33, srv734985817=84, srv875481994=91, srv920171789=93, srv1211298562=6, srv405526359=65, srv850660886=88, srv1912691226=41, srv376466799=61, srv1083786200=2, srv757031419=85, srv1557583544=26, srv952022054=97, srv858105288=89, srv575348855=78, srv1410641729=17, srv1360192701=13, srv1357387009=12, srv1291358295=9, srv1453321812=19, srv1173590814=4, srv1301135483=10, srv2071624044=47, srv804990028=86, srv222048042=54, srv316514595=59, srv705186133=83, srv539641945=76, srv874201700=90, srv377219443=62, srv947869794=95, srv1084473663=3, srv445983922=67, srv1512310625=22, srv1912368617=40, srv247256714=55, srv526791786=72, srv527712537=75, srv1383601966=16, srv1417673539=18, srv492014377=70, srv2020972211=46, srv394793467=64, srv1949745280=44, srv1056804226=1, srv526967456=73, srv1504133808=21, srv546667648=77, srv1181780157=5, srv2123101866=50, srv1648324776=30, srv1528894511=23, srv671300967=82, srv1605583587=28, srv1316612339=11, srv2078891279=48, srv1530272425=24, srv896841697=92, srv166660718=32, srv1472834402=20, srv1609689979=29, srv1570375596=27, srv45306137=68, srv2139687630=53, srv378491318=63, srv1709085905=34, srv1870159289=39, srv12607558=8, srv1944845973=43, srv2699086=58, srv2000441483=45, srv1718197167=35, srv1362232575=14, srv1746812611=36, srv2130038467=52, srv80959634=87, srv251550218=57} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-29 02:53:47,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-29 02:53:47,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-29 02:53:47,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-29 02:53:47,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-29 02:53:47,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-29 02:53:47,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-29 02:53:47,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-29 02:53:47,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-29 02:53:47,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-29 02:53:47,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-29 02:53:47,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-29 02:53:47,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-29 02:53:47,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-29 02:53:47,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-29 02:53:47,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-29 02:53:47,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-29 02:53:47,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-29 02:53:47,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-29 02:53:47,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-29 02:53:47,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-29 02:53:47,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-29 02:53:47,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-29 02:53:47,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-29 02:53:47,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-29 02:53:47,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-29 02:53:47,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-29 02:53:47,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-29 02:53:47,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-29 02:53:47,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-29 02:53:47,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-29 02:53:47,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-29 02:53:47,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-29 02:53:47,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-29 02:53:47,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-29 02:53:47,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-29 02:53:47,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-29 02:53:47,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-29 02:53:47,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-29 02:53:47,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-29 02:53:47,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-29 02:53:47,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-29 02:53:47,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-29 02:53:47,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-29 02:53:47,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-29 02:53:47,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-29 02:53:47,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-29 02:53:47,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-29 02:53:47,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-29 02:53:47,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-29 02:53:47,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-29 02:53:47,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-29 02:53:47,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-29 02:53:47,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-29 02:53:47,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-29 02:53:47,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-29 02:53:47,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-29 02:53:47,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-29 02:53:47,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-29 02:53:47,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-29 02:53:47,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-29 02:53:47,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-29 02:53:47,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-29 02:53:47,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-29 02:53:47,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-29 02:53:47,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-29 02:53:47,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-29 02:53:47,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-29 02:53:47,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-29 02:53:47,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-29 02:53:47,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-29 02:53:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-29 02:53:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-29 02:53:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-29 02:53:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-29 02:53:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-29 02:53:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-29 02:53:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-29 02:53:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-29 02:53:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-29 02:53:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-29 02:53:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-29 02:53:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-29 02:53:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-29 02:53:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-29 02:53:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-29 02:53:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-29 02:53:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-29 02:53:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-29 02:53:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-29 02:53:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-29 02:53:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-29 02:53:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-29 02:53:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-29 02:53:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-29 02:53:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-29 02:53:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-29 02:53:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-29 02:53:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-29 02:53:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-29 02:53:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-29 02:53:47,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-29 02:53:47,510 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041385470900378675, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.051654989448305974, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041385470900378675, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18016666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24108333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 02:53:47,511 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-29 02:53:47,511 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18433580958844636, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041385470900378675, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.051654989448305974, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041385470900378675, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18016666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24108333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-29 02:53:47,791 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 383 ms to try 5000 different iterations.  Found a solution that moves 2627 regions; Going from a computed imbalance of 0.18433580958844636 to a new imbalance of 0.013204234714064083. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024061379714488705); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.033541437929244945, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29188888888888886, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024061379714488705); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04516666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 02:53:47,803 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1746812611:78 , srv405526359:80 , srv995497987:80 , srv1056804226:81 , srv920171789:81 , srv378491318:82 , srv527712537:82 , srv539641945:82 , srv875481994:82 , srv1181780157:83 , srv1472834402:83 , srv1530272425:83 , srv154504776:83 , srv1557583544:83 , srv1648324776:83 , srv952022054:83 , srv1083786200:84 , srv12607558:84 , srv1301135483:84 , srv1360192701:84 , srv1362232575:84 , srv1417673539:84 , srv1504133808:84 , srv1605583587:84 , srv1665950797:84 , srv1912368617:84 , srv1933740486:84 , srv2071624044:84 , srv2078891279:84 , srv208905869:84 , srv445983922:84 , srv492014377:84 , srv546667648:84 , srv575348855:84 , srv705186133:84 , srv998548958:84 , srv1252357142:85 , srv1381400068:85 , srv1512310625:85 , srv1528894511:85 , srv1944845973:85 , srv1949745280:85 , srv2130038467:85 , srv2699086:85 , srv372177068:85 , srv622695514:85 , srv648196981:85 , srv850660886:85 , srv948387237:85 , srv1084473663:86 , srv1609689979:86 , srv1696749078:86 , srv1813019142:86 , srv2139687630:86 , srv377219443:86 , srv394793467:86 , srv427239674:86 , srv454312366:86 , srv734985817:86 , srv858105288:86 , srv874201700:86 , srv896841697:86 , srv1173590814:87 , srv1211298562:87 , srv1316612339:87 , srv1383601966:87 , srv1410641729:87 , srv1570375596:87 , srv1709085905:87 , srv1772557584:87 , srv222048042:87 , srv376466799:87 , srv526791786:87 , srv527378034:87 , srv671300967:87 , srv804990028:87 , srv1038186288:88 , srv166660718:88 , srv1718197167:88 , srv1912691226:88 , srv2000441483:88 , srv2020972211:88 , srv2123101866:88 , srv247256714:88 , srv251550218:88 , srv316514595:88 , srv45306137:88 , srv657486363:88 , srv1453321812:89 , srv1870159289:89 , srv2123159871:89 , srv498909885:89 , srv757031419:89 , srv80959634:89 , srv1291358295:90 , srv1357387009:90 , srv526967456:90 , srv938184466:90 , srv947869794:90 , srv250824897:530 }
2023-11-29 02:53:47,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-29 02:53:47,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1252357142=7, srv1772557584=37, srv1813019142=38, srv948387237=96, srv454312366=69, srv498909885=71, srv1381400068=15, srv372177068=60, srv427239674=66, srv938184466=94, srv657486363=81, srv1665950797=31, srv208905869=49, srv1038186288=0, srv2123159871=51, srv250824897=56, srv154504776=25, srv527378034=74, srv648196981=80, srv995497987=98, srv622695514=79, srv998548958=99, srv1933740486=42, srv1696749078=33, srv734985817=84, srv875481994=91, srv920171789=93, srv1211298562=6, srv405526359=65, srv850660886=88, srv1912691226=41, srv376466799=61, srv1083786200=2, srv757031419=85, srv1557583544=26, srv952022054=97, srv858105288=89, srv575348855=78, srv1410641729=17, srv1360192701=13, srv1357387009=12, srv1291358295=9, srv1453321812=19, srv1173590814=4, srv1301135483=10, srv2071624044=47, srv804990028=86, srv222048042=54, srv316514595=59, srv705186133=83, srv539641945=76, srv874201700=90, srv377219443=62, srv947869794=95, srv1084473663=3, srv445983922=67, srv1512310625=22, srv1912368617=40, srv247256714=55, srv526791786=72, srv527712537=75, srv1383601966=16, srv1417673539=18, srv492014377=70, srv2020972211=46, srv394793467=64, srv1949745280=44, srv1056804226=1, srv526967456=73, srv1504133808=21, srv546667648=77, srv1181780157=5, srv2123101866=50, srv1648324776=30, srv1528894511=23, srv671300967=82, srv1605583587=28, srv1316612339=11, srv2078891279=48, srv1530272425=24, srv896841697=92, srv166660718=32, srv1472834402=20, srv1609689979=29, srv1570375596=27, srv45306137=68, srv2139687630=53, srv378491318=63, srv1709085905=34, srv1870159289=39, srv12607558=8, srv1944845973=43, srv2699086=58, srv2000441483=45, srv1718197167=35, srv1362232575=14, srv1746812611=36, srv2130038467=52, srv80959634=87, srv251550218=57} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-29 02:53:47,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-29 02:53:47,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-29 02:53:47,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-29 02:53:47,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-29 02:53:47,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-29 02:53:47,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-29 02:53:47,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-29 02:53:47,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-29 02:53:47,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-29 02:53:47,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-29 02:53:47,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-29 02:53:47,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-29 02:53:47,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-29 02:53:47,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-29 02:53:47,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-29 02:53:47,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-29 02:53:47,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-29 02:53:47,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-29 02:53:47,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-29 02:53:47,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-29 02:53:47,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-29 02:53:47,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-29 02:53:47,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-29 02:53:47,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-29 02:53:47,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-29 02:53:47,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-29 02:53:47,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-29 02:53:47,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-29 02:53:47,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-29 02:53:47,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-29 02:53:47,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-29 02:53:47,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-29 02:53:47,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-29 02:53:47,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-29 02:53:47,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-29 02:53:47,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-29 02:53:47,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-29 02:53:47,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-29 02:53:47,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-29 02:53:47,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-29 02:53:47,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-29 02:53:47,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-29 02:53:47,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-29 02:53:47,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-29 02:53:47,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-29 02:53:47,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-29 02:53:47,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-29 02:53:47,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-29 02:53:47,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-29 02:53:47,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-29 02:53:47,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-29 02:53:47,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-29 02:53:47,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-29 02:53:47,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-29 02:53:47,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-29 02:53:47,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-29 02:53:47,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-29 02:53:47,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-29 02:53:47,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-29 02:53:47,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-29 02:53:47,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-29 02:53:47,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-29 02:53:47,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-29 02:53:47,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-29 02:53:47,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-29 02:53:47,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-29 02:53:47,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-29 02:53:47,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-29 02:53:47,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-29 02:53:47,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-29 02:53:47,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-29 02:53:47,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-29 02:53:47,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-29 02:53:47,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-29 02:53:47,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-29 02:53:47,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-29 02:53:47,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-29 02:53:47,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-29 02:53:47,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-29 02:53:47,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-29 02:53:47,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-29 02:53:47,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-29 02:53:47,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-29 02:53:47,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-29 02:53:47,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-29 02:53:47,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-29 02:53:47,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-29 02:53:47,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-29 02:53:47,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-29 02:53:47,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-29 02:53:47,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-29 02:53:47,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-29 02:53:47,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-29 02:53:47,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-29 02:53:47,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-29 02:53:47,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-29 02:53:47,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-29 02:53:47,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-29 02:53:47,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-29 02:53:47,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-29 02:53:47,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-29 02:53:47,974 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024061379714488705); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.033541437929244945, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024061379714488705); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04516666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 02:53:47,974 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-29 02:53:47,974 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013185840066907406, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024061379714488705); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.033541437929244945, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024061379714488705); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04516666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-29 02:53:48,360 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 501 ms to try 5000 different iterations.  Found a solution that moves 947 regions; Going from a computed imbalance of 0.013185840066907406 to a new imbalance of 8.645714968398492E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007231928209967171); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007856742013183862); MoveCostFunction : (multiplier=7.0, imbalance=0.10522222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007231928209967171); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 02:53:48,362 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2020972211:83 , srv45306137:83 , srv734985817:83 , srv995497987:83 , srv1038186288:84 , srv1056804226:84 , srv1173590814:84 , srv12607558:84 , srv1291358295:84 , srv1357387009:84 , srv1605583587:84 , srv166660718:84 , srv1772557584:84 , srv1870159289:84 , srv1949745280:84 , srv2123159871:84 , srv247256714:84 , srv316514595:84 , srv377219443:84 , srv405526359:84 , srv445983922:84 , srv454312366:84 , srv526967456:84 , srv657486363:84 , srv920171789:84 , srv938184466:84 , srv947869794:84 , srv1084473663:85 , srv1181780157:85 , srv1252357142:85 , srv1301135483:85 , srv1316612339:85 , srv1360192701:85 , srv1453321812:85 , srv1504133808:85 , srv1528894511:85 , srv1609689979:85 , srv1648324776:85 , srv1813019142:85 , srv1912691226:85 , srv2000441483:85 , srv2071624044:85 , srv208905869:85 , srv2139687630:85 , srv251550218:85 , srv372177068:85 , srv376466799:85 , srv498909885:85 , srv648196981:85 , srv757031419:85 , srv804990028:85 , srv858105288:85 , srv948387237:85 , srv1083786200:86 , srv1362232575:86 , srv1381400068:86 , srv1383601966:86 , srv1410641729:86 , srv1417673539:86 , srv1472834402:86 , srv1665950797:86 , srv1718197167:86 , srv1933740486:86 , srv1944845973:86 , srv2078891279:86 , srv2130038467:86 , srv526791786:86 , srv539641945:86 , srv546667648:86 , srv705186133:86 , srv80959634:86 , srv850660886:86 , srv874201700:86 , srv998548958:86 , srv1211298562:87 , srv1512310625:87 , srv1530272425:87 , srv154504776:87 , srv1557583544:87 , srv1570375596:87 , srv1709085905:87 , srv1746812611:87 , srv1912368617:87 , srv2123101866:87 , srv222048042:87 , srv2699086:87 , srv394793467:87 , srv527712537:87 , srv671300967:87 , srv952022054:87 , srv1696749078:88 , srv492014377:88 , srv527378034:88 , srv575348855:88 , srv622695514:88 , srv875481994:88 , srv896841697:88 , srv378491318:89 , srv427239674:89 , srv250824897:534 }
2023-11-29 02:53:48,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-29 02:53:48,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1252357142=7, srv1772557584=37, srv1813019142=38, srv948387237=96, srv454312366=69, srv498909885=71, srv1381400068=15, srv372177068=60, srv427239674=66, srv938184466=94, srv657486363=81, srv1665950797=31, srv208905869=49, srv1038186288=0, srv2123159871=51, srv250824897=56, srv154504776=25, srv527378034=74, srv648196981=80, srv995497987=98, srv622695514=79, srv998548958=99, srv1933740486=42, srv1696749078=33, srv734985817=84, srv875481994=91, srv920171789=93, srv1211298562=6, srv405526359=65, srv850660886=88, srv1912691226=41, srv376466799=61, srv1083786200=2, srv757031419=85, srv1557583544=26, srv952022054=97, srv858105288=89, srv575348855=78, srv1410641729=17, srv1360192701=13, srv1357387009=12, srv1291358295=9, srv1453321812=19, srv1173590814=4, srv1301135483=10, srv2071624044=47, srv804990028=86, srv222048042=54, srv316514595=59, srv705186133=83, srv539641945=76, srv874201700=90, srv377219443=62, srv947869794=95, srv1084473663=3, srv445983922=67, srv1512310625=22, srv1912368617=40, srv247256714=55, srv526791786=72, srv527712537=75, srv1383601966=16, srv1417673539=18, srv492014377=70, srv2020972211=46, srv394793467=64, srv1949745280=44, srv1056804226=1, srv526967456=73, srv1504133808=21, srv546667648=77, srv1181780157=5, srv2123101866=50, srv1648324776=30, srv1528894511=23, srv671300967=82, srv1605583587=28, srv1316612339=11, srv2078891279=48, srv1530272425=24, srv896841697=92, srv166660718=32, srv1472834402=20, srv1609689979=29, srv1570375596=27, srv45306137=68, srv2139687630=53, srv378491318=63, srv1709085905=34, srv1870159289=39, srv12607558=8, srv1944845973=43, srv2699086=58, srv2000441483=45, srv1718197167=35, srv1362232575=14, srv1746812611=36, srv2130038467=52, srv80959634=87, srv251550218=57} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-29 02:53:48,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-29 02:53:48,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-29 02:53:48,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-29 02:53:48,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-29 02:53:48,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-29 02:53:48,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-29 02:53:48,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-29 02:53:48,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-29 02:53:48,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-29 02:53:48,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-29 02:53:48,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-29 02:53:48,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-29 02:53:48,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-29 02:53:48,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-29 02:53:48,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-29 02:53:48,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-29 02:53:48,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-29 02:53:48,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-29 02:53:48,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-29 02:53:48,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-29 02:53:48,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-29 02:53:48,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-29 02:53:48,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-29 02:53:48,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-29 02:53:48,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-29 02:53:48,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-29 02:53:48,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-29 02:53:48,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-29 02:53:48,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-29 02:53:48,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-29 02:53:48,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-29 02:53:48,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-29 02:53:48,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-29 02:53:48,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-29 02:53:48,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-29 02:53:48,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-29 02:53:48,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-29 02:53:48,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-29 02:53:48,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-29 02:53:48,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-29 02:53:48,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-29 02:53:48,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-29 02:53:48,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-29 02:53:48,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-29 02:53:48,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-29 02:53:48,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-29 02:53:48,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-29 02:53:48,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-29 02:53:48,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-29 02:53:48,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-29 02:53:48,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-29 02:53:48,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-29 02:53:48,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-29 02:53:48,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-29 02:53:48,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-29 02:53:48,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-29 02:53:48,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-29 02:53:48,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-29 02:53:48,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-29 02:53:48,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-29 02:53:48,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-29 02:53:48,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-29 02:53:48,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-29 02:53:48,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-29 02:53:48,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-29 02:53:48,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-29 02:53:48,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-29 02:53:48,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-29 02:53:48,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-29 02:53:48,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-29 02:53:48,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-29 02:53:48,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-29 02:53:48,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-29 02:53:48,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-29 02:53:48,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-29 02:53:48,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-29 02:53:48,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-29 02:53:48,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-29 02:53:48,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-29 02:53:48,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-29 02:53:48,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-29 02:53:48,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-29 02:53:48,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-29 02:53:48,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-29 02:53:48,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-29 02:53:48,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-29 02:53:48,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-29 02:53:48,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-29 02:53:48,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-29 02:53:48,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-29 02:53:48,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-29 02:53:48,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-29 02:53:48,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-29 02:53:48,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-29 02:53:48,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-29 02:53:48,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-29 02:53:48,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-29 02:53:48,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-29 02:53:48,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-29 02:53:48,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-29 02:53:48,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-29 02:53:48,507 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007231928209967171); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007856742013183862); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007231928209967171); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 02:53:48,507 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-29 02:53:48,508 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.579404611118806E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007231928209967171); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007856742013183862); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007231928209967171); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-29 02:53:48,847 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 445 ms to try 5000 different iterations.  Found a solution that moves 277 regions; Going from a computed imbalance of 8.579404611118806E-4 to a new imbalance of 2.1081178377286114E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.909066927536782E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.03077777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.909066927536782E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 02:53:48,848 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv622695514:82 , srv2130038467:83 , srv372177068:83 , srv376466799:83 , srv45306137:83 , srv575348855:83 , srv671300967:83 , srv850660886:83 , srv1083786200:84 , srv1173590814:84 , srv1211298562:84 , srv1291358295:84 , srv1301135483:84 , srv1357387009:84 , srv1381400068:84 , srv1383601966:84 , srv1453321812:84 , srv1530272425:84 , srv1570375596:84 , srv1605583587:84 , srv1696749078:84 , srv1709085905:84 , srv1912368617:84 , srv1949745280:84 , srv2020972211:84 , srv2071624044:84 , srv2123101866:84 , srv2699086:84 , srv378491318:84 , srv394793467:84 , srv492014377:84 , srv526791786:84 , srv527378034:84 , srv546667648:84 , srv734985817:84 , srv804990028:84 , srv80959634:84 , srv874201700:84 , srv875481994:84 , srv896841697:84 , srv995497987:84 , srv1038186288:85 , srv1056804226:85 , srv1084473663:85 , srv1181780157:85 , srv1252357142:85 , srv12607558:85 , srv1316612339:85 , srv1360192701:85 , srv1362232575:85 , srv1410641729:85 , srv1528894511:85 , srv154504776:85 , srv1557583544:85 , srv1609689979:85 , srv1648324776:85 , srv1665950797:85 , srv166660718:85 , srv1718197167:85 , srv1746812611:85 , srv1772557584:85 , srv1813019142:85 , srv1870159289:85 , srv1912691226:85 , srv1944845973:85 , srv2000441483:85 , srv2078891279:85 , srv2123159871:85 , srv2139687630:85 , srv222048042:85 , srv251550218:85 , srv316514595:85 , srv405526359:85 , srv427239674:85 , srv445983922:85 , srv498909885:85 , srv527712537:85 , srv539641945:85 , srv648196981:85 , srv705186133:85 , srv858105288:85 , srv920171789:85 , srv947869794:85 , srv948387237:85 , srv952022054:85 , srv998548958:85 , srv1417673539:86 , srv1472834402:86 , srv1512310625:86 , srv1933740486:86 , srv208905869:86 , srv247256714:86 , srv377219443:86 , srv454312366:86 , srv526967456:86 , srv657486363:86 , srv757031419:86 , srv1504133808:87 , srv938184466:87 , srv250824897:620 }
2023-11-29 02:53:48,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-29 02:53:48,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1252357142=7, srv1772557584=37, srv1813019142=38, srv948387237=96, srv454312366=69, srv498909885=71, srv1381400068=15, srv372177068=60, srv427239674=66, srv938184466=94, srv657486363=81, srv1665950797=31, srv208905869=49, srv1038186288=0, srv2123159871=51, srv250824897=56, srv154504776=25, srv527378034=74, srv648196981=80, srv995497987=98, srv622695514=79, srv998548958=99, srv1933740486=42, srv1696749078=33, srv734985817=84, srv875481994=91, srv920171789=93, srv1211298562=6, srv405526359=65, srv850660886=88, srv1912691226=41, srv376466799=61, srv1083786200=2, srv757031419=85, srv1557583544=26, srv952022054=97, srv858105288=89, srv575348855=78, srv1410641729=17, srv1360192701=13, srv1357387009=12, srv1291358295=9, srv1453321812=19, srv1173590814=4, srv1301135483=10, srv2071624044=47, srv804990028=86, srv222048042=54, srv316514595=59, srv705186133=83, srv539641945=76, srv874201700=90, srv377219443=62, srv947869794=95, srv1084473663=3, srv445983922=67, srv1512310625=22, srv1912368617=40, srv247256714=55, srv526791786=72, srv527712537=75, srv1383601966=16, srv1417673539=18, srv492014377=70, srv2020972211=46, srv394793467=64, srv1949745280=44, srv1056804226=1, srv526967456=73, srv1504133808=21, srv546667648=77, srv1181780157=5, srv2123101866=50, srv1648324776=30, srv1528894511=23, srv671300967=82, srv1605583587=28, srv1316612339=11, srv2078891279=48, srv1530272425=24, srv896841697=92, srv166660718=32, srv1472834402=20, srv1609689979=29, srv1570375596=27, srv45306137=68, srv2139687630=53, srv378491318=63, srv1709085905=34, srv1870159289=39, srv12607558=8, srv1944845973=43, srv2699086=58, srv2000441483=45, srv1718197167=35, srv1362232575=14, srv1746812611=36, srv2130038467=52, srv80959634=87, srv251550218=57} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-29 02:53:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-29 02:53:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-29 02:53:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-29 02:53:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-29 02:53:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-29 02:53:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-29 02:53:48,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-29 02:53:48,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-29 02:53:48,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-29 02:53:48,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-29 02:53:48,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-29 02:53:48,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-29 02:53:48,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-29 02:53:48,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-29 02:53:48,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-29 02:53:48,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-29 02:53:48,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-29 02:53:48,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-29 02:53:48,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-29 02:53:48,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-29 02:53:48,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-29 02:53:48,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-29 02:53:48,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-29 02:53:48,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-29 02:53:48,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-29 02:53:48,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-29 02:53:48,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-29 02:53:48,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-29 02:53:48,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-29 02:53:48,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-29 02:53:48,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-29 02:53:48,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-29 02:53:48,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-29 02:53:48,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-29 02:53:48,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-29 02:53:48,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-29 02:53:48,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-29 02:53:48,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-29 02:53:48,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-29 02:53:48,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-29 02:53:48,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-29 02:53:48,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-29 02:53:48,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-29 02:53:48,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-29 02:53:48,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-29 02:53:48,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-29 02:53:48,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-29 02:53:48,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-29 02:53:48,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-29 02:53:48,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-29 02:53:48,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-29 02:53:48,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-29 02:53:48,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-29 02:53:48,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-29 02:53:48,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-29 02:53:48,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-29 02:53:48,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-29 02:53:48,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-29 02:53:48,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-29 02:53:48,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-29 02:53:48,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-29 02:53:48,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-29 02:53:48,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-29 02:53:48,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-29 02:53:48,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-29 02:53:48,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-29 02:53:48,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-29 02:53:48,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-29 02:53:48,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-29 02:53:48,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-29 02:53:48,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-29 02:53:48,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-29 02:53:48,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-29 02:53:48,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-29 02:53:48,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-29 02:53:48,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-29 02:53:48,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-29 02:53:48,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-29 02:53:48,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-29 02:53:48,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-29 02:53:48,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-29 02:53:48,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-29 02:53:48,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-29 02:53:48,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-29 02:53:48,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-29 02:53:48,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-29 02:53:48,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-29 02:53:48,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-29 02:53:48,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-29 02:53:48,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-29 02:53:48,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-29 02:53:48,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-29 02:53:48,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-29 02:53:48,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-29 02:53:48,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-29 02:53:48,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-29 02:53:48,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-29 02:53:48,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-29 02:53:48,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-29 02:53:48,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-29 02:53:48,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-29 02:53:49,023 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.914158292148118E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.909066927536782E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.909066927536782E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 02:53:49,024 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv622695514:82 , srv2130038467:83 , srv372177068:83 , srv376466799:83 , srv45306137:83 , srv575348855:83 , srv671300967:83 , srv850660886:83 , srv1083786200:84 , srv1173590814:84 , srv1211298562:84 , srv1291358295:84 , srv1301135483:84 , srv1357387009:84 , srv1381400068:84 , srv1383601966:84 , srv1453321812:84 , srv1530272425:84 , srv1570375596:84 , srv1605583587:84 , srv1696749078:84 , srv1709085905:84 , srv1912368617:84 , srv1949745280:84 , srv2020972211:84 , srv2071624044:84 , srv2123101866:84 , srv2699086:84 , srv378491318:84 , srv394793467:84 , srv492014377:84 , srv526791786:84 , srv527378034:84 , srv546667648:84 , srv734985817:84 , srv804990028:84 , srv80959634:84 , srv874201700:84 , srv875481994:84 , srv896841697:84 , srv995497987:84 , srv1038186288:85 , srv1056804226:85 , srv1084473663:85 , srv1181780157:85 , srv1252357142:85 , srv12607558:85 , srv1316612339:85 , srv1360192701:85 , srv1362232575:85 , srv1410641729:85 , srv1528894511:85 , srv154504776:85 , srv1557583544:85 , srv1609689979:85 , srv1648324776:85 , srv1665950797:85 , srv166660718:85 , srv1718197167:85 , srv1746812611:85 , srv1772557584:85 , srv1813019142:85 , srv1870159289:85 , srv1912691226:85 , srv1944845973:85 , srv2000441483:85 , srv2078891279:85 , srv2123159871:85 , srv2139687630:85 , srv222048042:85 , srv251550218:85 , srv316514595:85 , srv405526359:85 , srv427239674:85 , srv445983922:85 , srv498909885:85 , srv527712537:85 , srv539641945:85 , srv648196981:85 , srv705186133:85 , srv858105288:85 , srv920171789:85 , srv947869794:85 , srv948387237:85 , srv952022054:85 , srv998548958:85 , srv1417673539:86 , srv1472834402:86 , srv1512310625:86 , srv1933740486:86 , srv208905869:86 , srv247256714:86 , srv377219443:86 , srv454312366:86 , srv526967456:86 , srv657486363:86 , srv757031419:86 , srv1504133808:87 , srv938184466:87 , srv250824897:620 }
2023-11-29 02:53:49,052 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=326 (was 310) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=10084 (was 10902)
2023-11-29 02:53:49,061 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=326, ProcessCount=171, AvailableMemoryMB=10084
2023-11-29 02:53:49,061 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-29 02:53:49,062 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-11-29 02:53:49,076 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1189327534:84 , srv1375322467:84 , srv1412040944:84 , srv1443904456:84 , srv1488878417:84 , srv1502639956:84 , srv1603144428:84 , srv1643837444:84 , srv1785385782:84 , srv1962235752:84 , srv1983811154:84 , srv286350832:84 , srv381549045:84 , srv399501677:84 , srv490535414:84 , srv499940906:84 , srv621006972:84 , srv648100491:84 , srv661579558:84 , srv666708939:84 , srv703344260:84 , srv744374871:84 , srv868352477:84 , srv897954357:84 , srv93152082:84 , srv939792565:84 , srv960466197:84 , srv979470168:84 , srv98450356:84 , srv8856753:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-11-29 02:53:49,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-29 02:53:49,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1603144428=6, srv1502639956=5, srv1643837444=7, srv98450356=29, srv960466197=27, srv1962235752=9, srv1189327534=0, srv666708939=19, srv1412040944=2, srv8856753=23, srv381549045=12, srv286350832=11, srv490535414=14, srv621006972=16, srv939792565=26, srv1488878417=4, srv1785385782=8, srv1443904456=3, srv499940906=15, srv897954357=24, srv1375322467=1, srv661579558=18, srv399501677=13, srv93152082=25, srv979470168=28, srv1983811154=10, srv744374871=21, srv703344260=20, srv868352477=22, srv648100491=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-29 02:53:49,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-29 02:53:49,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-29 02:53:49,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-29 02:53:49,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-29 02:53:49,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-29 02:53:49,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-29 02:53:49,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-29 02:53:49,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-29 02:53:49,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-29 02:53:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-29 02:53:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-29 02:53:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-29 02:53:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-29 02:53:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-29 02:53:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-29 02:53:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-29 02:53:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-29 02:53:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-29 02:53:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-29 02:53:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-29 02:53:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-29 02:53:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-29 02:53:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-29 02:53:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-29 02:53:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-29 02:53:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-29 02:53:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-29 02:53:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-29 02:53:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-29 02:53:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-29 02:53:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-29 02:53:49,106 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0899801898145238, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 02:53:49,106 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912476567904263, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0899801898145238, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-11-29 02:55:49,094 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5877604 different iterations.  Found a solution that moves 1848 regions; Going from a computed imbalance of 0.9912476567904263 to a new imbalance of 4.313324190526491E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6844444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 02:55:49,101 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1189327534:90 , srv1375322467:90 , srv1412040944:90 , srv1443904456:90 , srv1488878417:90 , srv1502639956:90 , srv1603144428:90 , srv1643837444:90 , srv1785385782:90 , srv1962235752:90 , srv1983811154:90 , srv286350832:90 , srv381549045:90 , srv399501677:90 , srv490535414:90 , srv499940906:90 , srv621006972:90 , srv648100491:90 , srv661579558:90 , srv666708939:90 , srv703344260:90 , srv744374871:90 , srv868352477:90 , srv8856753:90 , srv897954357:90 , srv93152082:90 , srv939792565:90 , srv960466197:90 , srv979470168:90 , srv98450356:90 }
2023-11-29 02:55:49,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-29 02:55:49,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1603144428=6, srv1502639956=5, srv1643837444=7, srv98450356=29, srv960466197=27, srv1962235752=9, srv1189327534=0, srv666708939=19, srv1412040944=2, srv8856753=23, srv381549045=12, srv286350832=11, srv490535414=14, srv621006972=16, srv939792565=26, srv1488878417=4, srv1785385782=8, srv1443904456=3, srv499940906=15, srv897954357=24, srv1375322467=1, srv661579558=18, srv399501677=13, srv93152082=25, srv979470168=28, srv1983811154=10, srv744374871=21, srv703344260=20, srv868352477=22, srv648100491=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-29 02:55:49,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-29 02:55:49,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-29 02:55:49,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-29 02:55:49,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-29 02:55:49,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-29 02:55:49,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-29 02:55:49,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-29 02:55:49,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-29 02:55:49,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-29 02:55:49,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-29 02:55:49,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-29 02:55:49,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-29 02:55:49,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-29 02:55:49,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-29 02:55:49,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-29 02:55:49,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-29 02:55:49,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-29 02:55:49,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-29 02:55:49,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-29 02:55:49,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-29 02:55:49,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-29 02:55:49,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-29 02:55:49,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-29 02:55:49,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-29 02:55:49,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-29 02:55:49,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-29 02:55:49,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-29 02:55:49,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-29 02:55:49,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-29 02:55:49,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-29 02:55:49,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-29 02:55:49,127 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 02:55:49,127 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1189327534:90 , srv1375322467:90 , srv1412040944:90 , srv1443904456:90 , srv1488878417:90 , srv1502639956:90 , srv1603144428:90 , srv1643837444:90 , srv1785385782:90 , srv1962235752:90 , srv1983811154:90 , srv286350832:90 , srv381549045:90 , srv399501677:90 , srv490535414:90 , srv499940906:90 , srv621006972:90 , srv648100491:90 , srv661579558:90 , srv666708939:90 , srv703344260:90 , srv744374871:90 , srv868352477:90 , srv8856753:90 , srv897954357:90 , srv93152082:90 , srv939792565:90 , srv960466197:90 , srv979470168:90 , srv98450356:90 }
2023-11-29 02:55:49,142 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=187 (was 326), ProcessCount=171 (was 171), AvailableMemoryMB=9449 (was 10084)