2023-10-22 22:53:27,789 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-10-22 22:53:28,121 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-10-22 22:53:28,227 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-10-22 22:53:28,294 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-10-22 22:53:28,302 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-10-22 22:53:28,318 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=272, ProcessCount=172, AvailableMemoryMB=8500
2023-10-22 22:53:28,320 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-10-22 22:53:28,320 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-10-22 22:53:28,471 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1007607331:84 , srv1022973816:84 , srv1023276684:84 , srv1027979800:84 , srv1058711266:84 , srv107064593:84 , srv1075687707:84 , srv1084466770:84 , srv1087439530:84 , srv1140071490:84 , srv1157620697:84 , srv1182077292:84 , srv121329624:84 , srv1232477068:84 , srv1247164952:84 , srv1248483829:84 , srv1279630521:84 , srv1291140994:84 , srv1316652475:84 , srv1339874225:84 , srv1347495862:84 , srv1379224399:84 , srv1396217083:84 , srv140854808:84 , srv1429189616:84 , srv1452822155:84 , srv146062130:84 , srv150396474:84 , srv1504375668:84 , srv1531856927:84 , srv1532304861:84 , srv1534894078:84 , srv1547299848:84 , srv1613231456:84 , srv1649258664:84 , srv1656191639:84 , srv1678789883:84 , srv1683658022:84 , srv1692258562:84 , srv1697495783:84 , srv1736582157:84 , srv1807723559:84 , srv1825887212:84 , srv1850832432:84 , srv1851783818:84 , srv186312421:84 , srv1878805556:84 , srv1891874915:84 , srv1900761902:84 , srv1901194179:84 , srv1909798403:84 , srv1919531078:84 , srv192759712:84 , srv1936918115:84 , srv2002195564:84 , srv2015313323:84 , srv2016753138:84 , srv2026340828:84 , srv2057486074:84 , srv2075669607:84 , srv2080948273:84 , srv2131483172:84 , srv27736167:84 , srv277435602:84 , srv281280183:84 , srv351252706:84 , srv354105641:84 , srv361665856:84 , srv407310507:84 , srv423600172:84 , srv428033075:84 , srv439142962:84 , srv453991595:84 , srv471427227:84 , srv482384673:84 , srv484328292:84 , srv527067218:84 , srv532662977:84 , srv578058828:84 , srv578478509:84 , srv616447357:84 , srv641107530:84 , srv641593164:84 , srv6463812:84 , srv677179713:84 , srv67860473:84 , srv681500502:84 , srv764291561:84 , srv772989289:84 , srv803979093:84 , srv808851681:84 , srv81301004:84 , srv833921080:84 , srv84466400:84 , srv889730449:84 , srv896820505:84 , srv9229214:84 , srv956616249:84 , srv991916064:84 , srv123322624:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-10-22 22:53:28,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-22 22:53:28,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv107064593=5, srv1452822155=26, srv1919531078=52, srv764291561=88, srv277435602=64, srv1891874915=48, srv1900761902=49, srv1547299848=33, srv1649258664=35, srv808851681=91, srv1248483829=16, srv1339874225=20, srv146062130=27, srv354105641=67, srv1087439530=8, srv641107530=82, srv361665856=68, srv1084466770=7, srv1901194179=50, srv2131483172=62, srv471427227=74, srv6463812=84, srv1379224399=22, srv27736167=63, srv833921080=93, srv1182077292=11, srv1157620697=10, srv1027979800=3, srv67860473=86, srv991916064=99, srv578478509=80, srv123322624=14, srv1825887212=43, srv150396474=28, srv1656191639=36, srv681500502=87, srv1504375668=29, srv1936918115=54, srv578058828=79, srv677179713=85, srv81301004=92, srv1291140994=18, srv1022973816=1, srv1429189616=25, srv2080948273=61, srv9229214=97, srv1697495783=40, srv641593164=83, srv1316652475=19, srv889730449=95, srv84466400=94, srv1347495862=21, srv803979093=90, srv192759712=53, srv1279630521=17, srv1851783818=45, srv482384673=75, srv140854808=24, srv1678789883=37, srv121329624=12, srv1247164952=15, srv1692258562=39, srv351252706=66, srv956616249=98, srv1683658022=38, srv2075669607=60, srv1532304861=31, srv1232477068=13, srv1140071490=9, srv2015313323=56, srv2057486074=59, srv484328292=76, srv1023276684=2, srv1007607331=0, srv1909798403=51, srv616447357=81, srv1878805556=47, srv772989289=89, srv1613231456=34, srv186312421=46, srv1807723559=42, srv1058711266=4, srv896820505=96, srv532662977=78, srv1534894078=32, srv1531856927=30, srv2002195564=55, srv1736582157=41, srv407310507=69, srv423600172=70, srv1396217083=23, srv2016753138=57, srv428033075=71, srv1850832432=44, srv1075687707=6, srv281280183=65, srv453991595=73, srv2026340828=58, srv527067218=77, srv439142962=72} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-22 22:53:28,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-22 22:53:28,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-22 22:53:28,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-22 22:53:28,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-22 22:53:28,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-22 22:53:28,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-22 22:53:28,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-22 22:53:28,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-22 22:53:28,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-22 22:53:28,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-22 22:53:28,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-22 22:53:28,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-22 22:53:28,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-22 22:53:28,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-22 22:53:28,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-22 22:53:28,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-22 22:53:28,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-22 22:53:28,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-22 22:53:28,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-22 22:53:28,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-22 22:53:28,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-22 22:53:28,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-22 22:53:28,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-22 22:53:28,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-22 22:53:28,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-22 22:53:28,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-22 22:53:28,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-22 22:53:28,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-22 22:53:28,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-22 22:53:28,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-22 22:53:28,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-22 22:53:28,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-22 22:53:28,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-22 22:53:28,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-22 22:53:28,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-22 22:53:28,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-22 22:53:28,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-22 22:53:28,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-22 22:53:28,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-22 22:53:28,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-22 22:53:28,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-22 22:53:28,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-22 22:53:28,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-22 22:53:28,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-22 22:53:28,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-22 22:53:28,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-22 22:53:28,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-22 22:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-22 22:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-22 22:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-22 22:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-22 22:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-22 22:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-22 22:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-22 22:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-22 22:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-22 22:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-22 22:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-22 22:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-22 22:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-22 22:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-22 22:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-22 22:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-22 22:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-22 22:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-22 22:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-22 22:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-22 22:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-22 22:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-22 22:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-22 22:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-22 22:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-22 22:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-22 22:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-22 22:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-22 22:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-22 22:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-22 22:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-22 22:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-22 22:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-22 22:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-22 22:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-22 22:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-22 22:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-22 22:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-22 22:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-22 22:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-22 22:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-22 22:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-22 22:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-22 22:53:28,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-22 22:53:28,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-22 22:53:28,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-22 22:53:28,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-22 22:53:28,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-22 22:53:28,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-22 22:53:28,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-22 22:53:28,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-22 22:53:28,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-22 22:53:28,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-22 22:53:28,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-22 22:53:28,846 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-22 22:53:28,846 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-22 22:53:28,847 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-22 22:53:29,638 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 982 ms to try 5000 different iterations.  Found a solution that moves 4694 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1785920482540774. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04052775671957784, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05593807703525208, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5215555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04052775671957784, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17358333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24258333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-22 22:53:29,662 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1007607331:82 , srv1396217083:82 , srv192759712:82 , srv453991595:82 , srv616447357:82 , srv107064593:83 , srv1084466770:83 , srv1140071490:83 , srv1429189616:83 , srv1534894078:83 , srv1692258562:83 , srv1936918115:83 , srv2015313323:83 , srv641593164:83 , srv833921080:83 , srv1248483829:84 , srv1339874225:84 , srv1347495862:84 , srv150396474:84 , srv1547299848:84 , srv1649258664:84 , srv1678789883:84 , srv1851783818:84 , srv1891874915:84 , srv1901194179:84 , srv2057486074:84 , srv27736167:84 , srv281280183:84 , srv354105641:84 , srv1058711266:85 , srv1247164952:85 , srv1279630521:85 , srv1531856927:85 , srv1532304861:85 , srv1656191639:85 , srv1850832432:85 , srv1878805556:85 , srv1900761902:85 , srv1919531078:85 , srv2016753138:85 , srv2080948273:85 , srv439142962:85 , srv484328292:85 , srv677179713:85 , srv808851681:85 , srv956616249:85 , srv1075687707:86 , srv1291140994:86 , srv1613231456:86 , srv1825887212:86 , srv1909798403:86 , srv2075669607:86 , srv277435602:86 , srv351252706:86 , srv423600172:86 , srv578058828:86 , srv578478509:86 , srv6463812:86 , srv681500502:86 , srv991916064:86 , srv1379224399:87 , srv1697495783:87 , srv186312421:87 , srv428033075:87 , srv9229214:87 , srv1022973816:88 , srv140854808:88 , srv146062130:88 , srv1504375668:88 , srv2026340828:88 , srv2131483172:88 , srv361665856:88 , srv482384673:88 , srv527067218:88 , srv67860473:88 , srv81301004:88 , srv896820505:88 , srv1027979800:89 , srv1182077292:89 , srv1452822155:89 , srv1807723559:89 , srv471427227:89 , srv889730449:89 , srv1023276684:90 , srv1232477068:90 , srv1316652475:90 , srv1683658022:90 , srv641107530:90 , srv803979093:90 , srv532662977:91 , srv764291561:91 , srv772989289:91 , srv84466400:91 , srv121329624:92 , srv1736582157:92 , srv407310507:92 , srv1087439530:93 , srv2002195564:93 , srv1157620697:94 , srv123322624:450 }
2023-10-22 22:53:29,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-22 22:53:29,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv107064593=5, srv1452822155=26, srv1919531078=52, srv764291561=88, srv277435602=64, srv1891874915=48, srv1900761902=49, srv1547299848=33, srv1649258664=35, srv808851681=91, srv1248483829=16, srv1339874225=20, srv146062130=27, srv354105641=67, srv1087439530=8, srv641107530=82, srv361665856=68, srv1084466770=7, srv1901194179=50, srv2131483172=62, srv471427227=74, srv6463812=84, srv1379224399=22, srv27736167=63, srv833921080=93, srv1182077292=11, srv1157620697=10, srv1027979800=3, srv67860473=86, srv991916064=99, srv578478509=80, srv123322624=14, srv1825887212=43, srv150396474=28, srv1656191639=36, srv681500502=87, srv1504375668=29, srv1936918115=54, srv578058828=79, srv677179713=85, srv81301004=92, srv1291140994=18, srv1022973816=1, srv1429189616=25, srv2080948273=61, srv9229214=97, srv1697495783=40, srv641593164=83, srv1316652475=19, srv889730449=95, srv84466400=94, srv1347495862=21, srv803979093=90, srv192759712=53, srv1279630521=17, srv1851783818=45, srv482384673=75, srv140854808=24, srv1678789883=37, srv121329624=12, srv1247164952=15, srv1692258562=39, srv351252706=66, srv956616249=98, srv1683658022=38, srv2075669607=60, srv1532304861=31, srv1232477068=13, srv1140071490=9, srv2015313323=56, srv2057486074=59, srv484328292=76, srv1023276684=2, srv1007607331=0, srv1909798403=51, srv616447357=81, srv1878805556=47, srv772989289=89, srv1613231456=34, srv186312421=46, srv1807723559=42, srv1058711266=4, srv896820505=96, srv532662977=78, srv1534894078=32, srv1531856927=30, srv2002195564=55, srv1736582157=41, srv407310507=69, srv423600172=70, srv1396217083=23, srv2016753138=57, srv428033075=71, srv1850832432=44, srv1075687707=6, srv281280183=65, srv453991595=73, srv2026340828=58, srv527067218=77, srv439142962=72} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-22 22:53:29,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-22 22:53:29,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-22 22:53:29,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-22 22:53:29,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-22 22:53:29,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-22 22:53:29,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-22 22:53:29,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-22 22:53:29,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-22 22:53:29,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-22 22:53:29,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-22 22:53:29,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-22 22:53:29,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-22 22:53:29,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-22 22:53:29,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-22 22:53:29,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-22 22:53:29,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-22 22:53:29,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-22 22:53:29,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-22 22:53:29,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-22 22:53:29,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-22 22:53:29,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-22 22:53:29,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-22 22:53:29,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-22 22:53:29,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-22 22:53:29,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-22 22:53:29,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-22 22:53:29,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-22 22:53:29,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-22 22:53:29,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-22 22:53:29,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-22 22:53:29,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-22 22:53:29,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-22 22:53:29,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-22 22:53:29,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-22 22:53:29,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-22 22:53:29,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-22 22:53:29,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-22 22:53:29,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-22 22:53:29,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-22 22:53:29,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-22 22:53:29,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-22 22:53:29,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-22 22:53:29,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-22 22:53:29,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-22 22:53:29,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-22 22:53:29,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-22 22:53:29,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-22 22:53:29,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-22 22:53:29,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-22 22:53:29,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-22 22:53:29,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-22 22:53:29,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-22 22:53:29,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-22 22:53:29,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-22 22:53:29,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-22 22:53:29,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-22 22:53:29,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-22 22:53:29,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-22 22:53:29,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-22 22:53:29,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-22 22:53:29,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-22 22:53:29,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-22 22:53:29,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-22 22:53:29,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-22 22:53:29,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-22 22:53:29,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-22 22:53:29,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-22 22:53:29,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-22 22:53:29,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-22 22:53:29,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-22 22:53:29,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-22 22:53:29,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-22 22:53:29,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-22 22:53:29,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-22 22:53:29,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-22 22:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-22 22:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-22 22:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-22 22:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-22 22:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-22 22:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-22 22:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-22 22:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-22 22:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-22 22:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-22 22:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-22 22:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-22 22:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-22 22:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-22 22:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-22 22:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-22 22:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-22 22:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-22 22:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-22 22:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-22 22:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-22 22:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-22 22:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-22 22:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-22 22:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-22 22:53:29,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-22 22:53:29,835 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04052775671957784, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05593807703525208, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04052775671957784, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17358333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24258333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-22 22:53:29,835 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-22 22:53:29,835 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1785591801635736, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04052775671957784, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05593807703525208, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04052775671957784, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17358333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24258333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-22 22:53:30,127 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 404 ms to try 5000 different iterations.  Found a solution that moves 2621 regions; Going from a computed imbalance of 0.1785591801635736 to a new imbalance of 0.015706460958171874. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023181323028976093); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03537200094959926, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2912222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023181323028976093); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012166666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04958333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-22 22:53:30,140 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1087439530:77 , srv1157620697:78 , srv407310507:79 , srv1316652475:80 , srv2002195564:80 , srv84466400:80 , srv1027979800:81 , srv121329624:81 , srv1736582157:81 , srv532662977:81 , srv772989289:81 , srv1232477068:82 , srv1683658022:82 , srv803979093:82 , srv889730449:82 , srv896820505:82 , srv1022973816:83 , srv1023276684:83 , srv1452822155:83 , srv146062130:83 , srv527067218:83 , srv641107530:83 , srv1291140994:84 , srv140854808:84 , srv1825887212:84 , srv2026340828:84 , srv2131483172:84 , srv423600172:84 , srv471427227:84 , srv764291561:84 , srv1075687707:85 , srv1182077292:85 , srv1347495862:85 , srv1379224399:85 , srv1504375668:85 , srv1532304861:85 , srv1613231456:85 , srv1678789883:85 , srv1807723559:85 , srv186312421:85 , srv351252706:85 , srv361665856:85 , srv67860473:85 , srv81301004:85 , srv1058711266:86 , srv1279630521:86 , srv1649258664:86 , srv1900761902:86 , srv1901194179:86 , srv1909798403:86 , srv1919531078:86 , srv2016753138:86 , srv2080948273:86 , srv354105641:86 , srv482384673:86 , srv578478509:86 , srv641593164:86 , srv6463812:86 , srv9229214:86 , srv107064593:87 , srv1084466770:87 , srv1429189616:87 , srv1531856927:87 , srv1534894078:87 , srv1547299848:87 , srv1692258562:87 , srv1697495783:87 , srv1850832432:87 , srv2015313323:87 , srv277435602:87 , srv808851681:87 , srv991916064:87 , srv1140071490:88 , srv1247164952:88 , srv1339874225:88 , srv150396474:88 , srv1878805556:88 , srv2057486074:88 , srv27736167:88 , srv428033075:88 , srv439142962:88 , srv484328292:88 , srv578058828:88 , srv677179713:88 , srv681500502:88 , srv1007607331:89 , srv1396217083:89 , srv1891874915:89 , srv1936918115:89 , srv2075669607:89 , srv281280183:89 , srv616447357:89 , srv956616249:89 , srv1248483829:90 , srv1656191639:90 , srv192759712:90 , srv453991595:90 , srv1851783818:92 , srv833921080:92 , srv123322624:530 }
2023-10-22 22:53:30,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-22 22:53:30,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv107064593=5, srv1452822155=26, srv1919531078=52, srv764291561=88, srv277435602=64, srv1891874915=48, srv1900761902=49, srv1547299848=33, srv1649258664=35, srv808851681=91, srv1248483829=16, srv1339874225=20, srv146062130=27, srv354105641=67, srv1087439530=8, srv641107530=82, srv361665856=68, srv1084466770=7, srv1901194179=50, srv2131483172=62, srv471427227=74, srv6463812=84, srv1379224399=22, srv27736167=63, srv833921080=93, srv1182077292=11, srv1157620697=10, srv1027979800=3, srv67860473=86, srv991916064=99, srv578478509=80, srv123322624=14, srv1825887212=43, srv150396474=28, srv1656191639=36, srv681500502=87, srv1504375668=29, srv1936918115=54, srv578058828=79, srv677179713=85, srv81301004=92, srv1291140994=18, srv1022973816=1, srv1429189616=25, srv2080948273=61, srv9229214=97, srv1697495783=40, srv641593164=83, srv1316652475=19, srv889730449=95, srv84466400=94, srv1347495862=21, srv803979093=90, srv192759712=53, srv1279630521=17, srv1851783818=45, srv482384673=75, srv140854808=24, srv1678789883=37, srv121329624=12, srv1247164952=15, srv1692258562=39, srv351252706=66, srv956616249=98, srv1683658022=38, srv2075669607=60, srv1532304861=31, srv1232477068=13, srv1140071490=9, srv2015313323=56, srv2057486074=59, srv484328292=76, srv1023276684=2, srv1007607331=0, srv1909798403=51, srv616447357=81, srv1878805556=47, srv772989289=89, srv1613231456=34, srv186312421=46, srv1807723559=42, srv1058711266=4, srv896820505=96, srv532662977=78, srv1534894078=32, srv1531856927=30, srv2002195564=55, srv1736582157=41, srv407310507=69, srv423600172=70, srv1396217083=23, srv2016753138=57, srv428033075=71, srv1850832432=44, srv1075687707=6, srv281280183=65, srv453991595=73, srv2026340828=58, srv527067218=77, srv439142962=72} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-22 22:53:30,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-22 22:53:30,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-22 22:53:30,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-22 22:53:30,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-22 22:53:30,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-22 22:53:30,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-22 22:53:30,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-22 22:53:30,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-22 22:53:30,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-22 22:53:30,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-22 22:53:30,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-22 22:53:30,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-22 22:53:30,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-22 22:53:30,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-22 22:53:30,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-22 22:53:30,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-22 22:53:30,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-22 22:53:30,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-22 22:53:30,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-22 22:53:30,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-22 22:53:30,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-22 22:53:30,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-22 22:53:30,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-22 22:53:30,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-22 22:53:30,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-22 22:53:30,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-22 22:53:30,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-22 22:53:30,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-22 22:53:30,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-22 22:53:30,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-22 22:53:30,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-22 22:53:30,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-22 22:53:30,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-22 22:53:30,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-22 22:53:30,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-22 22:53:30,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-22 22:53:30,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-22 22:53:30,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-22 22:53:30,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-22 22:53:30,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-22 22:53:30,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-22 22:53:30,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-22 22:53:30,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-22 22:53:30,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-22 22:53:30,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-22 22:53:30,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-22 22:53:30,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-22 22:53:30,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-22 22:53:30,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-22 22:53:30,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-22 22:53:30,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-22 22:53:30,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-22 22:53:30,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-22 22:53:30,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-22 22:53:30,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-22 22:53:30,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-22 22:53:30,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-22 22:53:30,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-22 22:53:30,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-22 22:53:30,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-22 22:53:30,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-22 22:53:30,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-22 22:53:30,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-22 22:53:30,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-22 22:53:30,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-22 22:53:30,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-22 22:53:30,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-22 22:53:30,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-22 22:53:30,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-22 22:53:30,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-22 22:53:30,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-22 22:53:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-22 22:53:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-22 22:53:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-22 22:53:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-22 22:53:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-22 22:53:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-22 22:53:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-22 22:53:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-22 22:53:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-22 22:53:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-22 22:53:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-22 22:53:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-22 22:53:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-22 22:53:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-22 22:53:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-22 22:53:30,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-22 22:53:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-22 22:53:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-22 22:53:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-22 22:53:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-22 22:53:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-22 22:53:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-22 22:53:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-22 22:53:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-22 22:53:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-22 22:53:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-22 22:53:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-22 22:53:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-22 22:53:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-22 22:53:30,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-22 22:53:30,313 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023181323028976093); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03537200094959926, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023181323028976093); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012166666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04958333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-22 22:53:30,314 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-22 22:53:30,314 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015688108323913156, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023181323028976093); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03537200094959926, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023181323028976093); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012166666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04958333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-22 22:53:30,547 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 343 ms to try 5000 different iterations.  Found a solution that moves 1041 regions; Going from a computed imbalance of 0.015688108323913156 to a new imbalance of 7.358911399016182E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004777112160979515); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006750316948122676); MoveCostFunction : (multiplier=7.0, imbalance=0.11566666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004777112160979515); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-22 22:53:30,549 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1851783818:82 , srv764291561:82 , srv833921080:82 , srv1656191639:83 , srv1697495783:83 , srv2075669607:83 , srv428033075:83 , srv578058828:83 , srv1022973816:84 , srv1023276684:84 , srv1182077292:84 , srv121329624:84 , srv1247164952:84 , srv1248483829:84 , srv1807723559:84 , srv277435602:84 , srv281280183:84 , srv361665856:84 , srv471427227:84 , srv484328292:84 , srv641107530:84 , srv681500502:84 , srv956616249:84 , srv991916064:84 , srv1140071490:85 , srv1232477068:85 , srv1452822155:85 , srv150396474:85 , srv1504375668:85 , srv1683658022:85 , srv1736582157:85 , srv1878805556:85 , srv1909798403:85 , srv192759712:85 , srv1936918115:85 , srv2002195564:85 , srv2026340828:85 , srv2131483172:85 , srv439142962:85 , srv482384673:85 , srv677179713:85 , srv772989289:85 , srv803979093:85 , srv81301004:85 , srv9229214:85 , srv1075687707:86 , srv1157620697:86 , srv1279630521:86 , srv1339874225:86 , srv1379224399:86 , srv140854808:86 , srv1531856927:86 , srv1613231456:86 , srv1850832432:86 , srv186312421:86 , srv1891874915:86 , srv1900761902:86 , srv1919531078:86 , srv2016753138:86 , srv2057486074:86 , srv2080948273:86 , srv351252706:86 , srv407310507:86 , srv453991595:86 , srv527067218:86 , srv532662977:86 , srv578478509:86 , srv616447357:86 , srv6463812:86 , srv67860473:86 , srv808851681:86 , srv1007607331:87 , srv1058711266:87 , srv1084466770:87 , srv1316652475:87 , srv1396217083:87 , srv1429189616:87 , srv1547299848:87 , srv1692258562:87 , srv1825887212:87 , srv1901194179:87 , srv2015313323:87 , srv27736167:87 , srv84466400:87 , srv889730449:87 , srv1027979800:88 , srv107064593:88 , srv1087439530:88 , srv1291140994:88 , srv1347495862:88 , srv146062130:88 , srv1534894078:88 , srv423600172:88 , srv641593164:88 , srv896820505:88 , srv1532304861:89 , srv1649258664:89 , srv1678789883:89 , srv354105641:89 , srv123322624:520 }
2023-10-22 22:53:30,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-22 22:53:30,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv107064593=5, srv1452822155=26, srv1919531078=52, srv764291561=88, srv277435602=64, srv1891874915=48, srv1900761902=49, srv1547299848=33, srv1649258664=35, srv808851681=91, srv1248483829=16, srv1339874225=20, srv146062130=27, srv354105641=67, srv1087439530=8, srv641107530=82, srv361665856=68, srv1084466770=7, srv1901194179=50, srv2131483172=62, srv471427227=74, srv6463812=84, srv1379224399=22, srv27736167=63, srv833921080=93, srv1182077292=11, srv1157620697=10, srv1027979800=3, srv67860473=86, srv991916064=99, srv578478509=80, srv123322624=14, srv1825887212=43, srv150396474=28, srv1656191639=36, srv681500502=87, srv1504375668=29, srv1936918115=54, srv578058828=79, srv677179713=85, srv81301004=92, srv1291140994=18, srv1022973816=1, srv1429189616=25, srv2080948273=61, srv9229214=97, srv1697495783=40, srv641593164=83, srv1316652475=19, srv889730449=95, srv84466400=94, srv1347495862=21, srv803979093=90, srv192759712=53, srv1279630521=17, srv1851783818=45, srv482384673=75, srv140854808=24, srv1678789883=37, srv121329624=12, srv1247164952=15, srv1692258562=39, srv351252706=66, srv956616249=98, srv1683658022=38, srv2075669607=60, srv1532304861=31, srv1232477068=13, srv1140071490=9, srv2015313323=56, srv2057486074=59, srv484328292=76, srv1023276684=2, srv1007607331=0, srv1909798403=51, srv616447357=81, srv1878805556=47, srv772989289=89, srv1613231456=34, srv186312421=46, srv1807723559=42, srv1058711266=4, srv896820505=96, srv532662977=78, srv1534894078=32, srv1531856927=30, srv2002195564=55, srv1736582157=41, srv407310507=69, srv423600172=70, srv1396217083=23, srv2016753138=57, srv428033075=71, srv1850832432=44, srv1075687707=6, srv281280183=65, srv453991595=73, srv2026340828=58, srv527067218=77, srv439142962=72} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-22 22:53:30,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-22 22:53:30,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-22 22:53:30,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-22 22:53:30,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-22 22:53:30,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-22 22:53:30,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-22 22:53:30,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-22 22:53:30,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-22 22:53:30,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-22 22:53:30,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-22 22:53:30,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-22 22:53:30,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-22 22:53:30,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-22 22:53:30,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-22 22:53:30,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-22 22:53:30,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-22 22:53:30,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-22 22:53:30,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-22 22:53:30,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-22 22:53:30,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-22 22:53:30,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-22 22:53:30,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-22 22:53:30,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-22 22:53:30,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-22 22:53:30,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-22 22:53:30,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-22 22:53:30,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-22 22:53:30,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-22 22:53:30,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-22 22:53:30,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-22 22:53:30,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-22 22:53:30,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-22 22:53:30,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-22 22:53:30,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-22 22:53:30,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-22 22:53:30,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-22 22:53:30,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-22 22:53:30,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-22 22:53:30,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-22 22:53:30,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-22 22:53:30,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-22 22:53:30,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-22 22:53:30,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-22 22:53:30,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-22 22:53:30,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-22 22:53:30,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-22 22:53:30,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-22 22:53:30,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-22 22:53:30,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-22 22:53:30,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-22 22:53:30,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-22 22:53:30,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-22 22:53:30,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-22 22:53:30,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-22 22:53:30,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-22 22:53:30,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-22 22:53:30,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-22 22:53:30,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-22 22:53:30,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-22 22:53:30,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-22 22:53:30,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-22 22:53:30,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-22 22:53:30,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-22 22:53:30,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-22 22:53:30,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-22 22:53:30,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-22 22:53:30,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-22 22:53:30,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-22 22:53:30,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-22 22:53:30,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-22 22:53:30,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-22 22:53:30,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-22 22:53:30,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-22 22:53:30,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-22 22:53:30,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-22 22:53:30,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-22 22:53:30,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-22 22:53:30,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-22 22:53:30,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-22 22:53:30,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-22 22:53:30,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-22 22:53:30,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-22 22:53:30,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-22 22:53:30,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-22 22:53:30,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-22 22:53:30,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-22 22:53:30,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-22 22:53:30,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-22 22:53:30,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-22 22:53:30,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-22 22:53:30,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-22 22:53:30,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-22 22:53:30,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-22 22:53:30,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-22 22:53:30,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-22 22:53:30,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-22 22:53:30,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-22 22:53:30,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-22 22:53:30,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-22 22:53:30,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-22 22:53:30,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-22 22:53:30,700 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004777112160979515); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006750316948122676); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004777112160979515); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-22 22:53:30,701 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-22 22:53:30,701 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.286019021056149E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004777112160979515); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006750316948122676); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004777112160979515); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-22 22:53:31,046 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 459 ms to try 5000 different iterations.  Found a solution that moves 231 regions; Going from a computed imbalance of 7.286019021056149E-4 to a new imbalance of 1.9902352446880915E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.025666666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-22 22:53:31,047 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1379224399:83 , srv1649258664:83 , srv1891874915:83 , srv27736167:83 , srv354105641:83 , srv453991595:83 , srv482384673:83 , srv67860473:83 , srv84466400:83 , srv956616249:83 , srv1007607331:84 , srv107064593:84 , srv1075687707:84 , srv1087439530:84 , srv1157620697:84 , srv1248483829:84 , srv1291140994:84 , srv1339874225:84 , srv1347495862:84 , srv1396217083:84 , srv140854808:84 , srv1452822155:84 , srv146062130:84 , srv1504375668:84 , srv1531856927:84 , srv1532304861:84 , srv1534894078:84 , srv1613231456:84 , srv1656191639:84 , srv1678789883:84 , srv1736582157:84 , srv1807723559:84 , srv1850832432:84 , srv1851783818:84 , srv186312421:84 , srv1878805556:84 , srv2015313323:84 , srv2075669607:84 , srv423600172:84 , srv428033075:84 , srv439142962:84 , srv532662977:84 , srv578478509:84 , srv616447357:84 , srv6463812:84 , srv677179713:84 , srv681500502:84 , srv803979093:84 , srv808851681:84 , srv81301004:84 , srv889730449:84 , srv9229214:84 , srv1023276684:85 , srv1027979800:85 , srv1058711266:85 , srv1084466770:85 , srv1182077292:85 , srv121329624:85 , srv1232477068:85 , srv1247164952:85 , srv1279630521:85 , srv1316652475:85 , srv1429189616:85 , srv150396474:85 , srv1547299848:85 , srv1683658022:85 , srv1692258562:85 , srv1697495783:85 , srv1825887212:85 , srv1900761902:85 , srv1901194179:85 , srv1909798403:85 , srv192759712:85 , srv1936918115:85 , srv2002195564:85 , srv2016753138:85 , srv2026340828:85 , srv2057486074:85 , srv2080948273:85 , srv2131483172:85 , srv277435602:85 , srv281280183:85 , srv351252706:85 , srv361665856:85 , srv407310507:85 , srv471427227:85 , srv484328292:85 , srv527067218:85 , srv578058828:85 , srv641107530:85 , srv764291561:85 , srv772989289:85 , srv833921080:85 , srv896820505:85 , srv991916064:85 , srv1140071490:86 , srv1919531078:86 , srv641593164:86 , srv1022973816:87 , srv123322624:642 }
2023-10-22 22:53:31,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-22 22:53:31,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv107064593=5, srv1452822155=26, srv1919531078=52, srv764291561=88, srv277435602=64, srv1891874915=48, srv1900761902=49, srv1547299848=33, srv1649258664=35, srv808851681=91, srv1248483829=16, srv1339874225=20, srv146062130=27, srv354105641=67, srv1087439530=8, srv641107530=82, srv361665856=68, srv1084466770=7, srv1901194179=50, srv2131483172=62, srv471427227=74, srv6463812=84, srv1379224399=22, srv27736167=63, srv833921080=93, srv1182077292=11, srv1157620697=10, srv1027979800=3, srv67860473=86, srv991916064=99, srv578478509=80, srv123322624=14, srv1825887212=43, srv150396474=28, srv1656191639=36, srv681500502=87, srv1504375668=29, srv1936918115=54, srv578058828=79, srv677179713=85, srv81301004=92, srv1291140994=18, srv1022973816=1, srv1429189616=25, srv2080948273=61, srv9229214=97, srv1697495783=40, srv641593164=83, srv1316652475=19, srv889730449=95, srv84466400=94, srv1347495862=21, srv803979093=90, srv192759712=53, srv1279630521=17, srv1851783818=45, srv482384673=75, srv140854808=24, srv1678789883=37, srv121329624=12, srv1247164952=15, srv1692258562=39, srv351252706=66, srv956616249=98, srv1683658022=38, srv2075669607=60, srv1532304861=31, srv1232477068=13, srv1140071490=9, srv2015313323=56, srv2057486074=59, srv484328292=76, srv1023276684=2, srv1007607331=0, srv1909798403=51, srv616447357=81, srv1878805556=47, srv772989289=89, srv1613231456=34, srv186312421=46, srv1807723559=42, srv1058711266=4, srv896820505=96, srv532662977=78, srv1534894078=32, srv1531856927=30, srv2002195564=55, srv1736582157=41, srv407310507=69, srv423600172=70, srv1396217083=23, srv2016753138=57, srv428033075=71, srv1850832432=44, srv1075687707=6, srv281280183=65, srv453991595=73, srv2026340828=58, srv527067218=77, srv439142962=72} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-22 22:53:31,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-22 22:53:31,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-22 22:53:31,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-22 22:53:31,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-22 22:53:31,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-22 22:53:31,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-22 22:53:31,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-22 22:53:31,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-22 22:53:31,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-22 22:53:31,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-22 22:53:31,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-22 22:53:31,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-22 22:53:31,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-22 22:53:31,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-22 22:53:31,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-22 22:53:31,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-22 22:53:31,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-22 22:53:31,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-22 22:53:31,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-22 22:53:31,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-22 22:53:31,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-22 22:53:31,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-22 22:53:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-22 22:53:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-22 22:53:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-22 22:53:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-22 22:53:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-22 22:53:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-22 22:53:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-22 22:53:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-22 22:53:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-22 22:53:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-22 22:53:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-22 22:53:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-22 22:53:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-22 22:53:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-22 22:53:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-22 22:53:31,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-22 22:53:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-22 22:53:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-22 22:53:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-22 22:53:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-22 22:53:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-22 22:53:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-22 22:53:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-22 22:53:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-22 22:53:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-22 22:53:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-22 22:53:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-22 22:53:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-22 22:53:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-22 22:53:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-22 22:53:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-22 22:53:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-22 22:53:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-22 22:53:31,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-22 22:53:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-22 22:53:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-22 22:53:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-22 22:53:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-22 22:53:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-22 22:53:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-22 22:53:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-22 22:53:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-22 22:53:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-22 22:53:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-22 22:53:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-22 22:53:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-22 22:53:31,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-22 22:53:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-22 22:53:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-22 22:53:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-22 22:53:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-22 22:53:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-22 22:53:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-22 22:53:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-22 22:53:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-22 22:53:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-22 22:53:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-22 22:53:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-22 22:53:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-22 22:53:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-22 22:53:31,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-22 22:53:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-22 22:53:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-22 22:53:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-22 22:53:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-22 22:53:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-22 22:53:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-22 22:53:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-22 22:53:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-22 22:53:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-22 22:53:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-22 22:53:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-22 22:53:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-22 22:53:31,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-22 22:53:31,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-22 22:53:31,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-22 22:53:31,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-22 22:53:31,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-22 22:53:31,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-22 22:53:31,195 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.8284855875433483E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-22 22:53:31,195 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1379224399:83 , srv1649258664:83 , srv1891874915:83 , srv27736167:83 , srv354105641:83 , srv453991595:83 , srv482384673:83 , srv67860473:83 , srv84466400:83 , srv956616249:83 , srv1007607331:84 , srv107064593:84 , srv1075687707:84 , srv1087439530:84 , srv1157620697:84 , srv1248483829:84 , srv1291140994:84 , srv1339874225:84 , srv1347495862:84 , srv1396217083:84 , srv140854808:84 , srv1452822155:84 , srv146062130:84 , srv1504375668:84 , srv1531856927:84 , srv1532304861:84 , srv1534894078:84 , srv1613231456:84 , srv1656191639:84 , srv1678789883:84 , srv1736582157:84 , srv1807723559:84 , srv1850832432:84 , srv1851783818:84 , srv186312421:84 , srv1878805556:84 , srv2015313323:84 , srv2075669607:84 , srv423600172:84 , srv428033075:84 , srv439142962:84 , srv532662977:84 , srv578478509:84 , srv616447357:84 , srv6463812:84 , srv677179713:84 , srv681500502:84 , srv803979093:84 , srv808851681:84 , srv81301004:84 , srv889730449:84 , srv9229214:84 , srv1023276684:85 , srv1027979800:85 , srv1058711266:85 , srv1084466770:85 , srv1182077292:85 , srv121329624:85 , srv1232477068:85 , srv1247164952:85 , srv1279630521:85 , srv1316652475:85 , srv1429189616:85 , srv150396474:85 , srv1547299848:85 , srv1683658022:85 , srv1692258562:85 , srv1697495783:85 , srv1825887212:85 , srv1900761902:85 , srv1901194179:85 , srv1909798403:85 , srv192759712:85 , srv1936918115:85 , srv2002195564:85 , srv2016753138:85 , srv2026340828:85 , srv2057486074:85 , srv2080948273:85 , srv2131483172:85 , srv277435602:85 , srv281280183:85 , srv351252706:85 , srv361665856:85 , srv407310507:85 , srv471427227:85 , srv484328292:85 , srv527067218:85 , srv578058828:85 , srv641107530:85 , srv764291561:85 , srv772989289:85 , srv833921080:85 , srv896820505:85 , srv991916064:85 , srv1140071490:86 , srv1919531078:86 , srv641593164:86 , srv1022973816:87 , srv123322624:642 }
2023-10-22 22:53:31,234 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=290 (was 272) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=7993 (was 8500)
2023-10-22 22:53:31,242 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=290, ProcessCount=172, AvailableMemoryMB=7992
2023-10-22 22:53:31,243 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-10-22 22:53:31,243 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-10-22 22:53:31,256 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1011482351:84 , srv1011494928:84 , srv1051778768:84 , srv1111483528:84 , srv1152334688:84 , srv1159295371:84 , srv1339735183:84 , srv1405438425:84 , srv1470410692:84 , srv1546665297:84 , srv1588857186:84 , srv174540322:84 , srv1826057923:84 , srv1942759120:84 , srv2055473134:84 , srv2080735281:84 , srv359058888:84 , srv368618309:84 , srv52591479:84 , srv624941668:84 , srv697689102:84 , srv803171071:84 , srv828272409:84 , srv844391610:84 , srv856251702:84 , srv93310737:84 , srv939208833:84 , srv939334945:84 , srv942173500:84 , srv1666914245:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-10-22 22:53:31,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-22 22:53:31,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1339735183=6, srv52591479=19, srv1159295371=5, srv1405438425=7, srv697689102=21, srv942173500=29, srv624941668=20, srv828272409=23, srv803171071=22, srv1666914245=11, srv939208833=27, srv1011482351=0, srv1942759120=14, srv93310737=26, srv856251702=25, srv1152334688=4, srv2080735281=16, srv1051778768=2, srv2055473134=15, srv1470410692=8, srv939334945=28, srv1826057923=13, srv1011494928=1, srv368618309=18, srv844391610=24, srv359058888=17, srv1546665297=9, srv174540322=12, srv1111483528=3, srv1588857186=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-22 22:53:31,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-22 22:53:31,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-22 22:53:31,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-22 22:53:31,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-22 22:53:31,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-22 22:53:31,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-22 22:53:31,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-22 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-22 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-22 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-22 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-22 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-22 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-22 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-22 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-22 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-22 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-22 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-22 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-22 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-22 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-22 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-22 22:53:31,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-22 22:53:31,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-22 22:53:31,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-22 22:53:31,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-22 22:53:31,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-22 22:53:31,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-22 22:53:31,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-22 22:53:31,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-22 22:53:31,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-10-22 22:53:31,299 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.05129591486132, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-22 22:53:31,299 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912354675017043, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.05129591486132, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-10-22 22:55:31,278 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6428392 different iterations.  Found a solution that moves 1847 regions; Going from a computed imbalance of 0.9912354675017043 to a new imbalance of 4.3109901406398434E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6840740740740741, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-22 22:55:31,285 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1011482351:90 , srv1011494928:90 , srv1051778768:90 , srv1111483528:90 , srv1152334688:90 , srv1159295371:90 , srv1339735183:90 , srv1405438425:90 , srv1470410692:90 , srv1546665297:90 , srv1588857186:90 , srv1666914245:90 , srv174540322:90 , srv1826057923:90 , srv1942759120:90 , srv2055473134:90 , srv2080735281:90 , srv359058888:90 , srv368618309:90 , srv52591479:90 , srv624941668:90 , srv697689102:90 , srv803171071:90 , srv828272409:90 , srv844391610:90 , srv856251702:90 , srv93310737:90 , srv939208833:90 , srv939334945:90 , srv942173500:90 }
2023-10-22 22:55:31,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-22 22:55:31,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1339735183=6, srv52591479=19, srv1159295371=5, srv1405438425=7, srv697689102=21, srv942173500=29, srv624941668=20, srv828272409=23, srv803171071=22, srv1666914245=11, srv939208833=27, srv1011482351=0, srv1942759120=14, srv93310737=26, srv856251702=25, srv1152334688=4, srv2080735281=16, srv1051778768=2, srv2055473134=15, srv1470410692=8, srv939334945=28, srv1826057923=13, srv1011494928=1, srv368618309=18, srv844391610=24, srv359058888=17, srv1546665297=9, srv174540322=12, srv1111483528=3, srv1588857186=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-22 22:55:31,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-22 22:55:31,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-22 22:55:31,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-22 22:55:31,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-22 22:55:31,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-22 22:55:31,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-22 22:55:31,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-22 22:55:31,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-22 22:55:31,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-22 22:55:31,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-22 22:55:31,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-22 22:55:31,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-22 22:55:31,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-22 22:55:31,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-22 22:55:31,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-22 22:55:31,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-22 22:55:31,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-22 22:55:31,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-22 22:55:31,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-22 22:55:31,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-22 22:55:31,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-22 22:55:31,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-22 22:55:31,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-22 22:55:31,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-22 22:55:31,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-22 22:55:31,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-22 22:55:31,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-22 22:55:31,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-22 22:55:31,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-22 22:55:31,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-22 22:55:31,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-10-22 22:55:31,306 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-22 22:55:31,306 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1011482351:90 , srv1011494928:90 , srv1051778768:90 , srv1111483528:90 , srv1152334688:90 , srv1159295371:90 , srv1339735183:90 , srv1405438425:90 , srv1470410692:90 , srv1546665297:90 , srv1588857186:90 , srv1666914245:90 , srv174540322:90 , srv1826057923:90 , srv1942759120:90 , srv2055473134:90 , srv2080735281:90 , srv359058888:90 , srv368618309:90 , srv52591479:90 , srv624941668:90 , srv697689102:90 , srv803171071:90 , srv828272409:90 , srv844391610:90 , srv856251702:90 , srv93310737:90 , srv939208833:90 , srv939334945:90 , srv942173500:90 }
2023-10-22 22:55:31,319 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=167 (was 290), ProcessCount=168 (was 172), AvailableMemoryMB=7970 (was 7992)