2023-03-22 06:56:03,887 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-03-22 06:56:04,284 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-03-22 06:56:04,431 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-03-22 06:56:04,511 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-22 06:56:04,521 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-22 06:56:04,537 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=365, ProcessCount=169, AvailableMemoryMB=7936
2023-03-22 06:56:04,540 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-22 06:56:04,541 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-22 06:56:04,707 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1002646601:84 , srv1006518286:84 , srv100875542:84 , srv1030188530:84 , srv1041625893:84 , srv1079648870:84 , srv1105078022:84 , srv1110884203:84 , srv1110904728:84 , srv1135069152:84 , srv1162712989:84 , srv1169473706:84 , srv1170201750:84 , srv1188076735:84 , srv1203194854:84 , srv122938707:84 , srv1242101469:84 , srv1289073926:84 , srv12934870:84 , srv131644742:84 , srv1322250421:84 , srv1348352588:84 , srv1370155698:84 , srv1390090356:84 , srv1406373300:84 , srv1447536367:84 , srv144871772:84 , srv1497048544:84 , srv1506022996:84 , srv1508774750:84 , srv1524483021:84 , srv1533937060:84 , srv1560520481:84 , srv1608283712:84 , srv1639347313:84 , srv1654442476:84 , srv1679936725:84 , srv1698078250:84 , srv1704984628:84 , srv1751903448:84 , srv1772817100:84 , srv1773157938:84 , srv1798954078:84 , srv1805856262:84 , srv1810766173:84 , srv1862786447:84 , srv1871410686:84 , srv1880938254:84 , srv1894941521:84 , srv1897414144:84 , srv1913070771:84 , srv1933091132:84 , srv1945693460:84 , srv1964755339:84 , srv1999086241:84 , srv2016212438:84 , srv2069768582:84 , srv2078705847:84 , srv2108230933:84 , srv2130245351:84 , srv242733953:84 , srv268202747:84 , srv273198626:84 , srv279041936:84 , srv301959504:84 , srv34764458:84 , srv348200921:84 , srv365380694:84 , srv373291152:84 , srv38034034:84 , srv383910674:84 , srv413045583:84 , srv421218815:84 , srv470867022:84 , srv471172571:84 , srv491042920:84 , srv505863123:84 , srv513349648:84 , srv51392575:84 , srv519902912:84 , srv522400634:84 , srv621174026:84 , srv629848040:84 , srv642016406:84 , srv656333699:84 , srv677706278:84 , srv780898834:84 , srv792001553:84 , srv80715395:84 , srv810833658:84 , srv817423182:84 , srv822845862:84 , srv824912135:84 , srv839235862:84 , srv937564275:84 , srv94095876:84 , srv975110211:84 , srv990274269:84 , srv991218668:84 , srv997666813:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-03-22 06:56:04,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-22 06:56:04,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1913070771=50, srv990274269=97, srv122938707=15, srv80715395=88, srv1289073926=17, srv822845862=91, srv1772817100=40, srv937564275=94, srv273198626=62, srv1110904728=8, srv94095876=95, srv100875542=2, srv383910674=70, srv1390090356=23, srv1560520481=32, srv1773157938=41, srv629848040=82, srv656333699=84, srv1370155698=22, srv242733953=60, srv1679936725=36, srv1105078022=6, srv2130245351=59, srv301959504=64, srv1654442476=35, srv1964755339=53, srv2069768582=56, srv991218668=98, srv1897414144=49, srv1805856262=43, srv1079648870=5, srv1945693460=52, srv1933091132=51, srv279041936=63, srv131644742=19, srv1871410686=46, srv144871772=26, srv365380694=67, srv1798954078=42, srv1169473706=11, srv413045583=71, srv471172571=74, srv997666813=99, srv421218815=72, srv975110211=96, srv817423182=90, srv519902912=79, srv491042920=75, srv1508774750=29, srv1880938254=47, srv373291152=68, srv2016212438=55, srv780898834=86, srv824912135=92, srv1447536367=25, srv34764458=65, srv470867022=73, srv1506022996=28, srv513349648=77, srv1533937060=31, srv677706278=85, srv505863123=76, srv1030188530=3, srv348200921=66, srv2078705847=57, srv621174026=81, srv642016406=83, srv51392575=78, srv839235862=93, srv1006518286=1, srv1110884203=7, srv1524483021=30, srv1698078250=37, srv1894941521=48, srv1242101469=16, srv1002646601=0, srv1322250421=20, srv2108230933=58, srv792001553=87, srv1170201750=12, srv1162712989=10, srv522400634=80, srv1862786447=45, srv1704984628=38, srv1203194854=14, srv1608283712=33, srv1639347313=34, srv1041625893=4, srv1751903448=39, srv1406373300=24, srv268202747=61, srv1497048544=27, srv1810766173=44, srv1999086241=54, srv1188076735=13, srv810833658=89, srv1348352588=21, srv12934870=18, srv1135069152=9, srv38034034=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-22 06:56:04,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-22 06:56:04,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-22 06:56:04,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-22 06:56:04,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-22 06:56:04,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-22 06:56:04,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-22 06:56:04,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-22 06:56:04,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-22 06:56:04,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-22 06:56:04,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-22 06:56:04,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-22 06:56:04,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-22 06:56:04,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-22 06:56:04,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-22 06:56:04,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-22 06:56:04,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-22 06:56:04,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-22 06:56:04,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-22 06:56:04,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-22 06:56:04,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-22 06:56:04,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-22 06:56:04,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-22 06:56:04,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-22 06:56:04,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-22 06:56:04,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-22 06:56:04,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-22 06:56:04,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-22 06:56:04,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-22 06:56:04,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-22 06:56:04,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-22 06:56:04,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-22 06:56:04,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-22 06:56:04,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-22 06:56:04,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-22 06:56:04,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-22 06:56:04,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-22 06:56:04,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-22 06:56:04,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-22 06:56:04,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-22 06:56:04,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-22 06:56:04,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-22 06:56:04,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-22 06:56:04,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-22 06:56:04,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-22 06:56:04,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-22 06:56:04,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-22 06:56:04,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-22 06:56:04,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-22 06:56:04,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-22 06:56:04,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-22 06:56:04,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-22 06:56:04,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-22 06:56:04,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-22 06:56:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-22 06:56:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-22 06:56:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-22 06:56:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-22 06:56:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-22 06:56:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-22 06:56:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-22 06:56:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-22 06:56:04,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-22 06:56:04,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-22 06:56:04,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-22 06:56:04,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-22 06:56:04,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-22 06:56:04,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-22 06:56:04,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-22 06:56:04,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-22 06:56:04,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-22 06:56:04,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-22 06:56:04,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-22 06:56:04,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-22 06:56:04,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-22 06:56:04,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-22 06:56:04,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-22 06:56:04,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-22 06:56:04,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-22 06:56:04,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-22 06:56:04,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-22 06:56:04,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-22 06:56:04,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-22 06:56:04,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-22 06:56:04,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-22 06:56:04,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-22 06:56:04,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-22 06:56:04,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-22 06:56:04,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-22 06:56:04,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-22 06:56:04,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-22 06:56:04,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-22 06:56:04,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-22 06:56:04,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-22 06:56:04,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-22 06:56:04,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-22 06:56:04,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-22 06:56:04,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-22 06:56:04,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-22 06:56:04,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-22 06:56:04,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-22 06:56:04,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-22 06:56:05,048 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 06:56:05,049 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-22 06:56:05,049 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-22 06:56:05,359 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 470 ms to try 5000 different iterations.  Found a solution that moves 4693 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17665583808653. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040718696209176244, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05393182710894043, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5214444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040718696209176244, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17116666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24533333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 06:56:05,374 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv817423182:81 , srv1079648870:82 , srv1773157938:82 , srv1162712989:83 , srv1289073926:83 , srv1506022996:83 , srv1894941521:83 , srv273198626:83 , srv621174026:83 , srv780898834:83 , srv1105078022:84 , srv12934870:84 , srv1447536367:84 , srv2108230933:84 , srv2130245351:84 , srv34764458:84 , srv383910674:84 , srv522400634:84 , srv642016406:84 , srv937564275:84 , srv990274269:84 , srv1006518286:85 , srv1030188530:85 , srv1110904728:85 , srv1169473706:85 , srv144871772:85 , srv1560520481:85 , srv1751903448:85 , srv1772817100:85 , srv1798954078:85 , srv1871410686:85 , srv1945693460:85 , srv1964755339:85 , srv1999086241:85 , srv268202747:85 , srv348200921:85 , srv470867022:85 , srv519902912:85 , srv822845862:85 , srv1170201750:86 , srv1188076735:86 , srv1348352588:86 , srv1370155698:86 , srv1406373300:86 , srv1508774750:86 , srv1524483021:86 , srv1639347313:86 , srv1698078250:86 , srv1933091132:86 , srv2078705847:86 , srv365380694:86 , srv38034034:86 , srv505863123:86 , srv629848040:86 , srv810833658:86 , srv94095876:86 , srv975110211:86 , srv100875542:87 , srv1110884203:87 , srv1135069152:87 , srv1390090356:87 , srv1608283712:87 , srv1704984628:87 , srv1810766173:87 , srv279041936:87 , srv301959504:87 , srv373291152:87 , srv471172571:87 , srv513349648:87 , srv51392575:87 , srv677706278:87 , srv991218668:87 , srv1002646601:88 , srv1041625893:88 , srv1497048544:88 , srv1654442476:88 , srv1862786447:88 , srv1913070771:88 , srv2016212438:88 , srv2069768582:88 , srv242733953:88 , srv80715395:88 , srv839235862:88 , srv1533937060:89 , srv1805856262:89 , srv1897414144:89 , srv413045583:89 , srv421218815:89 , srv491042920:89 , srv824912135:89 , srv1880938254:90 , srv656333699:90 , srv122938707:91 , srv1242101469:91 , srv1322250421:91 , srv1203194854:92 , srv131644742:92 , srv1679936725:92 , srv792001553:95 , srv997666813:452 }
2023-03-22 06:56:05,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-22 06:56:05,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1913070771=50, srv990274269=97, srv122938707=15, srv80715395=88, srv1289073926=17, srv822845862=91, srv1772817100=40, srv937564275=94, srv273198626=62, srv1110904728=8, srv94095876=95, srv100875542=2, srv383910674=70, srv1390090356=23, srv1560520481=32, srv1773157938=41, srv629848040=82, srv656333699=84, srv1370155698=22, srv242733953=60, srv1679936725=36, srv1105078022=6, srv2130245351=59, srv301959504=64, srv1654442476=35, srv1964755339=53, srv2069768582=56, srv991218668=98, srv1897414144=49, srv1805856262=43, srv1079648870=5, srv1945693460=52, srv1933091132=51, srv279041936=63, srv131644742=19, srv1871410686=46, srv144871772=26, srv365380694=67, srv1798954078=42, srv1169473706=11, srv413045583=71, srv471172571=74, srv997666813=99, srv421218815=72, srv975110211=96, srv817423182=90, srv519902912=79, srv491042920=75, srv1508774750=29, srv1880938254=47, srv373291152=68, srv2016212438=55, srv780898834=86, srv824912135=92, srv1447536367=25, srv34764458=65, srv470867022=73, srv1506022996=28, srv513349648=77, srv1533937060=31, srv677706278=85, srv505863123=76, srv1030188530=3, srv348200921=66, srv2078705847=57, srv621174026=81, srv642016406=83, srv51392575=78, srv839235862=93, srv1006518286=1, srv1110884203=7, srv1524483021=30, srv1698078250=37, srv1894941521=48, srv1242101469=16, srv1002646601=0, srv1322250421=20, srv2108230933=58, srv792001553=87, srv1170201750=12, srv1162712989=10, srv522400634=80, srv1862786447=45, srv1704984628=38, srv1203194854=14, srv1608283712=33, srv1639347313=34, srv1041625893=4, srv1751903448=39, srv1406373300=24, srv268202747=61, srv1497048544=27, srv1810766173=44, srv1999086241=54, srv1188076735=13, srv810833658=89, srv1348352588=21, srv12934870=18, srv1135069152=9, srv38034034=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-22 06:56:05,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-22 06:56:05,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-22 06:56:05,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-22 06:56:05,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-22 06:56:05,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-22 06:56:05,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-22 06:56:05,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-22 06:56:05,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-22 06:56:05,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-22 06:56:05,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-22 06:56:05,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-22 06:56:05,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-22 06:56:05,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-22 06:56:05,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-22 06:56:05,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-22 06:56:05,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-22 06:56:05,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-22 06:56:05,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-22 06:56:05,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-22 06:56:05,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-22 06:56:05,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-22 06:56:05,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-22 06:56:05,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-22 06:56:05,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-22 06:56:05,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-22 06:56:05,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-22 06:56:05,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-22 06:56:05,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-22 06:56:05,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-22 06:56:05,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-22 06:56:05,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-22 06:56:05,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-22 06:56:05,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-22 06:56:05,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-22 06:56:05,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-22 06:56:05,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-22 06:56:05,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-22 06:56:05,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-22 06:56:05,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-22 06:56:05,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-22 06:56:05,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-22 06:56:05,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-22 06:56:05,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-22 06:56:05,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-22 06:56:05,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-22 06:56:05,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-22 06:56:05,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-22 06:56:05,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-22 06:56:05,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-22 06:56:05,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-22 06:56:05,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-22 06:56:05,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-22 06:56:05,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-22 06:56:05,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-22 06:56:05,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-22 06:56:05,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-22 06:56:05,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-22 06:56:05,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-22 06:56:05,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-22 06:56:05,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-22 06:56:05,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-22 06:56:05,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-22 06:56:05,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-22 06:56:05,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-22 06:56:05,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-22 06:56:05,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-22 06:56:05,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-22 06:56:05,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-22 06:56:05,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-22 06:56:05,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-22 06:56:05,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-22 06:56:05,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-22 06:56:05,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-22 06:56:05,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-22 06:56:05,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-22 06:56:05,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-22 06:56:05,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-22 06:56:05,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-22 06:56:05,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-22 06:56:05,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-22 06:56:05,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-22 06:56:05,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-22 06:56:05,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-22 06:56:05,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-22 06:56:05,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-22 06:56:05,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-22 06:56:05,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-22 06:56:05,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-22 06:56:05,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-22 06:56:05,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-22 06:56:05,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-22 06:56:05,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-22 06:56:05,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-22 06:56:05,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-22 06:56:05,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-22 06:56:05,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-22 06:56:05,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-22 06:56:05,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-22 06:56:05,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-22 06:56:05,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-22 06:56:05,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-22 06:56:05,599 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040718696209176244, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05393182710894043, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040718696209176244, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17116666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24533333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 06:56:05,599 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-22 06:56:05,599 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17662297699817583, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040718696209176244, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05393182710894043, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040718696209176244, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17116666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24533333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-22 06:56:05,968 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 514 ms to try 5000 different iterations.  Found a solution that moves 2646 regions; Going from a computed imbalance of 0.17662297699817583 to a new imbalance of 0.013578355243172299. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024291943438606182); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03804421787905726, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.294, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024291943438606182); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010416666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04325, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 06:56:05,981 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv792001553:76 , srv122938707:78 , srv131644742:79 , srv1242101469:80 , srv1203194854:81 , srv1322250421:81 , srv1679936725:82 , srv1897414144:82 , srv242733953:82 , srv421218815:82 , srv656333699:82 , srv1110884203:83 , srv1497048544:83 , srv1654442476:83 , srv1805856262:83 , srv2016212438:83 , srv491042920:83 , srv513349648:83 , srv80715395:83 , srv1002646601:84 , srv1041625893:84 , srv1406373300:84 , srv1533937060:84 , srv1698078250:84 , srv1862786447:84 , srv2069768582:84 , srv373291152:84 , srv471172571:84 , srv677706278:84 , srv810833658:84 , srv824912135:84 , srv839235862:84 , srv1006518286:85 , srv1030188530:85 , srv1110904728:85 , srv1348352588:85 , srv1370155698:85 , srv1390090356:85 , srv144871772:85 , srv1964755339:85 , srv1999086241:85 , srv279041936:85 , srv365380694:85 , srv413045583:85 , srv505863123:85 , srv94095876:85 , srv975110211:85 , srv1135069152:86 , srv1170201750:86 , srv1188076735:86 , srv1560520481:86 , srv1608283712:86 , srv1639347313:86 , srv1751903448:86 , srv1871410686:86 , srv1880938254:86 , srv2078705847:86 , srv2130245351:86 , srv38034034:86 , srv629848040:86 , srv642016406:86 , srv990274269:86 , srv991218668:86 , srv100875542:87 , srv1169473706:87 , srv1508774750:87 , srv1704984628:87 , srv1772817100:87 , srv1810766173:87 , srv1913070771:87 , srv1945693460:87 , srv2108230933:87 , srv268202747:87 , srv301959504:87 , srv519902912:87 , srv780898834:87 , srv822845862:87 , srv937564275:87 , srv1105078022:88 , srv1162712989:88 , srv1524483021:88 , srv1773157938:88 , srv1798954078:88 , srv273198626:88 , srv34764458:88 , srv348200921:88 , srv383910674:88 , srv470867022:88 , srv51392575:88 , srv1079648870:89 , srv1289073926:89 , srv12934870:89 , srv1447536367:89 , srv1894941521:89 , srv1933091132:89 , srv1506022996:90 , srv522400634:90 , srv621174026:90 , srv817423182:93 , srv997666813:538 }
2023-03-22 06:56:05,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-22 06:56:05,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1913070771=50, srv990274269=97, srv122938707=15, srv80715395=88, srv1289073926=17, srv822845862=91, srv1772817100=40, srv937564275=94, srv273198626=62, srv1110904728=8, srv94095876=95, srv100875542=2, srv383910674=70, srv1390090356=23, srv1560520481=32, srv1773157938=41, srv629848040=82, srv656333699=84, srv1370155698=22, srv242733953=60, srv1679936725=36, srv1105078022=6, srv2130245351=59, srv301959504=64, srv1654442476=35, srv1964755339=53, srv2069768582=56, srv991218668=98, srv1897414144=49, srv1805856262=43, srv1079648870=5, srv1945693460=52, srv1933091132=51, srv279041936=63, srv131644742=19, srv1871410686=46, srv144871772=26, srv365380694=67, srv1798954078=42, srv1169473706=11, srv413045583=71, srv471172571=74, srv997666813=99, srv421218815=72, srv975110211=96, srv817423182=90, srv519902912=79, srv491042920=75, srv1508774750=29, srv1880938254=47, srv373291152=68, srv2016212438=55, srv780898834=86, srv824912135=92, srv1447536367=25, srv34764458=65, srv470867022=73, srv1506022996=28, srv513349648=77, srv1533937060=31, srv677706278=85, srv505863123=76, srv1030188530=3, srv348200921=66, srv2078705847=57, srv621174026=81, srv642016406=83, srv51392575=78, srv839235862=93, srv1006518286=1, srv1110884203=7, srv1524483021=30, srv1698078250=37, srv1894941521=48, srv1242101469=16, srv1002646601=0, srv1322250421=20, srv2108230933=58, srv792001553=87, srv1170201750=12, srv1162712989=10, srv522400634=80, srv1862786447=45, srv1704984628=38, srv1203194854=14, srv1608283712=33, srv1639347313=34, srv1041625893=4, srv1751903448=39, srv1406373300=24, srv268202747=61, srv1497048544=27, srv1810766173=44, srv1999086241=54, srv1188076735=13, srv810833658=89, srv1348352588=21, srv12934870=18, srv1135069152=9, srv38034034=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-22 06:56:06,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-22 06:56:06,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-22 06:56:06,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-22 06:56:06,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-22 06:56:06,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-22 06:56:06,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-22 06:56:06,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-22 06:56:06,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-22 06:56:06,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-22 06:56:06,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-22 06:56:06,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-22 06:56:06,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-22 06:56:06,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-22 06:56:06,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-22 06:56:06,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-22 06:56:06,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-22 06:56:06,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-22 06:56:06,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-22 06:56:06,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-22 06:56:06,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-22 06:56:06,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-22 06:56:06,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-22 06:56:06,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-22 06:56:06,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-22 06:56:06,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-22 06:56:06,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-22 06:56:06,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-22 06:56:06,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-22 06:56:06,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-22 06:56:06,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-22 06:56:06,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-22 06:56:06,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-22 06:56:06,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-22 06:56:06,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-22 06:56:06,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-22 06:56:06,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-22 06:56:06,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-22 06:56:06,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-22 06:56:06,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-22 06:56:06,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-22 06:56:06,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-22 06:56:06,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-22 06:56:06,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-22 06:56:06,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-22 06:56:06,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-22 06:56:06,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-22 06:56:06,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-22 06:56:06,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-22 06:56:06,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-22 06:56:06,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-22 06:56:06,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-22 06:56:06,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-22 06:56:06,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-22 06:56:06,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-22 06:56:06,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-22 06:56:06,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-22 06:56:06,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-22 06:56:06,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-22 06:56:06,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-22 06:56:06,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-22 06:56:06,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-22 06:56:06,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-22 06:56:06,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-22 06:56:06,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-22 06:56:06,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-22 06:56:06,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-22 06:56:06,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-22 06:56:06,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-22 06:56:06,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-22 06:56:06,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-22 06:56:06,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-22 06:56:06,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-22 06:56:06,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-22 06:56:06,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-22 06:56:06,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-22 06:56:06,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-22 06:56:06,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-22 06:56:06,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-22 06:56:06,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-22 06:56:06,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-22 06:56:06,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-22 06:56:06,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-22 06:56:06,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-22 06:56:06,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-22 06:56:06,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-22 06:56:06,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-22 06:56:06,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-22 06:56:06,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-22 06:56:06,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-22 06:56:06,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-22 06:56:06,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-22 06:56:06,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-22 06:56:06,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-22 06:56:06,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-22 06:56:06,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-22 06:56:06,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-22 06:56:06,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-22 06:56:06,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-22 06:56:06,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-22 06:56:06,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-22 06:56:06,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-22 06:56:06,145 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024291943438606182); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03804421787905726, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024291943438606182); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010416666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04325, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 06:56:06,146 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-22 06:56:06,146 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013559827555172082, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024291943438606182); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03804421787905726, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024291943438606182); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010416666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04325, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-22 06:56:06,555 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 523 ms to try 5000 different iterations.  Found a solution that moves 978 regions; Going from a computed imbalance of 0.013559827555172082 to a new imbalance of 7.431060529971667E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006538231594101115); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00823340762517771); MoveCostFunction : (multiplier=7.0, imbalance=0.10866666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006538231594101115); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 06:56:06,558 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1880938254:81 , srv1913070771:82 , srv1524483021:83 , srv1810766173:83 , srv1933091132:83 , srv413045583:83 , srv51392575:83 , srv522400634:83 , srv100875542:84 , srv1203194854:84 , srv12934870:84 , srv1704984628:84 , srv1798954078:84 , srv301959504:84 , srv621174026:84 , srv817423182:84 , srv824912135:84 , srv1002646601:85 , srv1135069152:85 , srv1169473706:85 , srv1188076735:85 , srv1322250421:85 , srv1390090356:85 , srv1447536367:85 , srv1506022996:85 , srv1508774750:85 , srv1533937060:85 , srv1639347313:85 , srv1679936725:85 , srv1772817100:85 , srv1805856262:85 , srv1862786447:85 , srv1894941521:85 , srv1945693460:85 , srv268202747:85 , srv348200921:85 , srv470867022:85 , srv519902912:85 , srv629848040:85 , srv656333699:85 , srv991218668:85 , srv1030188530:86 , srv1041625893:86 , srv1079648870:86 , srv1105078022:86 , srv1170201750:86 , srv1242101469:86 , srv1289073926:86 , srv1348352588:86 , srv1370155698:86 , srv1497048544:86 , srv1560520481:86 , srv1608283712:86 , srv1751903448:86 , srv1871410686:86 , srv2016212438:86 , srv2069768582:86 , srv2078705847:86 , srv2108230933:86 , srv273198626:86 , srv279041936:86 , srv34764458:86 , srv365380694:86 , srv373291152:86 , srv38034034:86 , srv383910674:86 , srv421218815:86 , srv491042920:86 , srv505863123:86 , srv792001553:86 , srv80715395:86 , srv822845862:86 , srv839235862:86 , srv94095876:86 , srv975110211:86 , srv1006518286:87 , srv1110884203:87 , srv1110904728:87 , srv1162712989:87 , srv131644742:87 , srv1406373300:87 , srv144871772:87 , srv1654442476:87 , srv1698078250:87 , srv1897414144:87 , srv2130245351:87 , srv242733953:87 , srv471172571:87 , srv513349648:87 , srv642016406:87 , srv677706278:87 , srv780898834:87 , srv810833658:87 , srv937564275:87 , srv990274269:87 , srv1773157938:88 , srv1964755339:88 , srv1999086241:88 , srv122938707:89 , srv997666813:526 }
2023-03-22 06:56:06,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-22 06:56:06,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1913070771=50, srv990274269=97, srv122938707=15, srv80715395=88, srv1289073926=17, srv822845862=91, srv1772817100=40, srv937564275=94, srv273198626=62, srv1110904728=8, srv94095876=95, srv100875542=2, srv383910674=70, srv1390090356=23, srv1560520481=32, srv1773157938=41, srv629848040=82, srv656333699=84, srv1370155698=22, srv242733953=60, srv1679936725=36, srv1105078022=6, srv2130245351=59, srv301959504=64, srv1654442476=35, srv1964755339=53, srv2069768582=56, srv991218668=98, srv1897414144=49, srv1805856262=43, srv1079648870=5, srv1945693460=52, srv1933091132=51, srv279041936=63, srv131644742=19, srv1871410686=46, srv144871772=26, srv365380694=67, srv1798954078=42, srv1169473706=11, srv413045583=71, srv471172571=74, srv997666813=99, srv421218815=72, srv975110211=96, srv817423182=90, srv519902912=79, srv491042920=75, srv1508774750=29, srv1880938254=47, srv373291152=68, srv2016212438=55, srv780898834=86, srv824912135=92, srv1447536367=25, srv34764458=65, srv470867022=73, srv1506022996=28, srv513349648=77, srv1533937060=31, srv677706278=85, srv505863123=76, srv1030188530=3, srv348200921=66, srv2078705847=57, srv621174026=81, srv642016406=83, srv51392575=78, srv839235862=93, srv1006518286=1, srv1110884203=7, srv1524483021=30, srv1698078250=37, srv1894941521=48, srv1242101469=16, srv1002646601=0, srv1322250421=20, srv2108230933=58, srv792001553=87, srv1170201750=12, srv1162712989=10, srv522400634=80, srv1862786447=45, srv1704984628=38, srv1203194854=14, srv1608283712=33, srv1639347313=34, srv1041625893=4, srv1751903448=39, srv1406373300=24, srv268202747=61, srv1497048544=27, srv1810766173=44, srv1999086241=54, srv1188076735=13, srv810833658=89, srv1348352588=21, srv12934870=18, srv1135069152=9, srv38034034=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-22 06:56:06,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-22 06:56:06,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-22 06:56:06,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-22 06:56:06,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-22 06:56:06,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-22 06:56:06,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-22 06:56:06,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-22 06:56:06,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-22 06:56:06,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-22 06:56:06,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-22 06:56:06,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-22 06:56:06,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-22 06:56:06,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-22 06:56:06,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-22 06:56:06,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-22 06:56:06,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-22 06:56:06,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-22 06:56:06,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-22 06:56:06,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-22 06:56:06,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-22 06:56:06,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-22 06:56:06,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-22 06:56:06,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-22 06:56:06,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-22 06:56:06,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-22 06:56:06,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-22 06:56:06,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-22 06:56:06,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-22 06:56:06,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-22 06:56:06,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-22 06:56:06,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-22 06:56:06,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-22 06:56:06,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-22 06:56:06,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-22 06:56:06,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-22 06:56:06,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-22 06:56:06,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-22 06:56:06,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-22 06:56:06,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-22 06:56:06,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-22 06:56:06,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-22 06:56:06,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-22 06:56:06,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-22 06:56:06,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-22 06:56:06,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-22 06:56:06,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-22 06:56:06,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-22 06:56:06,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-22 06:56:06,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-22 06:56:06,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-22 06:56:06,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-22 06:56:06,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-22 06:56:06,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-22 06:56:06,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-22 06:56:06,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-22 06:56:06,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-22 06:56:06,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-22 06:56:06,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-22 06:56:06,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-22 06:56:06,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-22 06:56:06,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-22 06:56:06,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-22 06:56:06,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-22 06:56:06,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-22 06:56:06,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-22 06:56:06,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-22 06:56:06,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-22 06:56:06,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-22 06:56:06,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-22 06:56:06,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-22 06:56:06,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-22 06:56:06,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-22 06:56:06,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-22 06:56:06,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-22 06:56:06,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-22 06:56:06,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-22 06:56:06,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-22 06:56:06,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-22 06:56:06,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-22 06:56:06,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-22 06:56:06,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-22 06:56:06,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-22 06:56:06,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-22 06:56:06,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-22 06:56:06,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-22 06:56:06,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-22 06:56:06,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-22 06:56:06,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-22 06:56:06,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-22 06:56:06,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-22 06:56:06,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-22 06:56:06,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-22 06:56:06,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-22 06:56:06,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-22 06:56:06,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-22 06:56:06,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-22 06:56:06,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-22 06:56:06,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-22 06:56:06,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-22 06:56:06,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-22 06:56:06,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-22 06:56:06,814 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006538231594101115); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00823340762517771); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006538231594101115); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 06:56:06,815 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-22 06:56:06,815 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.362579506297399E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006538231594101115); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00823340762517771); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006538231594101115); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-22 06:56:07,168 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 511 ms to try 5000 different iterations.  Found a solution that moves 283 regions; Going from a computed imbalance of 7.362579506297399E-4 to a new imbalance of 3.444914574367899E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.03144444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 06:56:07,169 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1608283712:83 , srv1679936725:83 , srv2078705847:83 , srv100875542:84 , srv1041625893:84 , srv1105078022:84 , srv1110904728:84 , srv1135069152:84 , srv1162712989:84 , srv1170201750:84 , srv122938707:84 , srv1289073926:84 , srv131644742:84 , srv1390090356:84 , srv1447536367:84 , srv1497048544:84 , srv1506022996:84 , srv1533937060:84 , srv1654442476:84 , srv1704984628:84 , srv1773157938:84 , srv1897414144:84 , srv1933091132:84 , srv1964755339:84 , srv1999086241:84 , srv2069768582:84 , srv279041936:84 , srv301959504:84 , srv34764458:84 , srv348200921:84 , srv38034034:84 , srv383910674:84 , srv470867022:84 , srv471172571:84 , srv491042920:84 , srv51392575:84 , srv677706278:84 , srv822845862:84 , srv839235862:84 , srv937564275:84 , srv991218668:84 , srv1002646601:85 , srv1006518286:85 , srv1030188530:85 , srv1079648870:85 , srv1110884203:85 , srv1169473706:85 , srv1188076735:85 , srv1203194854:85 , srv12934870:85 , srv1322250421:85 , srv1348352588:85 , srv1370155698:85 , srv1406373300:85 , srv144871772:85 , srv1508774750:85 , srv1524483021:85 , srv1560520481:85 , srv1639347313:85 , srv1698078250:85 , srv1751903448:85 , srv1772817100:85 , srv1798954078:85 , srv1805856262:85 , srv1810766173:85 , srv1862786447:85 , srv1871410686:85 , srv1880938254:85 , srv1894941521:85 , srv1913070771:85 , srv1945693460:85 , srv2016212438:85 , srv2108230933:85 , srv2130245351:85 , srv242733953:85 , srv365380694:85 , srv373291152:85 , srv413045583:85 , srv421218815:85 , srv505863123:85 , srv513349648:85 , srv519902912:85 , srv522400634:85 , srv621174026:85 , srv629848040:85 , srv642016406:85 , srv656333699:85 , srv780898834:85 , srv792001553:85 , srv80715395:85 , srv810833658:85 , srv817423182:85 , srv824912135:85 , srv94095876:85 , srv975110211:85 , srv990274269:85 , srv1242101469:86 , srv268202747:86 , srv273198626:86 , srv997666813:626 }
2023-03-22 06:56:07,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-22 06:56:07,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1913070771=50, srv990274269=97, srv122938707=15, srv80715395=88, srv1289073926=17, srv822845862=91, srv1772817100=40, srv937564275=94, srv273198626=62, srv1110904728=8, srv94095876=95, srv100875542=2, srv383910674=70, srv1390090356=23, srv1560520481=32, srv1773157938=41, srv629848040=82, srv656333699=84, srv1370155698=22, srv242733953=60, srv1679936725=36, srv1105078022=6, srv2130245351=59, srv301959504=64, srv1654442476=35, srv1964755339=53, srv2069768582=56, srv991218668=98, srv1897414144=49, srv1805856262=43, srv1079648870=5, srv1945693460=52, srv1933091132=51, srv279041936=63, srv131644742=19, srv1871410686=46, srv144871772=26, srv365380694=67, srv1798954078=42, srv1169473706=11, srv413045583=71, srv471172571=74, srv997666813=99, srv421218815=72, srv975110211=96, srv817423182=90, srv519902912=79, srv491042920=75, srv1508774750=29, srv1880938254=47, srv373291152=68, srv2016212438=55, srv780898834=86, srv824912135=92, srv1447536367=25, srv34764458=65, srv470867022=73, srv1506022996=28, srv513349648=77, srv1533937060=31, srv677706278=85, srv505863123=76, srv1030188530=3, srv348200921=66, srv2078705847=57, srv621174026=81, srv642016406=83, srv51392575=78, srv839235862=93, srv1006518286=1, srv1110884203=7, srv1524483021=30, srv1698078250=37, srv1894941521=48, srv1242101469=16, srv1002646601=0, srv1322250421=20, srv2108230933=58, srv792001553=87, srv1170201750=12, srv1162712989=10, srv522400634=80, srv1862786447=45, srv1704984628=38, srv1203194854=14, srv1608283712=33, srv1639347313=34, srv1041625893=4, srv1751903448=39, srv1406373300=24, srv268202747=61, srv1497048544=27, srv1810766173=44, srv1999086241=54, srv1188076735=13, srv810833658=89, srv1348352588=21, srv12934870=18, srv1135069152=9, srv38034034=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-22 06:56:07,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-22 06:56:07,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-22 06:56:07,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-22 06:56:07,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-22 06:56:07,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-22 06:56:07,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-22 06:56:07,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-22 06:56:07,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-22 06:56:07,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-22 06:56:07,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-22 06:56:07,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-22 06:56:07,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-22 06:56:07,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-22 06:56:07,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-22 06:56:07,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-22 06:56:07,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-22 06:56:07,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-22 06:56:07,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-22 06:56:07,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-22 06:56:07,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-22 06:56:07,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-22 06:56:07,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-22 06:56:07,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-22 06:56:07,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-22 06:56:07,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-22 06:56:07,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-22 06:56:07,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-22 06:56:07,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-22 06:56:07,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-22 06:56:07,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-22 06:56:07,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-22 06:56:07,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-22 06:56:07,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-22 06:56:07,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-22 06:56:07,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-22 06:56:07,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-22 06:56:07,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-22 06:56:07,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-22 06:56:07,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-22 06:56:07,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-22 06:56:07,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-22 06:56:07,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-22 06:56:07,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-22 06:56:07,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-22 06:56:07,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-22 06:56:07,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-22 06:56:07,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-22 06:56:07,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-22 06:56:07,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-22 06:56:07,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-22 06:56:07,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-22 06:56:07,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-22 06:56:07,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-22 06:56:07,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-22 06:56:07,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-22 06:56:07,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-22 06:56:07,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-22 06:56:07,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-22 06:56:07,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-22 06:56:07,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-22 06:56:07,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-22 06:56:07,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-22 06:56:07,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-22 06:56:07,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-22 06:56:07,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-22 06:56:07,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-22 06:56:07,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-22 06:56:07,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-22 06:56:07,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-22 06:56:07,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-22 06:56:07,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-22 06:56:07,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-22 06:56:07,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-22 06:56:07,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-22 06:56:07,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-22 06:56:07,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-22 06:56:07,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-22 06:56:07,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-22 06:56:07,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-22 06:56:07,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-22 06:56:07,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-22 06:56:07,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-22 06:56:07,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-22 06:56:07,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-22 06:56:07,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-22 06:56:07,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-22 06:56:07,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-22 06:56:07,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-22 06:56:07,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-22 06:56:07,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-22 06:56:07,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-22 06:56:07,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-22 06:56:07,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-22 06:56:07,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-22 06:56:07,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-22 06:56:07,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-22 06:56:07,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-22 06:56:07,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-22 06:56:07,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-22 06:56:07,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-22 06:56:07,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-22 06:56:07,327 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 06:56:07,327 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-22 06:56:07,328 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.246753738991438E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-22 06:56:07,616 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 406 ms to try 5000 different iterations.  Found a solution that moves 27 regions; Going from a computed imbalance of 3.246753738991438E-5 to a new imbalance of 6.2211367805469E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0013400504203456161); MoveCostFunction : (multiplier=7.0, imbalance=0.003); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 06:56:07,617 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1242101469:83 , srv1508774750:83 , srv268202747:83 , srv273198626:83 , srv817423182:83 , srv1002646601:84 , srv1006518286:84 , srv100875542:84 , srv1041625893:84 , srv1079648870:84 , srv1105078022:84 , srv1110884203:84 , srv1135069152:84 , srv1162712989:84 , srv1169473706:84 , srv1170201750:84 , srv1188076735:84 , srv1203194854:84 , srv122938707:84 , srv1289073926:84 , srv12934870:84 , srv131644742:84 , srv1322250421:84 , srv1348352588:84 , srv1370155698:84 , srv1406373300:84 , srv1447536367:84 , srv144871772:84 , srv1506022996:84 , srv1524483021:84 , srv1533937060:84 , srv1560520481:84 , srv1608283712:84 , srv1639347313:84 , srv1654442476:84 , srv1679936725:84 , srv1698078250:84 , srv1704984628:84 , srv1751903448:84 , srv1772817100:84 , srv1773157938:84 , srv1798954078:84 , srv1805856262:84 , srv1810766173:84 , srv1862786447:84 , srv1871410686:84 , srv1880938254:84 , srv1894941521:84 , srv1897414144:84 , srv1913070771:84 , srv1933091132:84 , srv1945693460:84 , srv1964755339:84 , srv1999086241:84 , srv2016212438:84 , srv2069768582:84 , srv2108230933:84 , srv2130245351:84 , srv242733953:84 , srv279041936:84 , srv301959504:84 , srv34764458:84 , srv348200921:84 , srv365380694:84 , srv373291152:84 , srv38034034:84 , srv383910674:84 , srv413045583:84 , srv421218815:84 , srv470867022:84 , srv471172571:84 , srv491042920:84 , srv505863123:84 , srv513349648:84 , srv51392575:84 , srv519902912:84 , srv522400634:84 , srv621174026:84 , srv629848040:84 , srv642016406:84 , srv656333699:84 , srv677706278:84 , srv780898834:84 , srv792001553:84 , srv80715395:84 , srv810833658:84 , srv822845862:84 , srv824912135:84 , srv839235862:84 , srv937564275:84 , srv94095876:84 , srv975110211:84 , srv990274269:84 , srv991218668:84 , srv1030188530:85 , srv1110904728:85 , srv1390090356:85 , srv1497048544:85 , srv2078705847:85 , srv997666813:684 }
2023-03-22 06:56:07,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-22 06:56:07,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1913070771=50, srv990274269=97, srv122938707=15, srv80715395=88, srv1289073926=17, srv822845862=91, srv1772817100=40, srv937564275=94, srv273198626=62, srv1110904728=8, srv94095876=95, srv100875542=2, srv383910674=70, srv1390090356=23, srv1560520481=32, srv1773157938=41, srv629848040=82, srv656333699=84, srv1370155698=22, srv242733953=60, srv1679936725=36, srv1105078022=6, srv2130245351=59, srv301959504=64, srv1654442476=35, srv1964755339=53, srv2069768582=56, srv991218668=98, srv1897414144=49, srv1805856262=43, srv1079648870=5, srv1945693460=52, srv1933091132=51, srv279041936=63, srv131644742=19, srv1871410686=46, srv144871772=26, srv365380694=67, srv1798954078=42, srv1169473706=11, srv413045583=71, srv471172571=74, srv997666813=99, srv421218815=72, srv975110211=96, srv817423182=90, srv519902912=79, srv491042920=75, srv1508774750=29, srv1880938254=47, srv373291152=68, srv2016212438=55, srv780898834=86, srv824912135=92, srv1447536367=25, srv34764458=65, srv470867022=73, srv1506022996=28, srv513349648=77, srv1533937060=31, srv677706278=85, srv505863123=76, srv1030188530=3, srv348200921=66, srv2078705847=57, srv621174026=81, srv642016406=83, srv51392575=78, srv839235862=93, srv1006518286=1, srv1110884203=7, srv1524483021=30, srv1698078250=37, srv1894941521=48, srv1242101469=16, srv1002646601=0, srv1322250421=20, srv2108230933=58, srv792001553=87, srv1170201750=12, srv1162712989=10, srv522400634=80, srv1862786447=45, srv1704984628=38, srv1203194854=14, srv1608283712=33, srv1639347313=34, srv1041625893=4, srv1751903448=39, srv1406373300=24, srv268202747=61, srv1497048544=27, srv1810766173=44, srv1999086241=54, srv1188076735=13, srv810833658=89, srv1348352588=21, srv12934870=18, srv1135069152=9, srv38034034=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-22 06:56:07,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-22 06:56:07,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-22 06:56:07,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-22 06:56:07,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-22 06:56:07,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-22 06:56:07,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-22 06:56:07,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-22 06:56:07,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-22 06:56:07,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-22 06:56:07,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-22 06:56:07,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-22 06:56:07,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-22 06:56:07,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-22 06:56:07,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-22 06:56:07,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-22 06:56:07,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-22 06:56:07,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-22 06:56:07,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-22 06:56:07,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-22 06:56:07,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-22 06:56:07,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-22 06:56:07,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-22 06:56:07,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-22 06:56:07,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-22 06:56:07,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-22 06:56:07,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-22 06:56:07,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-22 06:56:07,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-22 06:56:07,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-22 06:56:07,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-22 06:56:07,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-22 06:56:07,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-22 06:56:07,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-22 06:56:07,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-22 06:56:07,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-22 06:56:07,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-22 06:56:07,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-22 06:56:07,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-22 06:56:07,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-22 06:56:07,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-22 06:56:07,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-22 06:56:07,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-22 06:56:07,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-22 06:56:07,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-22 06:56:07,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-22 06:56:07,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-22 06:56:07,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-22 06:56:07,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-22 06:56:07,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-22 06:56:07,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-22 06:56:07,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-22 06:56:07,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-22 06:56:07,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-22 06:56:07,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-22 06:56:07,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-22 06:56:07,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-22 06:56:07,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-22 06:56:07,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-22 06:56:07,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-22 06:56:07,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-22 06:56:07,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-22 06:56:07,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-22 06:56:07,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-22 06:56:07,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-22 06:56:07,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-22 06:56:07,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-22 06:56:07,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-22 06:56:07,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-22 06:56:07,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-22 06:56:07,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-22 06:56:07,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-22 06:56:07,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-22 06:56:07,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-22 06:56:07,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-22 06:56:07,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-22 06:56:07,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-22 06:56:07,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-22 06:56:07,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-22 06:56:07,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-22 06:56:07,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-22 06:56:07,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-22 06:56:07,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-22 06:56:07,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-22 06:56:07,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-22 06:56:07,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-22 06:56:07,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-22 06:56:07,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-22 06:56:07,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-22 06:56:07,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-22 06:56:07,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-22 06:56:07,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-22 06:56:07,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-22 06:56:07,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-22 06:56:07,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-22 06:56:07,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-22 06:56:07,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-22 06:56:07,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-22 06:56:07,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-22 06:56:07,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-22 06:56:07,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-22 06:56:07,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-22 06:56:07,780 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.032078739728369E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0013400504203456161); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 06:56:07,781 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1242101469:83 , srv1508774750:83 , srv268202747:83 , srv273198626:83 , srv817423182:83 , srv1002646601:84 , srv1006518286:84 , srv100875542:84 , srv1041625893:84 , srv1079648870:84 , srv1105078022:84 , srv1110884203:84 , srv1135069152:84 , srv1162712989:84 , srv1169473706:84 , srv1170201750:84 , srv1188076735:84 , srv1203194854:84 , srv122938707:84 , srv1289073926:84 , srv12934870:84 , srv131644742:84 , srv1322250421:84 , srv1348352588:84 , srv1370155698:84 , srv1406373300:84 , srv1447536367:84 , srv144871772:84 , srv1506022996:84 , srv1524483021:84 , srv1533937060:84 , srv1560520481:84 , srv1608283712:84 , srv1639347313:84 , srv1654442476:84 , srv1679936725:84 , srv1698078250:84 , srv1704984628:84 , srv1751903448:84 , srv1772817100:84 , srv1773157938:84 , srv1798954078:84 , srv1805856262:84 , srv1810766173:84 , srv1862786447:84 , srv1871410686:84 , srv1880938254:84 , srv1894941521:84 , srv1897414144:84 , srv1913070771:84 , srv1933091132:84 , srv1945693460:84 , srv1964755339:84 , srv1999086241:84 , srv2016212438:84 , srv2069768582:84 , srv2108230933:84 , srv2130245351:84 , srv242733953:84 , srv279041936:84 , srv301959504:84 , srv34764458:84 , srv348200921:84 , srv365380694:84 , srv373291152:84 , srv38034034:84 , srv383910674:84 , srv413045583:84 , srv421218815:84 , srv470867022:84 , srv471172571:84 , srv491042920:84 , srv505863123:84 , srv513349648:84 , srv51392575:84 , srv519902912:84 , srv522400634:84 , srv621174026:84 , srv629848040:84 , srv642016406:84 , srv656333699:84 , srv677706278:84 , srv780898834:84 , srv792001553:84 , srv80715395:84 , srv810833658:84 , srv822845862:84 , srv824912135:84 , srv839235862:84 , srv937564275:84 , srv94095876:84 , srv975110211:84 , srv990274269:84 , srv991218668:84 , srv1030188530:85 , srv1110904728:85 , srv1390090356:85 , srv1497048544:85 , srv2078705847:85 , srv997666813:684 }
2023-03-22 06:56:07,819 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=365 (was 365), ProcessCount=169 (was 169), AvailableMemoryMB=7435 (was 7936)
2023-03-22 06:56:07,833 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=365, ProcessCount=169, AvailableMemoryMB=7435
2023-03-22 06:56:07,834 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-22 06:56:07,834 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-03-22 06:56:07,856 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1181247300:84 , srv1185732070:84 , srv1252051797:84 , srv1284119567:84 , srv1299534752:84 , srv1303103165:84 , srv1383209183:84 , srv1404962234:84 , srv1598098442:84 , srv1627299220:84 , srv168998106:84 , srv1712366737:84 , srv1721551622:84 , srv188595865:84 , srv1965719679:84 , srv2036947969:84 , srv2070622371:84 , srv209000211:84 , srv2120213942:84 , srv293210010:84 , srv436689437:84 , srv486954778:84 , srv554711348:84 , srv579449009:84 , srv686479004:84 , srv687021427:84 , srv725878146:84 , srv933742070:84 , srv996994779:84 , srv1801021534:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-03-22 06:56:07,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-22 06:56:07,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2120213942=19, srv1721551622=12, srv188595865=14, srv1712366737=11, srv1965719679=15, srv1181247300=0, srv168998106=10, srv554711348=23, srv1801021534=13, srv725878146=27, srv1303103165=5, srv1627299220=9, srv579449009=24, srv486954778=22, srv686479004=25, srv1383209183=6, srv436689437=21, srv293210010=20, srv996994779=29, srv2036947969=16, srv1284119567=3, srv687021427=26, srv1252051797=2, srv1185732070=1, srv933742070=28, srv1404962234=7, srv1299534752=4, srv1598098442=8, srv2070622371=17, srv209000211=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-22 06:56:07,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-22 06:56:07,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-22 06:56:07,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-22 06:56:07,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-22 06:56:07,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-22 06:56:07,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-22 06:56:07,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-22 06:56:07,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-22 06:56:07,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-22 06:56:07,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-22 06:56:07,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-22 06:56:07,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-22 06:56:07,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-22 06:56:07,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-22 06:56:07,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-22 06:56:07,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-22 06:56:07,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-22 06:56:07,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-22 06:56:07,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-22 06:56:07,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-22 06:56:07,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-22 06:56:07,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-22 06:56:07,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-22 06:56:07,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-22 06:56:07,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-22 06:56:07,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-22 06:56:07,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-22 06:56:07,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-22 06:56:07,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-22 06:56:07,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-22 06:56:07,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-22 06:56:07,908 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.144215420340102, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 06:56:07,908 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912647461344705, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.144215420340102, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-03-22 06:58:07,882 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5987524 different iterations.  Found a solution that moves 1868 regions; Going from a computed imbalance of 0.9912647461344705 to a new imbalance of 4.360005188259462E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6918518518518518, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 06:58:07,888 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1181247300:90 , srv1185732070:90 , srv1252051797:90 , srv1284119567:90 , srv1299534752:90 , srv1303103165:90 , srv1383209183:90 , srv1404962234:90 , srv1598098442:90 , srv1627299220:90 , srv168998106:90 , srv1712366737:90 , srv1721551622:90 , srv1801021534:90 , srv188595865:90 , srv1965719679:90 , srv2036947969:90 , srv2070622371:90 , srv209000211:90 , srv2120213942:90 , srv293210010:90 , srv436689437:90 , srv486954778:90 , srv554711348:90 , srv579449009:90 , srv686479004:90 , srv687021427:90 , srv725878146:90 , srv933742070:90 , srv996994779:90 }
2023-03-22 06:58:07,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-22 06:58:07,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2120213942=19, srv1721551622=12, srv188595865=14, srv1712366737=11, srv1965719679=15, srv1181247300=0, srv168998106=10, srv554711348=23, srv1801021534=13, srv725878146=27, srv1303103165=5, srv1627299220=9, srv579449009=24, srv486954778=22, srv686479004=25, srv1383209183=6, srv436689437=21, srv293210010=20, srv996994779=29, srv2036947969=16, srv1284119567=3, srv687021427=26, srv1252051797=2, srv1185732070=1, srv933742070=28, srv1404962234=7, srv1299534752=4, srv1598098442=8, srv2070622371=17, srv209000211=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-22 06:58:07,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-22 06:58:07,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-22 06:58:07,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-22 06:58:07,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-22 06:58:07,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-22 06:58:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-22 06:58:07,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-22 06:58:07,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-22 06:58:07,909 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 06:58:07,909 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1181247300:90 , srv1185732070:90 , srv1252051797:90 , srv1284119567:90 , srv1299534752:90 , srv1303103165:90 , srv1383209183:90 , srv1404962234:90 , srv1598098442:90 , srv1627299220:90 , srv168998106:90 , srv1712366737:90 , srv1721551622:90 , srv1801021534:90 , srv188595865:90 , srv1965719679:90 , srv2036947969:90 , srv2070622371:90 , srv209000211:90 , srv2120213942:90 , srv293210010:90 , srv436689437:90 , srv486954778:90 , srv554711348:90 , srv579449009:90 , srv686479004:90 , srv687021427:90 , srv725878146:90 , srv933742070:90 , srv996994779:90 }
2023-03-22 06:58:07,921 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=249 (was 365), ProcessCount=168 (was 169), AvailableMemoryMB=7377 (was 7435)