2023-05-21 09:53:49,068 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-05-21 09:53:49,396 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-05-21 09:53:49,503 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-05-21 09:53:49,564 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-21 09:53:49,573 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-05-21 09:53:49,589 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=385, ProcessCount=172, AvailableMemoryMB=5931
2023-05-21 09:53:49,592 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-21 09:53:49,592 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-05-21 09:53:49,737 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1020632191:84 , srv1043532542:84 , srv1057461677:84 , srv1067708032:84 , srv1110050061:84 , srv1150732408:84 , srv1168580950:84 , srv1172832826:84 , srv1225255868:84 , srv1227030741:84 , srv1236770666:84 , srv1248424345:84 , srv1249922459:84 , srv1259913752:84 , srv1280174846:84 , srv1296776046:84 , srv129740468:84 , srv1331184830:84 , srv1341571587:84 , srv137141713:84 , srv1382680384:84 , srv1394064645:84 , srv1409224498:84 , srv1435522531:84 , srv1470668401:84 , srv1504441239:84 , srv1527628102:84 , srv1575349163:84 , srv1593359900:84 , srv1606241962:84 , srv1606618762:84 , srv1685947084:84 , srv1694172188:84 , srv1707544748:84 , srv1730418073:84 , srv1733785397:84 , srv1752588221:84 , srv1759243985:84 , srv1818459769:84 , srv1861764666:84 , srv1866008452:84 , srv1866396234:84 , srv1876572500:84 , srv1879610028:84 , srv1881908509:84 , srv1910622478:84 , srv1929393436:84 , srv1931343701:84 , srv1931962101:84 , srv1960675524:84 , srv1963786981:84 , srv1990790275:84 , srv199370520:84 , srv2008170321:84 , srv201579326:84 , srv2029798643:84 , srv2037238108:84 , srv2044417097:84 , srv2089196946:84 , srv2109924143:84 , srv217030544:84 , srv235537938:84 , srv279850625:84 , srv279861754:84 , srv280492567:84 , srv280537757:84 , srv284323488:84 , srv342716057:84 , srv373212885:84 , srv378412289:84 , srv379390076:84 , srv412210646:84 , srv414723370:84 , srv415145836:84 , srv42481633:84 , srv504421373:84 , srv52810255:84 , srv585868281:84 , srv590763413:84 , srv625980047:84 , srv63575863:84 , srv671797338:84 , srv708993201:84 , srv721197596:84 , srv737480752:84 , srv759328949:84 , srv803738697:84 , srv8119570:84 , srv82357663:84 , srv837689220:84 , srv858073043:84 , srv892847115:84 , srv894455251:84 , srv942390259:84 , srv944059869:84 , srv95534543:84 , srv956087012:84 , srv970157299:84 , srv9959201:84 , srv1245562233:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-05-21 09:53:49,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-21 09:53:49,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv373212885=69, srv1043532542=1, srv1818459769=39, srv1225255868=8, srv803738697=87, srv1931343701=48, srv1435522531=24, srv280492567=65, srv1866396234=42, srv942390259=94, srv1861764666=40, srv378412289=70, srv759328949=86, srv1172832826=7, srv1960675524=50, srv1168580950=6, srv504421373=76, srv42481633=75, srv1227030741=9, srv1470668401=25, srv1296776046=16, srv2044417097=58, srv837689220=90, srv2037238108=57, srv625980047=80, srv1382680384=21, srv1881908509=45, srv1150732408=5, srv1752588221=37, srv1394064645=22, srv95534543=96, srv9959201=99, srv1527628102=27, srv1694172188=33, srv1504441239=26, srv414723370=73, srv944059869=95, srv201579326=55, srv1759243985=38, srv1245562233=11, srv1910622478=46, srv970157299=98, srv1331184830=18, srv590763413=79, srv1990790275=52, srv2109924143=60, srv1409224498=23, srv1929393436=47, srv1730418073=35, srv8119570=88, srv737480752=85, srv199370520=53, srv1606618762=31, srv1280174846=15, srv52810255=77, srv2089196946=59, srv280537757=66, srv1236770666=10, srv137141713=20, srv415145836=74, srv1866008452=41, srv1057461677=2, srv1020632191=0, srv1963786981=51, srv1259913752=14, srv1341571587=19, srv342716057=68, srv708993201=83, srv1707544748=34, srv217030544=61, srv1931962101=49, srv1879610028=44, srv1733785397=36, srv2029798643=56, srv379390076=71, srv956087012=97, srv1575349163=28, srv1876572500=43, srv1110050061=4, srv1685947084=32, srv279850625=63, srv1067708032=3, srv284323488=67, srv412210646=72, srv721197596=84, srv858073043=91, srv1593359900=29, srv235537938=62, srv279861754=64, srv63575863=81, srv129740468=17, srv892847115=92, srv1248424345=12, srv1606241962=30, srv894455251=93, srv671797338=82, srv82357663=89, srv585868281=78, srv1249922459=13, srv2008170321=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-21 09:53:49,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-21 09:53:49,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-21 09:53:49,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-21 09:53:49,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-21 09:53:49,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-21 09:53:49,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-21 09:53:49,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-21 09:53:49,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-21 09:53:49,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-21 09:53:49,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-21 09:53:49,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-21 09:53:49,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-21 09:53:49,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-21 09:53:49,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-21 09:53:49,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-21 09:53:49,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-21 09:53:49,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-21 09:53:49,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-21 09:53:49,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-21 09:53:49,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-21 09:53:49,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-21 09:53:49,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-21 09:53:49,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-21 09:53:49,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-21 09:53:49,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-21 09:53:49,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-21 09:53:49,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-21 09:53:49,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-21 09:53:49,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-21 09:53:49,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-21 09:53:49,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-21 09:53:49,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-21 09:53:49,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-21 09:53:49,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-21 09:53:49,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-21 09:53:49,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-21 09:53:49,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-21 09:53:49,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-21 09:53:49,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-21 09:53:49,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-21 09:53:49,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-21 09:53:49,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-21 09:53:49,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-21 09:53:49,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-21 09:53:49,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-21 09:53:49,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-21 09:53:49,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-21 09:53:49,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-21 09:53:49,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-21 09:53:49,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-21 09:53:49,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-21 09:53:49,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-21 09:53:49,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-21 09:53:49,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-21 09:53:49,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-21 09:53:49,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-21 09:53:49,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-21 09:53:49,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-21 09:53:49,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-21 09:53:49,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-21 09:53:49,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-21 09:53:49,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-21 09:53:49,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-21 09:53:49,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-21 09:53:49,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-21 09:53:49,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-21 09:53:49,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-21 09:53:49,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-21 09:53:49,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-21 09:53:49,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-21 09:53:49,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-21 09:53:49,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-21 09:53:49,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-21 09:53:49,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-21 09:53:49,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-21 09:53:49,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-21 09:53:49,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-21 09:53:49,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-21 09:53:49,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-21 09:53:49,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-21 09:53:49,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-21 09:53:49,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-21 09:53:49,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-21 09:53:49,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-21 09:53:49,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-21 09:53:49,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-21 09:53:49,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-21 09:53:49,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-21 09:53:49,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-21 09:53:49,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-21 09:53:49,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-21 09:53:49,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-21 09:53:49,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-21 09:53:49,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-21 09:53:49,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-21 09:53:49,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-21 09:53:49,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-21 09:53:49,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-21 09:53:49,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-21 09:53:49,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-21 09:53:49,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-21 09:53:50,082 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 09:53:50,083 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-21 09:53:50,083 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-21 09:53:50,413 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 516 ms to try 5000 different iterations.  Found a solution that moves 4744 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1708784572511504. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03876089245474445, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05414574835659212, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5271111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03876089245474445, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.166, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23291666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 09:53:50,428 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1280174846:82 , srv1694172188:82 , srv1110050061:83 , srv1527628102:83 , srv1707544748:83 , srv2008170321:83 , srv2037238108:83 , srv235537938:83 , srv412210646:83 , srv504421373:83 , srv63575863:83 , srv721197596:83 , srv837689220:83 , srv892847115:83 , srv95534543:83 , srv1227030741:84 , srv1236770666:84 , srv1248424345:84 , srv1382680384:84 , srv1575349163:84 , srv1733785397:84 , srv1910622478:84 , srv342716057:84 , srv414723370:84 , srv42481633:84 , srv8119570:84 , srv956087012:84 , srv1043532542:85 , srv1341571587:85 , srv1606241962:85 , srv1752588221:85 , srv1759243985:85 , srv1861764666:85 , srv1931962101:85 , srv1963786981:85 , srv2089196946:85 , srv2109924143:85 , srv279861754:85 , srv415145836:85 , srv82357663:85 , srv858073043:85 , srv942390259:85 , srv1067708032:86 , srv1249922459:86 , srv1296776046:86 , srv137141713:86 , srv1435522531:86 , srv1504441239:86 , srv1593359900:86 , srv1931343701:86 , srv1960675524:86 , srv373212885:86 , srv585868281:86 , srv625980047:86 , srv708993201:86 , srv894455251:86 , srv9959201:86 , srv1020632191:87 , srv1168580950:87 , srv1259913752:87 , srv129740468:87 , srv1331184830:87 , srv1470668401:87 , srv1879610028:87 , srv1990790275:87 , srv199370520:87 , srv2029798643:87 , srv2044417097:87 , srv284323488:87 , srv671797338:87 , srv759328949:87 , srv1150732408:88 , srv1606618762:88 , srv217030544:88 , srv280492567:88 , srv378412289:88 , srv379390076:88 , srv52810255:88 , srv590763413:88 , srv1409224498:89 , srv1866396234:89 , srv279850625:89 , srv803738697:89 , srv970157299:89 , srv1730418073:90 , srv1818459769:90 , srv1876572500:90 , srv280537757:90 , srv1057461677:91 , srv1225255868:91 , srv1685947084:91 , srv1866008452:91 , srv1929393436:91 , srv1881908509:92 , srv944059869:92 , srv1394064645:94 , srv737480752:95 , srv201579326:97 , srv1172832826:98 , srv1245562233:434 }
2023-05-21 09:53:50,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-21 09:53:50,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv373212885=69, srv1043532542=1, srv1818459769=39, srv1225255868=8, srv803738697=87, srv1931343701=48, srv1435522531=24, srv280492567=65, srv1866396234=42, srv942390259=94, srv1861764666=40, srv378412289=70, srv759328949=86, srv1172832826=7, srv1960675524=50, srv1168580950=6, srv504421373=76, srv42481633=75, srv1227030741=9, srv1470668401=25, srv1296776046=16, srv2044417097=58, srv837689220=90, srv2037238108=57, srv625980047=80, srv1382680384=21, srv1881908509=45, srv1150732408=5, srv1752588221=37, srv1394064645=22, srv95534543=96, srv9959201=99, srv1527628102=27, srv1694172188=33, srv1504441239=26, srv414723370=73, srv944059869=95, srv201579326=55, srv1759243985=38, srv1245562233=11, srv1910622478=46, srv970157299=98, srv1331184830=18, srv590763413=79, srv1990790275=52, srv2109924143=60, srv1409224498=23, srv1929393436=47, srv1730418073=35, srv8119570=88, srv737480752=85, srv199370520=53, srv1606618762=31, srv1280174846=15, srv52810255=77, srv2089196946=59, srv280537757=66, srv1236770666=10, srv137141713=20, srv415145836=74, srv1866008452=41, srv1057461677=2, srv1020632191=0, srv1963786981=51, srv1259913752=14, srv1341571587=19, srv342716057=68, srv708993201=83, srv1707544748=34, srv217030544=61, srv1931962101=49, srv1879610028=44, srv1733785397=36, srv2029798643=56, srv379390076=71, srv956087012=97, srv1575349163=28, srv1876572500=43, srv1110050061=4, srv1685947084=32, srv279850625=63, srv1067708032=3, srv284323488=67, srv412210646=72, srv721197596=84, srv858073043=91, srv1593359900=29, srv235537938=62, srv279861754=64, srv63575863=81, srv129740468=17, srv892847115=92, srv1248424345=12, srv1606241962=30, srv894455251=93, srv671797338=82, srv82357663=89, srv585868281=78, srv1249922459=13, srv2008170321=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-21 09:53:50,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-21 09:53:50,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-21 09:53:50,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-21 09:53:50,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-21 09:53:50,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-21 09:53:50,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-21 09:53:50,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-21 09:53:50,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-21 09:53:50,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-21 09:53:50,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-21 09:53:50,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-21 09:53:50,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-21 09:53:50,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-21 09:53:50,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-21 09:53:50,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-21 09:53:50,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-21 09:53:50,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-21 09:53:50,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-21 09:53:50,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-21 09:53:50,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-21 09:53:50,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-21 09:53:50,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-21 09:53:50,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-21 09:53:50,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-21 09:53:50,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-21 09:53:50,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-21 09:53:50,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-21 09:53:50,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-21 09:53:50,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-21 09:53:50,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-21 09:53:50,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-21 09:53:50,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-21 09:53:50,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-21 09:53:50,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-21 09:53:50,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-21 09:53:50,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-21 09:53:50,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-21 09:53:50,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-21 09:53:50,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-21 09:53:50,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-21 09:53:50,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-21 09:53:50,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-21 09:53:50,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-21 09:53:50,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-21 09:53:50,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-21 09:53:50,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-21 09:53:50,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-21 09:53:50,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-21 09:53:50,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-21 09:53:50,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-21 09:53:50,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-21 09:53:50,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-21 09:53:50,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-21 09:53:50,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-21 09:53:50,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-21 09:53:50,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-21 09:53:50,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-21 09:53:50,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-21 09:53:50,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-21 09:53:50,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-21 09:53:50,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-21 09:53:50,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-21 09:53:50,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-21 09:53:50,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-21 09:53:50,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-21 09:53:50,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-21 09:53:50,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-21 09:53:50,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-21 09:53:50,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-21 09:53:50,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-21 09:53:50,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-21 09:53:50,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-21 09:53:50,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-21 09:53:50,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-21 09:53:50,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-21 09:53:50,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-21 09:53:50,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-21 09:53:50,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-21 09:53:50,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-21 09:53:50,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-21 09:53:50,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-21 09:53:50,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-21 09:53:50,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-21 09:53:50,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-21 09:53:50,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-21 09:53:50,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-21 09:53:50,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-21 09:53:50,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-21 09:53:50,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-21 09:53:50,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-21 09:53:50,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-21 09:53:50,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-21 09:53:50,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-21 09:53:50,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-21 09:53:50,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-21 09:53:50,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-21 09:53:50,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-21 09:53:50,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-21 09:53:50,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-21 09:53:50,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-21 09:53:50,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-21 09:53:50,624 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03876089245474445, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05414574835659212, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03876089245474445, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.166, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23291666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 09:53:50,624 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-21 09:53:50,624 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1708452390531636, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03876089245474445, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05414574835659212, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03876089245474445, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.166, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23291666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-21 09:53:50,945 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 465 ms to try 5000 different iterations.  Found a solution that moves 2571 regions; Going from a computed imbalance of 0.1708452390531636 to a new imbalance of 0.012903394276946986. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02080737252670961); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030277430263968078, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2856666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02080737252670961); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042166666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 09:53:50,955 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1172832826:73 , srv201579326:76 , srv1394064645:77 , srv737480752:78 , srv944059869:79 , srv1225255868:81 , srv1685947084:81 , srv1730418073:81 , srv1818459769:81 , srv1881908509:81 , srv1057461677:82 , srv1866008452:82 , srv1866396234:82 , srv1929393436:82 , srv217030544:82 , srv280537757:82 , srv803738697:82 , srv970157299:82 , srv1150732408:83 , srv1259913752:83 , srv1606618762:83 , srv1876572500:83 , srv279850625:83 , srv379390076:83 , srv52810255:83 , srv590763413:83 , srv1331184830:84 , srv1990790275:84 , srv280492567:84 , srv1296776046:85 , srv199370520:85 , srv2029798643:85 , srv2044417097:85 , srv378412289:85 , srv625980047:85 , srv671797338:85 , srv9959201:85 , srv1020632191:86 , srv1067708032:86 , srv1249922459:86 , srv129740468:86 , srv1341571587:86 , srv1409224498:86 , srv1470668401:86 , srv1504441239:86 , srv1593359900:86 , srv1606241962:86 , srv1752588221:86 , srv1861764666:86 , srv1931962101:86 , srv284323488:86 , srv415145836:86 , srv1168580950:87 , srv137141713:87 , srv1575349163:87 , srv1733785397:87 , srv1879610028:87 , srv1931343701:87 , srv1963786981:87 , srv2089196946:87 , srv373212885:87 , srv42481633:87 , srv585868281:87 , srv63575863:87 , srv708993201:87 , srv759328949:87 , srv858073043:87 , srv892847115:87 , srv894455251:87 , srv956087012:87 , srv1043532542:88 , srv1110050061:88 , srv1382680384:88 , srv1435522531:88 , srv1707544748:88 , srv1759243985:88 , srv1910622478:88 , srv1960675524:88 , srv2037238108:88 , srv342716057:88 , srv412210646:88 , srv8119570:88 , srv942390259:88 , srv1227030741:89 , srv1248424345:89 , srv1527628102:89 , srv2008170321:89 , srv2109924143:89 , srv279861754:89 , srv414723370:89 , srv504421373:89 , srv721197596:89 , srv82357663:89 , srv95534543:89 , srv1280174846:90 , srv837689220:90 , srv1236770666:91 , srv1694172188:91 , srv235537938:94 , srv1245562233:525 }
2023-05-21 09:53:50,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-21 09:53:50,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv373212885=69, srv1043532542=1, srv1818459769=39, srv1225255868=8, srv803738697=87, srv1931343701=48, srv1435522531=24, srv280492567=65, srv1866396234=42, srv942390259=94, srv1861764666=40, srv378412289=70, srv759328949=86, srv1172832826=7, srv1960675524=50, srv1168580950=6, srv504421373=76, srv42481633=75, srv1227030741=9, srv1470668401=25, srv1296776046=16, srv2044417097=58, srv837689220=90, srv2037238108=57, srv625980047=80, srv1382680384=21, srv1881908509=45, srv1150732408=5, srv1752588221=37, srv1394064645=22, srv95534543=96, srv9959201=99, srv1527628102=27, srv1694172188=33, srv1504441239=26, srv414723370=73, srv944059869=95, srv201579326=55, srv1759243985=38, srv1245562233=11, srv1910622478=46, srv970157299=98, srv1331184830=18, srv590763413=79, srv1990790275=52, srv2109924143=60, srv1409224498=23, srv1929393436=47, srv1730418073=35, srv8119570=88, srv737480752=85, srv199370520=53, srv1606618762=31, srv1280174846=15, srv52810255=77, srv2089196946=59, srv280537757=66, srv1236770666=10, srv137141713=20, srv415145836=74, srv1866008452=41, srv1057461677=2, srv1020632191=0, srv1963786981=51, srv1259913752=14, srv1341571587=19, srv342716057=68, srv708993201=83, srv1707544748=34, srv217030544=61, srv1931962101=49, srv1879610028=44, srv1733785397=36, srv2029798643=56, srv379390076=71, srv956087012=97, srv1575349163=28, srv1876572500=43, srv1110050061=4, srv1685947084=32, srv279850625=63, srv1067708032=3, srv284323488=67, srv412210646=72, srv721197596=84, srv858073043=91, srv1593359900=29, srv235537938=62, srv279861754=64, srv63575863=81, srv129740468=17, srv892847115=92, srv1248424345=12, srv1606241962=30, srv894455251=93, srv671797338=82, srv82357663=89, srv585868281=78, srv1249922459=13, srv2008170321=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-21 09:53:50,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-21 09:53:50,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-21 09:53:50,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-21 09:53:50,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-21 09:53:50,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-21 09:53:50,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-21 09:53:50,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-21 09:53:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-21 09:53:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-21 09:53:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-21 09:53:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-21 09:53:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-21 09:53:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-21 09:53:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-21 09:53:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-21 09:53:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-21 09:53:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-21 09:53:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-21 09:53:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-21 09:53:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-21 09:53:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-21 09:53:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-21 09:53:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-21 09:53:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-21 09:53:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-21 09:53:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-21 09:53:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-21 09:53:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-21 09:53:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-21 09:53:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-21 09:53:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-21 09:53:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-21 09:53:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-21 09:53:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-21 09:53:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-21 09:53:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-21 09:53:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-21 09:53:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-21 09:53:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-21 09:53:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-21 09:53:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-21 09:53:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-21 09:53:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-21 09:53:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-21 09:53:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-21 09:53:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-21 09:53:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-21 09:53:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-21 09:53:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-21 09:53:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-21 09:53:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-21 09:53:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-21 09:53:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-21 09:53:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-21 09:53:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-21 09:53:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-21 09:53:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-21 09:53:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-21 09:53:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-21 09:53:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-21 09:53:50,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-21 09:53:50,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-21 09:53:50,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-21 09:53:50,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-21 09:53:50,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-21 09:53:50,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-21 09:53:50,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-21 09:53:50,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-21 09:53:50,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-21 09:53:50,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-21 09:53:51,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-21 09:53:51,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-21 09:53:51,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-21 09:53:51,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-21 09:53:51,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-21 09:53:51,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-21 09:53:51,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-21 09:53:51,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-21 09:53:51,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-21 09:53:51,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-21 09:53:51,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-21 09:53:51,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-21 09:53:51,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-21 09:53:51,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-21 09:53:51,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-21 09:53:51,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-21 09:53:51,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-21 09:53:51,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-21 09:53:51,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-21 09:53:51,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-21 09:53:51,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-21 09:53:51,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-21 09:53:51,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-21 09:53:51,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-21 09:53:51,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-21 09:53:51,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-21 09:53:51,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-21 09:53:51,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-21 09:53:51,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-21 09:53:51,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-21 09:53:51,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-21 09:53:51,188 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02080737252670961); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030277430263968078, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02080737252670961); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042166666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 09:53:51,188 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-21 09:53:51,188 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.012885391750171265, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02080737252670961); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030277430263968078, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02080737252670961); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042166666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-21 09:53:51,482 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 446 ms to try 5000 different iterations.  Found a solution that moves 953 regions; Going from a computed imbalance of 0.012885391750171265 to a new imbalance of 3.799204857032207E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004658161240544447); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006267511942419625); MoveCostFunction : (multiplier=7.0, imbalance=0.1058888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004658161240544447); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00275); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 09:53:51,485 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv235537938:81 , srv1236770666:82 , srv1409224498:82 , srv1470668401:83 , srv1879610028:83 , srv2109924143:83 , srv1020632191:84 , srv1043532542:84 , srv1168580950:84 , srv1227030741:84 , srv1248424345:84 , srv129740468:84 , srv1435522531:84 , srv1694172188:84 , srv1866008452:84 , srv1929393436:84 , srv1931343701:84 , srv1960675524:84 , srv201579326:84 , srv279861754:84 , srv759328949:84 , srv82357663:84 , srv1067708032:85 , srv1280174846:85 , srv137141713:85 , srv1504441239:85 , srv1527628102:85 , srv1685947084:85 , srv1759243985:85 , srv1881908509:85 , srv279850625:85 , srv280492567:85 , srv284323488:85 , srv373212885:85 , srv378412289:85 , srv504421373:85 , srv585868281:85 , srv708993201:85 , srv737480752:85 , srv8119570:85 , srv837689220:85 , srv894455251:85 , srv942390259:85 , srv95534543:85 , srv1057461677:86 , srv1150732408:86 , srv1172832826:86 , srv1225255868:86 , srv1249922459:86 , srv1296776046:86 , srv1341571587:86 , srv1394064645:86 , srv1593359900:86 , srv1606241962:86 , srv1606618762:86 , srv1730418073:86 , srv1733785397:86 , srv1818459769:86 , srv1861764666:86 , srv1910622478:86 , srv1963786981:86 , srv1990790275:86 , srv199370520:86 , srv2029798643:86 , srv2044417097:86 , srv2089196946:86 , srv280537757:86 , srv342716057:86 , srv412210646:86 , srv414723370:86 , srv42481633:86 , srv52810255:86 , srv590763413:86 , srv625980047:86 , srv671797338:86 , srv721197596:86 , srv858073043:86 , srv944059869:86 , srv956087012:86 , srv9959201:86 , srv1110050061:87 , srv1331184830:87 , srv1382680384:87 , srv1575349163:87 , srv1707544748:87 , srv1752588221:87 , srv1866396234:87 , srv1876572500:87 , srv1931962101:87 , srv2008170321:87 , srv379390076:87 , srv415145836:87 , srv63575863:87 , srv803738697:87 , srv892847115:87 , srv970157299:87 , srv1259913752:88 , srv2037238108:88 , srv217030544:88 , srv1245562233:540 }
2023-05-21 09:53:51,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-21 09:53:51,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv373212885=69, srv1043532542=1, srv1818459769=39, srv1225255868=8, srv803738697=87, srv1931343701=48, srv1435522531=24, srv280492567=65, srv1866396234=42, srv942390259=94, srv1861764666=40, srv378412289=70, srv759328949=86, srv1172832826=7, srv1960675524=50, srv1168580950=6, srv504421373=76, srv42481633=75, srv1227030741=9, srv1470668401=25, srv1296776046=16, srv2044417097=58, srv837689220=90, srv2037238108=57, srv625980047=80, srv1382680384=21, srv1881908509=45, srv1150732408=5, srv1752588221=37, srv1394064645=22, srv95534543=96, srv9959201=99, srv1527628102=27, srv1694172188=33, srv1504441239=26, srv414723370=73, srv944059869=95, srv201579326=55, srv1759243985=38, srv1245562233=11, srv1910622478=46, srv970157299=98, srv1331184830=18, srv590763413=79, srv1990790275=52, srv2109924143=60, srv1409224498=23, srv1929393436=47, srv1730418073=35, srv8119570=88, srv737480752=85, srv199370520=53, srv1606618762=31, srv1280174846=15, srv52810255=77, srv2089196946=59, srv280537757=66, srv1236770666=10, srv137141713=20, srv415145836=74, srv1866008452=41, srv1057461677=2, srv1020632191=0, srv1963786981=51, srv1259913752=14, srv1341571587=19, srv342716057=68, srv708993201=83, srv1707544748=34, srv217030544=61, srv1931962101=49, srv1879610028=44, srv1733785397=36, srv2029798643=56, srv379390076=71, srv956087012=97, srv1575349163=28, srv1876572500=43, srv1110050061=4, srv1685947084=32, srv279850625=63, srv1067708032=3, srv284323488=67, srv412210646=72, srv721197596=84, srv858073043=91, srv1593359900=29, srv235537938=62, srv279861754=64, srv63575863=81, srv129740468=17, srv892847115=92, srv1248424345=12, srv1606241962=30, srv894455251=93, srv671797338=82, srv82357663=89, srv585868281=78, srv1249922459=13, srv2008170321=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-21 09:53:51,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-21 09:53:51,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-21 09:53:51,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-21 09:53:51,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-21 09:53:51,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-21 09:53:51,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-21 09:53:51,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-21 09:53:51,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-21 09:53:51,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-21 09:53:51,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-21 09:53:51,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-21 09:53:51,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-21 09:53:51,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-21 09:53:51,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-21 09:53:51,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-21 09:53:51,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-21 09:53:51,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-21 09:53:51,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-21 09:53:51,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-21 09:53:51,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-21 09:53:51,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-21 09:53:51,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-21 09:53:51,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-21 09:53:51,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-21 09:53:51,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-21 09:53:51,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-21 09:53:51,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-21 09:53:51,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-21 09:53:51,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-21 09:53:51,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-21 09:53:51,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-21 09:53:51,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-21 09:53:51,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-21 09:53:51,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-21 09:53:51,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-21 09:53:51,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-21 09:53:51,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-21 09:53:51,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-21 09:53:51,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-21 09:53:51,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-21 09:53:51,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-21 09:53:51,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-21 09:53:51,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-21 09:53:51,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-21 09:53:51,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-21 09:53:51,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-21 09:53:51,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-21 09:53:51,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-21 09:53:51,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-21 09:53:51,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-21 09:53:51,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-21 09:53:51,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-21 09:53:51,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-21 09:53:51,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-21 09:53:51,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-21 09:53:51,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-21 09:53:51,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-21 09:53:51,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-21 09:53:51,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-21 09:53:51,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-21 09:53:51,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-21 09:53:51,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-21 09:53:51,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-21 09:53:51,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-21 09:53:51,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-21 09:53:51,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-21 09:53:51,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-21 09:53:51,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-21 09:53:51,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-21 09:53:51,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-21 09:53:51,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-21 09:53:51,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-21 09:53:51,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-21 09:53:51,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-21 09:53:51,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-21 09:53:51,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-21 09:53:51,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-21 09:53:51,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-21 09:53:51,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-21 09:53:51,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-21 09:53:51,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-21 09:53:51,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-21 09:53:51,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-21 09:53:51,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-21 09:53:51,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-21 09:53:51,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-21 09:53:51,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-21 09:53:51,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-21 09:53:51,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-21 09:53:51,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-21 09:53:51,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-21 09:53:51,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-21 09:53:51,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-21 09:53:51,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-21 09:53:51,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-21 09:53:51,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-21 09:53:51,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-21 09:53:51,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-21 09:53:51,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-21 09:53:51,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-21 09:53:51,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-21 09:53:51,685 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004658161240544447); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006267511942419625); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004658161240544447); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00275); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 09:53:51,686 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-21 09:53:51,686 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.7324743707729254E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004658161240544447); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006267511942419625); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004658161240544447); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00275); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-21 09:53:52,096 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 548 ms to try 5000 different iterations.  Found a solution that moves 202 regions; Going from a computed imbalance of 3.7324743707729254E-4 to a new imbalance of 1.1983851156307916E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.022444444444444444); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 09:53:52,096 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1876572500:82 , srv1057461677:83 , srv1382680384:83 , srv2008170321:83 , srv2037238108:83 , srv414723370:83 , srv1110050061:84 , srv1168580950:84 , srv1225255868:84 , srv1249922459:84 , srv1259913752:84 , srv1331184830:84 , srv137141713:84 , srv1435522531:84 , srv1575349163:84 , srv1593359900:84 , srv1707544748:84 , srv1730418073:84 , srv1752588221:84 , srv1759243985:84 , srv1818459769:84 , srv1866396234:84 , srv1881908509:84 , srv1910622478:84 , srv1960675524:84 , srv1963786981:84 , srv199370520:84 , srv2029798643:84 , srv2089196946:84 , srv217030544:84 , srv235537938:84 , srv279861754:84 , srv280537757:84 , srv284323488:84 , srv342716057:84 , srv373212885:84 , srv378412289:84 , srv379390076:84 , srv415145836:84 , srv504421373:84 , srv585868281:84 , srv708993201:84 , srv721197596:84 , srv737480752:84 , srv759328949:84 , srv803738697:84 , srv82357663:84 , srv837689220:84 , srv858073043:84 , srv894455251:84 , srv970157299:84 , srv9959201:84 , srv1020632191:85 , srv1043532542:85 , srv1067708032:85 , srv1150732408:85 , srv1172832826:85 , srv1227030741:85 , srv1236770666:85 , srv1248424345:85 , srv1280174846:85 , srv1296776046:85 , srv129740468:85 , srv1341571587:85 , srv1394064645:85 , srv1409224498:85 , srv1504441239:85 , srv1527628102:85 , srv1606241962:85 , srv1606618762:85 , srv1685947084:85 , srv1694172188:85 , srv1733785397:85 , srv1861764666:85 , srv1866008452:85 , srv1879610028:85 , srv1929393436:85 , srv1931343701:85 , srv1931962101:85 , srv1990790275:85 , srv201579326:85 , srv2044417097:85 , srv2109924143:85 , srv279850625:85 , srv280492567:85 , srv412210646:85 , srv42481633:85 , srv52810255:85 , srv590763413:85 , srv625980047:85 , srv63575863:85 , srv671797338:85 , srv8119570:85 , srv892847115:85 , srv942390259:85 , srv944059869:85 , srv95534543:85 , srv956087012:85 , srv1470668401:86 , srv1245562233:643 }
2023-05-21 09:53:52,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-21 09:53:52,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv373212885=69, srv1043532542=1, srv1818459769=39, srv1225255868=8, srv803738697=87, srv1931343701=48, srv1435522531=24, srv280492567=65, srv1866396234=42, srv942390259=94, srv1861764666=40, srv378412289=70, srv759328949=86, srv1172832826=7, srv1960675524=50, srv1168580950=6, srv504421373=76, srv42481633=75, srv1227030741=9, srv1470668401=25, srv1296776046=16, srv2044417097=58, srv837689220=90, srv2037238108=57, srv625980047=80, srv1382680384=21, srv1881908509=45, srv1150732408=5, srv1752588221=37, srv1394064645=22, srv95534543=96, srv9959201=99, srv1527628102=27, srv1694172188=33, srv1504441239=26, srv414723370=73, srv944059869=95, srv201579326=55, srv1759243985=38, srv1245562233=11, srv1910622478=46, srv970157299=98, srv1331184830=18, srv590763413=79, srv1990790275=52, srv2109924143=60, srv1409224498=23, srv1929393436=47, srv1730418073=35, srv8119570=88, srv737480752=85, srv199370520=53, srv1606618762=31, srv1280174846=15, srv52810255=77, srv2089196946=59, srv280537757=66, srv1236770666=10, srv137141713=20, srv415145836=74, srv1866008452=41, srv1057461677=2, srv1020632191=0, srv1963786981=51, srv1259913752=14, srv1341571587=19, srv342716057=68, srv708993201=83, srv1707544748=34, srv217030544=61, srv1931962101=49, srv1879610028=44, srv1733785397=36, srv2029798643=56, srv379390076=71, srv956087012=97, srv1575349163=28, srv1876572500=43, srv1110050061=4, srv1685947084=32, srv279850625=63, srv1067708032=3, srv284323488=67, srv412210646=72, srv721197596=84, srv858073043=91, srv1593359900=29, srv235537938=62, srv279861754=64, srv63575863=81, srv129740468=17, srv892847115=92, srv1248424345=12, srv1606241962=30, srv894455251=93, srv671797338=82, srv82357663=89, srv585868281=78, srv1249922459=13, srv2008170321=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-21 09:53:52,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-21 09:53:52,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-21 09:53:52,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-21 09:53:52,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-21 09:53:52,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-21 09:53:52,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-21 09:53:52,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-21 09:53:52,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-21 09:53:52,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-21 09:53:52,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-21 09:53:52,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-21 09:53:52,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-21 09:53:52,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-21 09:53:52,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-21 09:53:52,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-21 09:53:52,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-21 09:53:52,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-21 09:53:52,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-21 09:53:52,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-21 09:53:52,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-21 09:53:52,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-21 09:53:52,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-21 09:53:52,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-21 09:53:52,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-21 09:53:52,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-21 09:53:52,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-21 09:53:52,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-21 09:53:52,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-21 09:53:52,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-21 09:53:52,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-21 09:53:52,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-21 09:53:52,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-21 09:53:52,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-21 09:53:52,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-21 09:53:52,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-21 09:53:52,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-21 09:53:52,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-21 09:53:52,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-21 09:53:52,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-21 09:53:52,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-21 09:53:52,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-21 09:53:52,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-21 09:53:52,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-21 09:53:52,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-21 09:53:52,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-21 09:53:52,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-21 09:53:52,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-21 09:53:52,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-21 09:53:52,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-21 09:53:52,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-21 09:53:52,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-21 09:53:52,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-21 09:53:52,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-21 09:53:52,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-21 09:53:52,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-21 09:53:52,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-21 09:53:52,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-21 09:53:52,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-21 09:53:52,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-21 09:53:52,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-21 09:53:52,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-21 09:53:52,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-21 09:53:52,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-21 09:53:52,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-21 09:53:52,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-21 09:53:52,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-21 09:53:52,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-21 09:53:52,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-21 09:53:52,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-21 09:53:52,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-21 09:53:52,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-21 09:53:52,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-21 09:53:52,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-21 09:53:52,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-21 09:53:52,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-21 09:53:52,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-21 09:53:52,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-21 09:53:52,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-21 09:53:52,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-21 09:53:52,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-21 09:53:52,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-21 09:53:52,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-21 09:53:52,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-21 09:53:52,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-21 09:53:52,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-21 09:53:52,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-21 09:53:52,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-21 09:53:52,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-21 09:53:52,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-21 09:53:52,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-21 09:53:52,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-21 09:53:52,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-21 09:53:52,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-21 09:53:52,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-21 09:53:52,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-21 09:53:52,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-21 09:53:52,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-21 09:53:52,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-21 09:53:52,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-21 09:53:52,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-21 09:53:52,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-21 09:53:52,223 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.0569416924998904E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 09:53:52,223 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1876572500:82 , srv1057461677:83 , srv1382680384:83 , srv2008170321:83 , srv2037238108:83 , srv414723370:83 , srv1110050061:84 , srv1168580950:84 , srv1225255868:84 , srv1249922459:84 , srv1259913752:84 , srv1331184830:84 , srv137141713:84 , srv1435522531:84 , srv1575349163:84 , srv1593359900:84 , srv1707544748:84 , srv1730418073:84 , srv1752588221:84 , srv1759243985:84 , srv1818459769:84 , srv1866396234:84 , srv1881908509:84 , srv1910622478:84 , srv1960675524:84 , srv1963786981:84 , srv199370520:84 , srv2029798643:84 , srv2089196946:84 , srv217030544:84 , srv235537938:84 , srv279861754:84 , srv280537757:84 , srv284323488:84 , srv342716057:84 , srv373212885:84 , srv378412289:84 , srv379390076:84 , srv415145836:84 , srv504421373:84 , srv585868281:84 , srv708993201:84 , srv721197596:84 , srv737480752:84 , srv759328949:84 , srv803738697:84 , srv82357663:84 , srv837689220:84 , srv858073043:84 , srv894455251:84 , srv970157299:84 , srv9959201:84 , srv1020632191:85 , srv1043532542:85 , srv1067708032:85 , srv1150732408:85 , srv1172832826:85 , srv1227030741:85 , srv1236770666:85 , srv1248424345:85 , srv1280174846:85 , srv1296776046:85 , srv129740468:85 , srv1341571587:85 , srv1394064645:85 , srv1409224498:85 , srv1504441239:85 , srv1527628102:85 , srv1606241962:85 , srv1606618762:85 , srv1685947084:85 , srv1694172188:85 , srv1733785397:85 , srv1861764666:85 , srv1866008452:85 , srv1879610028:85 , srv1929393436:85 , srv1931343701:85 , srv1931962101:85 , srv1990790275:85 , srv201579326:85 , srv2044417097:85 , srv2109924143:85 , srv279850625:85 , srv280492567:85 , srv412210646:85 , srv42481633:85 , srv52810255:85 , srv590763413:85 , srv625980047:85 , srv63575863:85 , srv671797338:85 , srv8119570:85 , srv892847115:85 , srv942390259:85 , srv944059869:85 , srv95534543:85 , srv956087012:85 , srv1470668401:86 , srv1245562233:643 }
2023-05-21 09:53:52,249 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=385 (was 385), ProcessCount=172 (was 172), AvailableMemoryMB=5301 (was 5931)
2023-05-21 09:53:52,258 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=385, ProcessCount=172, AvailableMemoryMB=5300
2023-05-21 09:53:52,259 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-21 09:53:52,259 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-05-21 09:53:52,267 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1062872677:84 , srv1116515470:84 , srv1368376289:84 , srv1422805635:84 , srv1765060241:84 , srv1784701813:84 , srv1784906647:84 , srv1848852995:84 , srv1871606446:84 , srv1872010112:84 , srv191833476:84 , srv1995845387:84 , srv2073331703:84 , srv2134318771:84 , srv298233712:84 , srv321556881:84 , srv33868097:84 , srv490716765:84 , srv504792746:84 , srv571735489:84 , srv588490059:84 , srv606224333:84 , srv617145463:84 , srv619143264:84 , srv681218696:84 , srv721235426:84 , srv860733063:84 , srv991390021:84 , srv995573466:84 , srv640722273:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-05-21 09:53:52,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-21 09:53:52,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1422805635=3, srv2134318771=13, srv1871606446=8, srv681218696=25, srv321556881=15, srv490716765=17, srv298233712=14, srv617145463=22, srv1848852995=7, srv995573466=29, srv619143264=23, srv640722273=24, srv1062872677=0, srv1784906647=6, srv1116515470=1, srv2073331703=12, srv606224333=21, srv1765060241=4, srv1368376289=2, srv571735489=19, srv1995845387=11, srv1872010112=9, srv991390021=28, srv191833476=10, srv504792746=18, srv860733063=27, srv33868097=16, srv588490059=20, srv1784701813=5, srv721235426=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-21 09:53:52,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-21 09:53:52,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-21 09:53:52,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-21 09:53:52,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-21 09:53:52,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-21 09:53:52,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-21 09:53:52,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-21 09:53:52,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-21 09:53:52,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-21 09:53:52,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-21 09:53:52,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-21 09:53:52,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-21 09:53:52,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-21 09:53:52,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-21 09:53:52,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-21 09:53:52,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-21 09:53:52,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-21 09:53:52,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-21 09:53:52,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-21 09:53:52,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-21 09:53:52,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-21 09:53:52,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-21 09:53:52,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-21 09:53:52,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-21 09:53:52,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-21 09:53:52,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-21 09:53:52,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-21 09:53:52,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-21 09:53:52,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-21 09:53:52,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-21 09:53:52,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-05-21 09:53:52,297 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1391516829464903, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 09:53:52,297 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912631505673523, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1391516829464903, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-05-21 09:55:52,281 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6780203 different iterations.  Found a solution that moves 1861 regions; Going from a computed imbalance of 0.9912631505673523 to a new imbalance of 4.3436668390529226E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6892592592592592, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 09:55:52,289 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1062872677:90 , srv1116515470:90 , srv1368376289:90 , srv1422805635:90 , srv1765060241:90 , srv1784701813:90 , srv1784906647:90 , srv1848852995:90 , srv1871606446:90 , srv1872010112:90 , srv191833476:90 , srv1995845387:90 , srv2073331703:90 , srv2134318771:90 , srv298233712:90 , srv321556881:90 , srv33868097:90 , srv490716765:90 , srv504792746:90 , srv571735489:90 , srv588490059:90 , srv606224333:90 , srv617145463:90 , srv619143264:90 , srv640722273:90 , srv681218696:90 , srv721235426:90 , srv860733063:90 , srv991390021:90 , srv995573466:90 }
2023-05-21 09:55:52,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-21 09:55:52,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1422805635=3, srv2134318771=13, srv1871606446=8, srv681218696=25, srv321556881=15, srv490716765=17, srv298233712=14, srv617145463=22, srv1848852995=7, srv995573466=29, srv619143264=23, srv640722273=24, srv1062872677=0, srv1784906647=6, srv1116515470=1, srv2073331703=12, srv606224333=21, srv1765060241=4, srv1368376289=2, srv571735489=19, srv1995845387=11, srv1872010112=9, srv991390021=28, srv191833476=10, srv504792746=18, srv860733063=27, srv33868097=16, srv588490059=20, srv1784701813=5, srv721235426=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-21 09:55:52,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-21 09:55:52,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-21 09:55:52,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-21 09:55:52,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-21 09:55:52,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-21 09:55:52,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-21 09:55:52,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-21 09:55:52,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-21 09:55:52,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-21 09:55:52,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-21 09:55:52,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-21 09:55:52,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-21 09:55:52,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-21 09:55:52,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-21 09:55:52,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-21 09:55:52,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-21 09:55:52,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-21 09:55:52,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-21 09:55:52,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-21 09:55:52,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-21 09:55:52,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-21 09:55:52,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-21 09:55:52,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-21 09:55:52,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-21 09:55:52,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-21 09:55:52,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-21 09:55:52,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-21 09:55:52,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-21 09:55:52,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-21 09:55:52,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-21 09:55:52,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-05-21 09:55:52,314 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 09:55:52,314 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1062872677:90 , srv1116515470:90 , srv1368376289:90 , srv1422805635:90 , srv1765060241:90 , srv1784701813:90 , srv1784906647:90 , srv1848852995:90 , srv1871606446:90 , srv1872010112:90 , srv191833476:90 , srv1995845387:90 , srv2073331703:90 , srv2134318771:90 , srv298233712:90 , srv321556881:90 , srv33868097:90 , srv490716765:90 , srv504792746:90 , srv571735489:90 , srv588490059:90 , srv606224333:90 , srv617145463:90 , srv619143264:90 , srv640722273:90 , srv681218696:90 , srv721235426:90 , srv860733063:90 , srv991390021:90 , srv995573466:90 }
2023-05-21 09:55:52,329 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=238 (was 385), ProcessCount=169 (was 172), AvailableMemoryMB=5268 (was 5300)