2023-11-03 09:56:49,733 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-11-03 09:56:50,281 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-11-03 09:56:50,467 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-11-03 09:56:50,596 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-03 09:56:50,606 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-03 09:56:50,628 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=765, ProcessCount=186, AvailableMemoryMB=7361
2023-11-03 09:56:50,632 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-03 09:56:50,632 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-03 09:56:50,903 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1010917114:84 , srv1022137033:84 , srv1051613703:84 , srv1054483328:84 , srv1099791352:84 , srv1115176353:84 , srv1132096909:84 , srv114245500:84 , srv1148014324:84 , srv1165923633:84 , srv1166543348:84 , srv1205902199:84 , srv1227590134:84 , srv124208164:84 , srv1261401019:84 , srv1282823100:84 , srv1293340595:84 , srv1301481788:84 , srv1312887252:84 , srv1314223348:84 , srv1395390195:84 , srv1397108857:84 , srv1406679869:84 , srv1407138052:84 , srv1491152944:84 , srv1551568019:84 , srv1553515562:84 , srv15904147:84 , srv1603009736:84 , srv1654197616:84 , srv16654665:84 , srv1684282107:84 , srv168661739:84 , srv1691270591:84 , srv1710308537:84 , srv174047814:84 , srv1752356126:84 , srv175503533:84 , srv1784842992:84 , srv1822611529:84 , srv1846596357:84 , srv1854977025:84 , srv185935697:84 , srv1859551848:84 , srv1898560452:84 , srv1963804194:84 , srv1983333462:84 , srv2017832979:84 , srv2021645621:84 , srv2074690878:84 , srv2102494808:84 , srv2124800768:84 , srv2142367600:84 , srv220345692:84 , srv229520179:84 , srv235535067:84 , srv238954871:84 , srv2517284:84 , srv255256698:84 , srv265401429:84 , srv275010445:84 , srv321243244:84 , srv338455363:84 , srv340484756:84 , srv354439772:84 , srv364882751:84 , srv371002311:84 , srv388151141:84 , srv419702987:84 , srv423805138:84 , srv457511938:84 , srv479089747:84 , srv522258147:84 , srv580284623:84 , srv583227260:84 , srv595216734:84 , srv626712747:84 , srv664164181:84 , srv671134591:84 , srv671537546:84 , srv681415052:84 , srv684316196:84 , srv709534282:84 , srv722548940:84 , srv737990995:84 , srv75010036:84 , srv78423983:84 , srv793010824:84 , srv869079245:84 , srv873259108:84 , srv887443714:84 , srv900347898:84 , srv90690551:84 , srv911329873:84 , srv917293727:84 , srv94531045:84 , srv950939096:84 , srv955335672:84 , srv956152597:84 , srv2045561954:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-11-03 09:56:50,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-03 09:56:50,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1261401019=14, srv175503533=37, srv681415052=81, srv887443714=91, srv321243244=62, srv580284623=74, srv1148014324=8, srv1022137033=1, srv1822611529=39, srv671134591=79, srv1710308537=34, srv583227260=75, srv423805138=70, srv1752356126=36, srv1553515562=26, srv1406679869=22, srv255256698=59, srv1397108857=21, srv1859551848=43, srv722548940=84, srv78423983=87, srv1654197616=29, srv1684282107=31, srv1846596357=40, srv185935697=42, srv956152597=99, srv1227590134=12, srv595216734=76, srv671537546=80, srv1054483328=3, srv1010917114=0, srv1407138052=23, srv684316196=82, srv1099791352=4, srv1784842992=38, srv793010824=88, srv114245500=7, srv1205902199=11, srv174047814=35, srv2021645621=48, srv238954871=57, srv2045561954=49, srv94531045=96, srv2017832979=47, srv1051613703=2, srv2102494808=51, srv340484756=64, srv1314223348=19, srv419702987=69, srv1293340595=16, srv1301481788=17, srv229520179=55, srv75010036=86, srv168661739=32, srv388151141=68, srv338455363=63, srv1854977025=41, srv1132096909=6, srv479089747=72, srv737990995=85, srv235535067=56, srv1312887252=18, srv16654665=30, srv1691270591=33, srv911329873=94, srv1898560452=44, srv1395390195=20, srv371002311=67, srv1165923633=9, srv2074690878=50, srv955335672=98, srv522258147=73, srv15904147=27, srv1491152944=24, srv1115176353=5, srv950939096=97, srv265401429=60, srv1603009736=28, srv869079245=89, srv1282823100=15, srv220345692=54, srv2142367600=53, srv664164181=78, srv354439772=65, srv364882751=66, srv626712747=77, srv90690551=93, srv457511938=71, srv900347898=92, srv917293727=95, srv1166543348=10, srv2124800768=52, srv275010445=61, srv1983333462=46, srv873259108=90, srv124208164=13, srv1551568019=25, srv2517284=58, srv709534282=83, srv1963804194=45} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-03 09:56:51,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-03 09:56:51,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-03 09:56:51,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-03 09:56:51,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-03 09:56:51,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-03 09:56:51,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-03 09:56:51,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-03 09:56:51,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-03 09:56:51,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-03 09:56:51,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-03 09:56:51,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-03 09:56:51,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-03 09:56:51,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-03 09:56:51,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-03 09:56:51,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-03 09:56:51,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-03 09:56:51,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-03 09:56:51,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-03 09:56:51,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-03 09:56:51,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-03 09:56:51,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-03 09:56:51,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-03 09:56:51,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-03 09:56:51,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-03 09:56:51,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-03 09:56:51,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-03 09:56:51,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-03 09:56:51,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-03 09:56:51,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-03 09:56:51,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-03 09:56:51,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-03 09:56:51,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-03 09:56:51,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-03 09:56:51,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-03 09:56:51,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-03 09:56:51,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-03 09:56:51,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-03 09:56:51,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-03 09:56:51,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-03 09:56:51,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-03 09:56:51,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-03 09:56:51,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-03 09:56:51,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-03 09:56:51,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-03 09:56:51,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-03 09:56:51,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-03 09:56:51,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-03 09:56:51,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-03 09:56:51,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-03 09:56:51,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-03 09:56:51,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-03 09:56:51,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-03 09:56:51,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-03 09:56:51,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-03 09:56:51,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-03 09:56:51,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-03 09:56:51,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-03 09:56:51,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-03 09:56:51,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-03 09:56:51,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-03 09:56:51,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-03 09:56:51,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-03 09:56:51,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-03 09:56:51,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-03 09:56:51,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-03 09:56:51,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-03 09:56:51,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-03 09:56:51,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-03 09:56:51,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-03 09:56:51,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-03 09:56:51,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-03 09:56:51,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-03 09:56:51,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-03 09:56:51,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-03 09:56:51,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-03 09:56:51,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-03 09:56:51,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-03 09:56:51,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-03 09:56:51,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-03 09:56:51,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-03 09:56:51,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-03 09:56:51,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-03 09:56:51,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-03 09:56:51,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-03 09:56:51,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-03 09:56:51,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-03 09:56:51,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-03 09:56:51,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-03 09:56:51,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-03 09:56:51,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-03 09:56:51,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-03 09:56:51,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-03 09:56:51,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-03 09:56:51,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-03 09:56:51,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-03 09:56:51,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-03 09:56:51,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-03 09:56:51,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-03 09:56:51,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-03 09:56:51,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-03 09:56:51,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-03 09:56:51,760 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 09:56:51,761 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-03 09:56:51,763 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-03 09:56:52,408 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1202 ms to try 5000 different iterations.  Found a solution that moves 4659 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18359769931624956. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04000592299332623, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05024295629924647, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5176666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04000592299332623, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17875, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24683333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 09:56:52,457 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1603009736:80 , srv479089747:80 , srv1054483328:81 , srv1301481788:81 , srv1407138052:81 , srv388151141:81 , srv681415052:81 , srv709534282:81 , srv900347898:81 , srv583227260:82 , srv887443714:82 , srv1099791352:83 , srv2021645621:83 , srv2124800768:83 , srv275010445:83 , srv423805138:83 , srv75010036:83 , srv1010917114:84 , srv1227590134:84 , srv174047814:84 , srv220345692:84 , srv955335672:84 , srv1115176353:85 , srv1132096909:85 , srv1165923633:85 , srv124208164:85 , srv1261401019:85 , srv1293340595:85 , srv1312887252:85 , srv1397108857:85 , srv1551568019:85 , srv1846596357:85 , srv1898560452:85 , srv255256698:85 , srv364882751:85 , srv522258147:85 , srv671134591:85 , srv722548940:85 , srv869079245:85 , srv90690551:85 , srv917293727:85 , srv94531045:85 , srv1022137033:86 , srv1051613703:86 , srv1148014324:86 , srv1406679869:86 , srv1553515562:86 , srv1654197616:86 , srv168661739:86 , srv1710308537:86 , srv1854977025:86 , srv2102494808:86 , srv2142367600:86 , srv229520179:86 , srv671537546:86 , srv911329873:86 , srv1314223348:87 , srv1684282107:87 , srv1691270591:87 , srv1752356126:87 , srv1784842992:87 , srv185935697:87 , srv235535067:87 , srv338455363:87 , srv419702987:87 , srv457511938:87 , srv78423983:87 , srv793010824:87 , srv1205902199:88 , srv2074690878:88 , srv595216734:88 , srv626712747:88 , srv684316196:88 , srv956152597:88 , srv1166543348:89 , srv16654665:89 , srv321243244:89 , srv340484756:89 , srv371002311:89 , srv1282823100:90 , srv1395390195:90 , srv15904147:90 , srv1859551848:90 , srv1963804194:90 , srv238954871:90 , srv2517284:90 , srv354439772:90 , srv580284623:90 , srv114245500:91 , srv1491152944:91 , srv1983333462:91 , srv873259108:91 , srv2017832979:92 , srv265401429:92 , srv664164181:92 , srv737990995:92 , srv1822611529:93 , srv950939096:93 , srv175503533:97 , srv2045561954:445 }
2023-11-03 09:56:52,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-03 09:56:52,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1261401019=14, srv175503533=37, srv681415052=81, srv887443714=91, srv321243244=62, srv580284623=74, srv1148014324=8, srv1022137033=1, srv1822611529=39, srv671134591=79, srv1710308537=34, srv583227260=75, srv423805138=70, srv1752356126=36, srv1553515562=26, srv1406679869=22, srv255256698=59, srv1397108857=21, srv1859551848=43, srv722548940=84, srv78423983=87, srv1654197616=29, srv1684282107=31, srv1846596357=40, srv185935697=42, srv956152597=99, srv1227590134=12, srv595216734=76, srv671537546=80, srv1054483328=3, srv1010917114=0, srv1407138052=23, srv684316196=82, srv1099791352=4, srv1784842992=38, srv793010824=88, srv114245500=7, srv1205902199=11, srv174047814=35, srv2021645621=48, srv238954871=57, srv2045561954=49, srv94531045=96, srv2017832979=47, srv1051613703=2, srv2102494808=51, srv340484756=64, srv1314223348=19, srv419702987=69, srv1293340595=16, srv1301481788=17, srv229520179=55, srv75010036=86, srv168661739=32, srv388151141=68, srv338455363=63, srv1854977025=41, srv1132096909=6, srv479089747=72, srv737990995=85, srv235535067=56, srv1312887252=18, srv16654665=30, srv1691270591=33, srv911329873=94, srv1898560452=44, srv1395390195=20, srv371002311=67, srv1165923633=9, srv2074690878=50, srv955335672=98, srv522258147=73, srv15904147=27, srv1491152944=24, srv1115176353=5, srv950939096=97, srv265401429=60, srv1603009736=28, srv869079245=89, srv1282823100=15, srv220345692=54, srv2142367600=53, srv664164181=78, srv354439772=65, srv364882751=66, srv626712747=77, srv90690551=93, srv457511938=71, srv900347898=92, srv917293727=95, srv1166543348=10, srv2124800768=52, srv275010445=61, srv1983333462=46, srv873259108=90, srv124208164=13, srv1551568019=25, srv2517284=58, srv709534282=83, srv1963804194=45} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-03 09:56:52,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-03 09:56:52,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-03 09:56:52,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-03 09:56:52,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-03 09:56:52,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-03 09:56:52,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-03 09:56:52,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-03 09:56:52,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-03 09:56:52,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-03 09:56:52,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-03 09:56:52,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-03 09:56:52,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-03 09:56:52,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-03 09:56:52,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-03 09:56:52,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-03 09:56:52,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-03 09:56:52,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-03 09:56:52,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-03 09:56:52,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-03 09:56:52,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-03 09:56:52,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-03 09:56:52,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-03 09:56:52,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-03 09:56:52,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-03 09:56:52,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-03 09:56:52,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-03 09:56:52,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-03 09:56:52,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-03 09:56:52,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-03 09:56:52,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-03 09:56:52,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-03 09:56:52,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-03 09:56:52,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-03 09:56:52,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-03 09:56:52,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-03 09:56:52,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-03 09:56:52,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-03 09:56:52,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-03 09:56:52,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-03 09:56:52,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-03 09:56:52,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-03 09:56:52,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-03 09:56:52,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-03 09:56:52,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-03 09:56:52,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-03 09:56:52,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-03 09:56:52,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-03 09:56:52,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-03 09:56:52,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-03 09:56:52,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-03 09:56:52,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-03 09:56:52,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-03 09:56:52,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-03 09:56:52,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-03 09:56:52,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-03 09:56:52,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-03 09:56:52,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-03 09:56:52,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-03 09:56:52,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-03 09:56:52,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-03 09:56:52,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-03 09:56:52,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-03 09:56:52,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-03 09:56:52,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-03 09:56:52,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-03 09:56:52,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-03 09:56:52,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-03 09:56:52,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-03 09:56:52,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-03 09:56:52,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-03 09:56:52,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-03 09:56:52,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-03 09:56:52,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-03 09:56:52,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-03 09:56:52,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-03 09:56:52,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-03 09:56:52,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-03 09:56:52,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-03 09:56:52,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-03 09:56:52,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-03 09:56:52,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-03 09:56:52,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-03 09:56:52,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-03 09:56:52,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-03 09:56:52,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-03 09:56:52,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-03 09:56:52,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-03 09:56:52,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-03 09:56:52,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-03 09:56:52,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-03 09:56:52,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-03 09:56:52,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-03 09:56:52,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-03 09:56:52,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-03 09:56:52,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-03 09:56:52,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-03 09:56:52,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-03 09:56:52,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-03 09:56:52,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-03 09:56:52,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-03 09:56:52,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-03 09:56:52,891 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04000592299332623, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05024295629924647, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04000592299332623, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17875, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24683333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 09:56:52,891 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-03 09:56:52,892 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18356507630098387, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04000592299332623, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05024295629924647, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04000592299332623, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17875, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24683333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-03 09:56:53,285 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 631 ms to try 5000 different iterations.  Found a solution that moves 2681 regions; Going from a computed imbalance of 0.18356507630098387 to a new imbalance of 0.014637843631033298. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019479445810351964); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03184206254705057, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29788888888888887, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019479445810351964); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043916666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 09:56:53,309 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv175503533:75 , srv737990995:78 , srv2017832979:79 , srv265401429:79 , srv114245500:80 , srv1822611529:80 , srv1395390195:81 , srv354439772:81 , srv873259108:81 , srv950939096:81 , srv1205902199:82 , srv1282823100:82 , srv1491152944:82 , srv15904147:82 , srv340484756:82 , srv664164181:82 , srv956152597:82 , srv1684282107:83 , srv1859551848:83 , srv2517284:83 , srv321243244:83 , srv580284623:83 , srv595216734:83 , srv684316196:83 , srv16654665:84 , srv1963804194:84 , srv338455363:84 , srv1148014324:85 , srv1166543348:85 , srv1261401019:85 , srv1406679869:85 , srv1691270591:85 , srv1983333462:85 , srv2142367600:85 , srv235535067:85 , srv626712747:85 , srv793010824:85 , srv90690551:85 , srv1022137033:86 , srv1115176353:86 , srv1165923633:86 , srv1293340595:86 , srv1314223348:86 , srv1553515562:86 , srv1654197616:86 , srv1710308537:86 , srv1784842992:86 , srv1846596357:86 , srv1854977025:86 , srv1898560452:86 , srv2074690878:86 , srv2102494808:86 , srv238954871:86 , srv255256698:86 , srv364882751:86 , srv371002311:86 , srv419702987:86 , srv78423983:86 , srv94531045:86 , srv1051613703:87 , srv1132096909:87 , srv124208164:87 , srv1551568019:87 , srv1752356126:87 , srv185935697:87 , srv457511938:87 , srv671134591:87 , srv671537546:87 , srv869079245:87 , srv911329873:87 , srv917293727:87 , srv1010917114:88 , srv1397108857:88 , srv168661739:88 , srv174047814:88 , srv2021645621:88 , srv2124800768:88 , srv275010445:88 , srv75010036:88 , srv1099791352:89 , srv1227590134:89 , srv1312887252:89 , srv220345692:89 , srv423805138:89 , srv722548940:89 , srv955335672:89 , srv1301481788:90 , srv229520179:90 , srv479089747:90 , srv681415052:90 , srv1054483328:91 , srv1407138052:91 , srv388151141:91 , srv522258147:91 , srv583227260:91 , srv709534282:91 , srv1603009736:92 , srv887443714:92 , srv900347898:94 , srv2045561954:502 }
2023-11-03 09:56:53,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-03 09:56:53,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1261401019=14, srv175503533=37, srv681415052=81, srv887443714=91, srv321243244=62, srv580284623=74, srv1148014324=8, srv1022137033=1, srv1822611529=39, srv671134591=79, srv1710308537=34, srv583227260=75, srv423805138=70, srv1752356126=36, srv1553515562=26, srv1406679869=22, srv255256698=59, srv1397108857=21, srv1859551848=43, srv722548940=84, srv78423983=87, srv1654197616=29, srv1684282107=31, srv1846596357=40, srv185935697=42, srv956152597=99, srv1227590134=12, srv595216734=76, srv671537546=80, srv1054483328=3, srv1010917114=0, srv1407138052=23, srv684316196=82, srv1099791352=4, srv1784842992=38, srv793010824=88, srv114245500=7, srv1205902199=11, srv174047814=35, srv2021645621=48, srv238954871=57, srv2045561954=49, srv94531045=96, srv2017832979=47, srv1051613703=2, srv2102494808=51, srv340484756=64, srv1314223348=19, srv419702987=69, srv1293340595=16, srv1301481788=17, srv229520179=55, srv75010036=86, srv168661739=32, srv388151141=68, srv338455363=63, srv1854977025=41, srv1132096909=6, srv479089747=72, srv737990995=85, srv235535067=56, srv1312887252=18, srv16654665=30, srv1691270591=33, srv911329873=94, srv1898560452=44, srv1395390195=20, srv371002311=67, srv1165923633=9, srv2074690878=50, srv955335672=98, srv522258147=73, srv15904147=27, srv1491152944=24, srv1115176353=5, srv950939096=97, srv265401429=60, srv1603009736=28, srv869079245=89, srv1282823100=15, srv220345692=54, srv2142367600=53, srv664164181=78, srv354439772=65, srv364882751=66, srv626712747=77, srv90690551=93, srv457511938=71, srv900347898=92, srv917293727=95, srv1166543348=10, srv2124800768=52, srv275010445=61, srv1983333462=46, srv873259108=90, srv124208164=13, srv1551568019=25, srv2517284=58, srv709534282=83, srv1963804194=45} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-03 09:56:53,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-03 09:56:53,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-03 09:56:53,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-03 09:56:53,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-03 09:56:53,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-03 09:56:53,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-03 09:56:53,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-03 09:56:53,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-03 09:56:53,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-03 09:56:53,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-03 09:56:53,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-03 09:56:53,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-03 09:56:53,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-03 09:56:53,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-03 09:56:53,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-03 09:56:53,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-03 09:56:53,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-03 09:56:53,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-03 09:56:53,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-03 09:56:53,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-03 09:56:53,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-03 09:56:53,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-03 09:56:53,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-03 09:56:53,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-03 09:56:53,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-03 09:56:53,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-03 09:56:53,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-03 09:56:53,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-03 09:56:53,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-03 09:56:53,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-03 09:56:53,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-03 09:56:53,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-03 09:56:53,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-03 09:56:53,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-03 09:56:53,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-03 09:56:53,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-03 09:56:53,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-03 09:56:53,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-03 09:56:53,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-03 09:56:53,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-03 09:56:53,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-03 09:56:53,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-03 09:56:53,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-03 09:56:53,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-03 09:56:53,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-03 09:56:53,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-03 09:56:53,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-03 09:56:53,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-03 09:56:53,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-03 09:56:53,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-03 09:56:53,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-03 09:56:53,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-03 09:56:53,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-03 09:56:53,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-03 09:56:53,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-03 09:56:53,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-03 09:56:53,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-03 09:56:53,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-03 09:56:53,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-03 09:56:53,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-03 09:56:53,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-03 09:56:53,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-03 09:56:53,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-03 09:56:53,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-03 09:56:53,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-03 09:56:53,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-03 09:56:53,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-03 09:56:53,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-03 09:56:53,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-03 09:56:53,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-03 09:56:53,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-03 09:56:53,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-03 09:56:53,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-03 09:56:53,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-03 09:56:53,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-03 09:56:53,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-03 09:56:53,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-03 09:56:53,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-03 09:56:53,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-03 09:56:53,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-03 09:56:53,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-03 09:56:53,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-03 09:56:53,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-03 09:56:53,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-03 09:56:53,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-03 09:56:53,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-03 09:56:53,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-03 09:56:53,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-03 09:56:53,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-03 09:56:53,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-03 09:56:53,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-03 09:56:53,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-03 09:56:53,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-03 09:56:53,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-03 09:56:53,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-03 09:56:53,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-03 09:56:53,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-03 09:56:53,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-03 09:56:53,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-03 09:56:53,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-03 09:56:53,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-03 09:56:53,721 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019479445810351964); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03184206254705057, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019479445810351964); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043916666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 09:56:53,721 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-03 09:56:53,722 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014619070867794987, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019479445810351964); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03184206254705057, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019479445810351964); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043916666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-03 09:56:54,103 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 563 ms to try 5000 different iterations.  Found a solution that moves 1041 regions; Going from a computed imbalance of 0.014619070867794987 to a new imbalance of 6.61702967221255E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004766658948975933); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006946969237999757); MoveCostFunction : (multiplier=7.0, imbalance=0.11566666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004766658948975933); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 09:56:54,107 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1983333462:81 , srv238954871:81 , srv522258147:81 , srv229520179:82 , srv1312887252:83 , srv185935697:83 , srv371002311:83 , srv900347898:83 , srv1051613703:84 , srv1166543348:84 , srv1491152944:84 , srv16654665:84 , srv1752356126:84 , srv1963804194:84 , srv2074690878:84 , srv220345692:84 , srv419702987:84 , srv457511938:84 , srv664164181:84 , srv671537546:84 , srv722548940:84 , srv78423983:84 , srv887443714:84 , srv955335672:84 , srv1022137033:85 , srv1054483328:85 , srv1132096909:85 , srv1148014324:85 , srv1227590134:85 , srv124208164:85 , srv1314223348:85 , srv1397108857:85 , srv1407138052:85 , srv1553515562:85 , srv1603009736:85 , srv168661739:85 , srv1691270591:85 , srv1710308537:85 , srv175503533:85 , srv1784842992:85 , srv1822611529:85 , srv1859551848:85 , srv2017832979:85 , srv2102494808:85 , srv2517284:85 , srv423805138:85 , srv580284623:85 , srv583227260:85 , srv626712747:85 , srv671134591:85 , srv709534282:85 , srv869079245:85 , srv873259108:85 , srv917293727:85 , srv950939096:85 , srv1010917114:86 , srv1099791352:86 , srv1293340595:86 , srv1301481788:86 , srv1395390195:86 , srv1406679869:86 , srv15904147:86 , srv1654197616:86 , srv174047814:86 , srv1846596357:86 , srv2021645621:86 , srv235535067:86 , srv255256698:86 , srv275010445:86 , srv321243244:86 , srv364882751:86 , srv388151141:86 , srv681415052:86 , srv684316196:86 , srv793010824:86 , srv911329873:86 , srv94531045:86 , srv1115176353:87 , srv114245500:87 , srv1165923633:87 , srv1205902199:87 , srv1282823100:87 , srv1551568019:87 , srv1854977025:87 , srv1898560452:87 , srv2124800768:87 , srv2142367600:87 , srv265401429:87 , srv338455363:87 , srv340484756:87 , srv354439772:87 , srv595216734:87 , srv737990995:87 , srv75010036:87 , srv1261401019:88 , srv1684282107:88 , srv479089747:88 , srv90690551:88 , srv956152597:88 , srv2045561954:553 }
2023-11-03 09:56:54,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-03 09:56:54,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1261401019=14, srv175503533=37, srv681415052=81, srv887443714=91, srv321243244=62, srv580284623=74, srv1148014324=8, srv1022137033=1, srv1822611529=39, srv671134591=79, srv1710308537=34, srv583227260=75, srv423805138=70, srv1752356126=36, srv1553515562=26, srv1406679869=22, srv255256698=59, srv1397108857=21, srv1859551848=43, srv722548940=84, srv78423983=87, srv1654197616=29, srv1684282107=31, srv1846596357=40, srv185935697=42, srv956152597=99, srv1227590134=12, srv595216734=76, srv671537546=80, srv1054483328=3, srv1010917114=0, srv1407138052=23, srv684316196=82, srv1099791352=4, srv1784842992=38, srv793010824=88, srv114245500=7, srv1205902199=11, srv174047814=35, srv2021645621=48, srv238954871=57, srv2045561954=49, srv94531045=96, srv2017832979=47, srv1051613703=2, srv2102494808=51, srv340484756=64, srv1314223348=19, srv419702987=69, srv1293340595=16, srv1301481788=17, srv229520179=55, srv75010036=86, srv168661739=32, srv388151141=68, srv338455363=63, srv1854977025=41, srv1132096909=6, srv479089747=72, srv737990995=85, srv235535067=56, srv1312887252=18, srv16654665=30, srv1691270591=33, srv911329873=94, srv1898560452=44, srv1395390195=20, srv371002311=67, srv1165923633=9, srv2074690878=50, srv955335672=98, srv522258147=73, srv15904147=27, srv1491152944=24, srv1115176353=5, srv950939096=97, srv265401429=60, srv1603009736=28, srv869079245=89, srv1282823100=15, srv220345692=54, srv2142367600=53, srv664164181=78, srv354439772=65, srv364882751=66, srv626712747=77, srv90690551=93, srv457511938=71, srv900347898=92, srv917293727=95, srv1166543348=10, srv2124800768=52, srv275010445=61, srv1983333462=46, srv873259108=90, srv124208164=13, srv1551568019=25, srv2517284=58, srv709534282=83, srv1963804194=45} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-03 09:56:54,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-03 09:56:54,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-03 09:56:54,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-03 09:56:54,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-03 09:56:54,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-03 09:56:54,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-03 09:56:54,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-03 09:56:54,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-03 09:56:54,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-03 09:56:54,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-03 09:56:54,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-03 09:56:54,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-03 09:56:54,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-03 09:56:54,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-03 09:56:54,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-03 09:56:54,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-03 09:56:54,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-03 09:56:54,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-03 09:56:54,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-03 09:56:54,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-03 09:56:54,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-03 09:56:54,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-03 09:56:54,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-03 09:56:54,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-03 09:56:54,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-03 09:56:54,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-03 09:56:54,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-03 09:56:54,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-03 09:56:54,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-03 09:56:54,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-03 09:56:54,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-03 09:56:54,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-03 09:56:54,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-03 09:56:54,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-03 09:56:54,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-03 09:56:54,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-03 09:56:54,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-03 09:56:54,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-03 09:56:54,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-03 09:56:54,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-03 09:56:54,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-03 09:56:54,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-03 09:56:54,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-03 09:56:54,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-03 09:56:54,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-03 09:56:54,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-03 09:56:54,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-03 09:56:54,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-03 09:56:54,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-03 09:56:54,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-03 09:56:54,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-03 09:56:54,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-03 09:56:54,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-03 09:56:54,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-03 09:56:54,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-03 09:56:54,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-03 09:56:54,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-03 09:56:54,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-03 09:56:54,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-03 09:56:54,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-03 09:56:54,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-03 09:56:54,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-03 09:56:54,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-03 09:56:54,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-03 09:56:54,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-03 09:56:54,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-03 09:56:54,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-03 09:56:54,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-03 09:56:54,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-03 09:56:54,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-03 09:56:54,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-03 09:56:54,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-03 09:56:54,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-03 09:56:54,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-03 09:56:54,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-03 09:56:54,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-03 09:56:54,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-03 09:56:54,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-03 09:56:54,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-03 09:56:54,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-03 09:56:54,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-03 09:56:54,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-03 09:56:54,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-03 09:56:54,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-03 09:56:54,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-03 09:56:54,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-03 09:56:54,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-03 09:56:54,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-03 09:56:54,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-03 09:56:54,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-03 09:56:54,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-03 09:56:54,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-03 09:56:54,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-03 09:56:54,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-03 09:56:54,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-03 09:56:54,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-03 09:56:54,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-03 09:56:54,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-03 09:56:54,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-03 09:56:54,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-03 09:56:54,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-03 09:56:54,296 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004766658948975933); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006946969237999757); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004766658948975933); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 09:56:54,296 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-03 09:56:54,297 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.544137294252516E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004766658948975933); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006946969237999757); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004766658948975933); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-03 09:56:54,761 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 588 ms to try 5000 different iterations.  Found a solution that moves 278 regions; Going from a computed imbalance of 6.544137294252516E-4 to a new imbalance of 2.0524039274977776E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.03088888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 09:56:54,763 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1115176353:83 , srv1165923633:83 , srv1282823100:83 , srv1551568019:83 , srv168661739:83 , srv1854977025:83 , srv1963804194:83 , srv911329873:83 , srv950939096:83 , srv1010917114:84 , srv1099791352:84 , srv1166543348:84 , srv1227590134:84 , srv1261401019:84 , srv1314223348:84 , srv1397108857:84 , srv1603009736:84 , srv1654197616:84 , srv1684282107:84 , srv175503533:84 , srv1784842992:84 , srv1822611529:84 , srv1859551848:84 , srv1898560452:84 , srv2021645621:84 , srv2074690878:84 , srv2124800768:84 , srv2142367600:84 , srv229520179:84 , srv235535067:84 , srv2517284:84 , srv275010445:84 , srv321243244:84 , srv338455363:84 , srv340484756:84 , srv354439772:84 , srv364882751:84 , srv388151141:84 , srv457511938:84 , srv479089747:84 , srv580284623:84 , srv583227260:84 , srv595216734:84 , srv626712747:84 , srv664164181:84 , srv722548940:84 , srv75010036:84 , srv793010824:84 , srv869079245:84 , srv887443714:84 , srv90690551:84 , srv956152597:84 , srv1022137033:85 , srv1051613703:85 , srv1054483328:85 , srv1132096909:85 , srv114245500:85 , srv1205902199:85 , srv1293340595:85 , srv1301481788:85 , srv1312887252:85 , srv1395390195:85 , srv1406679869:85 , srv1407138052:85 , srv1491152944:85 , srv15904147:85 , srv16654665:85 , srv1691270591:85 , srv1710308537:85 , srv174047814:85 , srv1752356126:85 , srv1846596357:85 , srv185935697:85 , srv1983333462:85 , srv2102494808:85 , srv220345692:85 , srv238954871:85 , srv255256698:85 , srv265401429:85 , srv371002311:85 , srv419702987:85 , srv423805138:85 , srv522258147:85 , srv671134591:85 , srv671537546:85 , srv681415052:85 , srv684316196:85 , srv709534282:85 , srv737990995:85 , srv78423983:85 , srv873259108:85 , srv900347898:85 , srv917293727:85 , srv94531045:85 , srv955335672:85 , srv1148014324:86 , srv124208164:86 , srv1553515562:86 , srv2017832979:86 , srv2045561954:642 }
2023-11-03 09:56:54,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-03 09:56:54,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1261401019=14, srv175503533=37, srv681415052=81, srv887443714=91, srv321243244=62, srv580284623=74, srv1148014324=8, srv1022137033=1, srv1822611529=39, srv671134591=79, srv1710308537=34, srv583227260=75, srv423805138=70, srv1752356126=36, srv1553515562=26, srv1406679869=22, srv255256698=59, srv1397108857=21, srv1859551848=43, srv722548940=84, srv78423983=87, srv1654197616=29, srv1684282107=31, srv1846596357=40, srv185935697=42, srv956152597=99, srv1227590134=12, srv595216734=76, srv671537546=80, srv1054483328=3, srv1010917114=0, srv1407138052=23, srv684316196=82, srv1099791352=4, srv1784842992=38, srv793010824=88, srv114245500=7, srv1205902199=11, srv174047814=35, srv2021645621=48, srv238954871=57, srv2045561954=49, srv94531045=96, srv2017832979=47, srv1051613703=2, srv2102494808=51, srv340484756=64, srv1314223348=19, srv419702987=69, srv1293340595=16, srv1301481788=17, srv229520179=55, srv75010036=86, srv168661739=32, srv388151141=68, srv338455363=63, srv1854977025=41, srv1132096909=6, srv479089747=72, srv737990995=85, srv235535067=56, srv1312887252=18, srv16654665=30, srv1691270591=33, srv911329873=94, srv1898560452=44, srv1395390195=20, srv371002311=67, srv1165923633=9, srv2074690878=50, srv955335672=98, srv522258147=73, srv15904147=27, srv1491152944=24, srv1115176353=5, srv950939096=97, srv265401429=60, srv1603009736=28, srv869079245=89, srv1282823100=15, srv220345692=54, srv2142367600=53, srv664164181=78, srv354439772=65, srv364882751=66, srv626712747=77, srv90690551=93, srv457511938=71, srv900347898=92, srv917293727=95, srv1166543348=10, srv2124800768=52, srv275010445=61, srv1983333462=46, srv873259108=90, srv124208164=13, srv1551568019=25, srv2517284=58, srv709534282=83, srv1963804194=45} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-03 09:56:54,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-03 09:56:54,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-03 09:56:54,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-03 09:56:54,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-03 09:56:54,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-03 09:56:54,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-03 09:56:54,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-03 09:56:54,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-03 09:56:54,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-03 09:56:54,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-03 09:56:54,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-03 09:56:54,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-03 09:56:54,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-03 09:56:54,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-03 09:56:54,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-03 09:56:54,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-03 09:56:54,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-03 09:56:54,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-03 09:56:54,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-03 09:56:54,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-03 09:56:54,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-03 09:56:54,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-03 09:56:54,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-03 09:56:54,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-03 09:56:54,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-03 09:56:54,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-03 09:56:54,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-03 09:56:54,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-03 09:56:54,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-03 09:56:54,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-03 09:56:54,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-03 09:56:54,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-03 09:56:54,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-03 09:56:54,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-03 09:56:54,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-03 09:56:54,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-03 09:56:54,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-03 09:56:54,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-03 09:56:54,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-03 09:56:54,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-03 09:56:54,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-03 09:56:54,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-03 09:56:54,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-03 09:56:54,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-03 09:56:54,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-03 09:56:54,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-03 09:56:54,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-03 09:56:54,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-03 09:56:54,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-03 09:56:54,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-03 09:56:54,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-03 09:56:54,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-03 09:56:54,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-03 09:56:54,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-03 09:56:54,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-03 09:56:54,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-03 09:56:54,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-03 09:56:54,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-03 09:56:54,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-03 09:56:54,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-03 09:56:54,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-03 09:56:54,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-03 09:56:54,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-03 09:56:54,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-03 09:56:54,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-03 09:56:54,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-03 09:56:54,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-03 09:56:54,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-03 09:56:54,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-03 09:56:54,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-03 09:56:54,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-03 09:56:54,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-03 09:56:54,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-03 09:56:54,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-03 09:56:54,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-03 09:56:54,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-03 09:56:54,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-03 09:56:54,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-03 09:56:54,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-03 09:56:54,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-03 09:56:54,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-03 09:56:54,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-03 09:56:54,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-03 09:56:54,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-03 09:56:54,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-03 09:56:54,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-03 09:56:54,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-03 09:56:54,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-03 09:56:54,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-03 09:56:54,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-03 09:56:54,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-03 09:56:54,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-03 09:56:54,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-03 09:56:54,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-03 09:56:54,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-03 09:56:54,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-03 09:56:54,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-03 09:56:54,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-03 09:56:54,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-03 09:56:54,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-03 09:56:54,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-03 09:56:55,055 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.8577441669512895E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 09:56:55,056 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1115176353:83 , srv1165923633:83 , srv1282823100:83 , srv1551568019:83 , srv168661739:83 , srv1854977025:83 , srv1963804194:83 , srv911329873:83 , srv950939096:83 , srv1010917114:84 , srv1099791352:84 , srv1166543348:84 , srv1227590134:84 , srv1261401019:84 , srv1314223348:84 , srv1397108857:84 , srv1603009736:84 , srv1654197616:84 , srv1684282107:84 , srv175503533:84 , srv1784842992:84 , srv1822611529:84 , srv1859551848:84 , srv1898560452:84 , srv2021645621:84 , srv2074690878:84 , srv2124800768:84 , srv2142367600:84 , srv229520179:84 , srv235535067:84 , srv2517284:84 , srv275010445:84 , srv321243244:84 , srv338455363:84 , srv340484756:84 , srv354439772:84 , srv364882751:84 , srv388151141:84 , srv457511938:84 , srv479089747:84 , srv580284623:84 , srv583227260:84 , srv595216734:84 , srv626712747:84 , srv664164181:84 , srv722548940:84 , srv75010036:84 , srv793010824:84 , srv869079245:84 , srv887443714:84 , srv90690551:84 , srv956152597:84 , srv1022137033:85 , srv1051613703:85 , srv1054483328:85 , srv1132096909:85 , srv114245500:85 , srv1205902199:85 , srv1293340595:85 , srv1301481788:85 , srv1312887252:85 , srv1395390195:85 , srv1406679869:85 , srv1407138052:85 , srv1491152944:85 , srv15904147:85 , srv16654665:85 , srv1691270591:85 , srv1710308537:85 , srv174047814:85 , srv1752356126:85 , srv1846596357:85 , srv185935697:85 , srv1983333462:85 , srv2102494808:85 , srv220345692:85 , srv238954871:85 , srv255256698:85 , srv265401429:85 , srv371002311:85 , srv419702987:85 , srv423805138:85 , srv522258147:85 , srv671134591:85 , srv671537546:85 , srv681415052:85 , srv684316196:85 , srv709534282:85 , srv737990995:85 , srv78423983:85 , srv873259108:85 , srv900347898:85 , srv917293727:85 , srv94531045:85 , srv955335672:85 , srv1148014324:86 , srv124208164:86 , srv1553515562:86 , srv2017832979:86 , srv2045561954:642 }
2023-11-03 09:56:55,126 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=765 (was 765), ProcessCount=186 (was 186), AvailableMemoryMB=6457 (was 7361)
2023-11-03 09:56:55,150 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=765, ProcessCount=186, AvailableMemoryMB=6456
2023-11-03 09:56:55,151 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-03 09:56:55,180 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-11-03 09:56:55,208 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1078177457:84 , srv1079449351:84 , srv1118913196:84 , srv112923999:84 , srv123804146:84 , srv1302944091:84 , srv1321767643:84 , srv158036691:84 , srv1666304866:84 , srv1760123287:84 , srv1765620585:84 , srv1777958292:84 , srv189765042:84 , srv2051825233:84 , srv274978690:84 , srv316658693:84 , srv382485464:84 , srv394343799:84 , srv498310368:84 , srv517728040:84 , srv562309095:84 , srv619059375:84 , srv635433691:84 , srv658069950:84 , srv674796390:84 , srv701832188:84 , srv817856542:84 , srv889797331:84 , srv958545809:84 , srv226643916:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-11-03 09:56:55,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-03 09:56:55,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1666304866=8, srv274978690=15, srv562309095=21, srv958545809=29, srv1765620585=10, srv1760123287=9, srv394343799=18, srv674796390=25, srv1321767643=6, srv189765042=12, srv817856542=27, srv112923999=3, srv1078177457=0, srv1118913196=2, srv658069950=24, srv1777958292=11, srv701832188=26, srv226643916=14, srv498310368=19, srv635433691=23, srv889797331=28, srv316658693=16, srv123804146=4, srv2051825233=13, srv1079449351=1, srv382485464=17, srv1302944091=5, srv158036691=7, srv517728040=20, srv619059375=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-03 09:56:55,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-03 09:56:55,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-03 09:56:55,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-03 09:56:55,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-03 09:56:55,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-03 09:56:55,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-03 09:56:55,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-03 09:56:55,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-03 09:56:55,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-03 09:56:55,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-03 09:56:55,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-03 09:56:55,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-03 09:56:55,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-03 09:56:55,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-03 09:56:55,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-03 09:56:55,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-03 09:56:55,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-03 09:56:55,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-03 09:56:55,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-03 09:56:55,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-03 09:56:55,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-03 09:56:55,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-03 09:56:55,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-03 09:56:55,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-03 09:56:55,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-03 09:56:55,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-03 09:56:55,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-03 09:56:55,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-03 09:56:55,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-03 09:56:55,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-03 09:56:55,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-03 09:56:55,319 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0684639496219512, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 09:56:55,319 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912408770933986, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0684639496219512, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-11-03 09:58:55,280 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6449655 different iterations.  Found a solution that moves 1862 regions; Going from a computed imbalance of 0.9912408770933986 to a new imbalance of 4.346000888939571E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6896296296296296, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 09:58:55,289 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1078177457:90 , srv1079449351:90 , srv1118913196:90 , srv112923999:90 , srv123804146:90 , srv1302944091:90 , srv1321767643:90 , srv158036691:90 , srv1666304866:90 , srv1760123287:90 , srv1765620585:90 , srv1777958292:90 , srv189765042:90 , srv2051825233:90 , srv226643916:90 , srv274978690:90 , srv316658693:90 , srv382485464:90 , srv394343799:90 , srv498310368:90 , srv517728040:90 , srv562309095:90 , srv619059375:90 , srv635433691:90 , srv658069950:90 , srv674796390:90 , srv701832188:90 , srv817856542:90 , srv889797331:90 , srv958545809:90 }
2023-11-03 09:58:55,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-03 09:58:55,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1666304866=8, srv274978690=15, srv562309095=21, srv958545809=29, srv1765620585=10, srv1760123287=9, srv394343799=18, srv674796390=25, srv1321767643=6, srv189765042=12, srv817856542=27, srv112923999=3, srv1078177457=0, srv1118913196=2, srv658069950=24, srv1777958292=11, srv701832188=26, srv226643916=14, srv498310368=19, srv635433691=23, srv889797331=28, srv316658693=16, srv123804146=4, srv2051825233=13, srv1079449351=1, srv382485464=17, srv1302944091=5, srv158036691=7, srv517728040=20, srv619059375=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-03 09:58:55,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-03 09:58:55,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-03 09:58:55,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-03 09:58:55,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-03 09:58:55,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-03 09:58:55,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-03 09:58:55,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-03 09:58:55,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-03 09:58:55,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-03 09:58:55,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-03 09:58:55,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-03 09:58:55,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-03 09:58:55,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-03 09:58:55,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-03 09:58:55,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-03 09:58:55,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-03 09:58:55,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-03 09:58:55,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-03 09:58:55,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-03 09:58:55,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-03 09:58:55,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-03 09:58:55,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-03 09:58:55,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-03 09:58:55,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-03 09:58:55,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-03 09:58:55,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-03 09:58:55,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-03 09:58:55,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-03 09:58:55,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-03 09:58:55,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-03 09:58:55,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-03 09:58:55,314 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-03 09:58:55,315 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1078177457:90 , srv1079449351:90 , srv1118913196:90 , srv112923999:90 , srv123804146:90 , srv1302944091:90 , srv1321767643:90 , srv158036691:90 , srv1666304866:90 , srv1760123287:90 , srv1765620585:90 , srv1777958292:90 , srv189765042:90 , srv2051825233:90 , srv226643916:90 , srv274978690:90 , srv316658693:90 , srv382485464:90 , srv394343799:90 , srv498310368:90 , srv517728040:90 , srv562309095:90 , srv619059375:90 , srv635433691:90 , srv658069950:90 , srv674796390:90 , srv701832188:90 , srv817856542:90 , srv889797331:90 , srv958545809:90 }
2023-11-03 09:58:55,331 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=855 (was 765) - SystemLoadAverage LEAK? -, ProcessCount=183 (was 186), AvailableMemoryMB=6462 (was 6456) - AvailableMemoryMB LEAK? -