2023-03-21 11:59:05,371 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-03-21 11:59:05,707 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-03-21 11:59:05,806 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-03-21 11:59:05,857 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-21 11:59:05,863 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-21 11:59:05,873 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=367, ProcessCount=166, AvailableMemoryMB=10245
2023-03-21 11:59:05,875 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-21 11:59:05,875 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-21 11:59:06,005 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv101312519:84 , srv1016216226:84 , srv104552650:84 , srv1064298867:84 , srv1067634110:84 , srv1073630369:84 , srv1080663564:84 , srv1095167169:84 , srv1105249884:84 , srv1109529933:84 , srv1113959346:84 , srv1122800980:84 , srv1176357197:84 , srv1207059625:84 , srv1225887123:84 , srv12506430:84 , srv1263289454:84 , srv1280020407:84 , srv1289089602:84 , srv1295203304:84 , srv1341970042:84 , srv1349750519:84 , srv1350989743:84 , srv1355207830:84 , srv1355564725:84 , srv1369030154:84 , srv1435137656:84 , srv1444216803:84 , srv144917355:84 , srv1455237357:84 , srv1477643094:84 , srv1484882757:84 , srv1582925722:84 , srv1589320635:84 , srv1594566092:84 , srv1605064869:84 , srv1628149721:84 , srv1633047795:84 , srv1657292964:84 , srv1692284123:84 , srv1697634846:84 , srv1713209126:84 , srv1719189962:84 , srv1743056572:84 , srv1748500844:84 , srv180068182:84 , srv1812985712:84 , srv1839572166:84 , srv1873467913:84 , srv1889145682:84 , srv1889193518:84 , srv1917075767:84 , srv19445360:84 , srv1947544082:84 , srv195911693:84 , srv1999237412:84 , srv2031018575:84 , srv2058754668:84 , srv2064079453:84 , srv2102292564:84 , srv2115468328:84 , srv255019042:84 , srv271443089:84 , srv282239295:84 , srv287695384:84 , srv315492423:84 , srv323413399:84 , srv327315246:84 , srv370908116:84 , srv394644503:84 , srv416754856:84 , srv430875558:84 , srv432161140:84 , srv438693070:84 , srv468796431:84 , srv471906454:84 , srv478157603:84 , srv522933450:84 , srv549442543:84 , srv554768824:84 , srv558460815:84 , srv561541261:84 , srv56273553:84 , srv651326644:84 , srv669079844:84 , srv711522768:84 , srv730516044:84 , srv736710871:84 , srv8510699:84 , srv871006360:84 , srv872793267:84 , srv888171015:84 , srv894003238:84 , srv904763069:84 , srv932923947:84 , srv933423432:84 , srv974644567:84 , srv991881910:84 , srv99199308:84 , srv1436364100:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-03-21 11:59:06,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-21 11:59:06,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1692284123=40, srv894003238=93, srv1064298867=3, srv438693070=74, srv974644567=97, srv370908116=69, srv1349750519=21, srv144917355=29, srv1657292964=39, srv1889193518=51, srv282239295=64, srv872793267=91, srv888171015=92, srv1369030154=25, srv1889145682=50, srv2058754668=58, srv1350989743=22, srv1582925722=33, srv1444216803=28, srv1917075767=52, srv991881910=98, srv1839572166=48, srv471906454=76, srv12506430=15, srv1873467913=49, srv1263289454=16, srv871006360=90, srv195911693=55, srv549442543=79, srv1436364100=27, srv478157603=77, srv1080663564=6, srv1719189962=43, srv1812985712=47, srv1605064869=36, srv1176357197=12, srv1743056572=44, srv1355207830=23, srv2031018575=57, srv558460815=81, srv730516044=87, srv430875558=72, srv416754856=71, srv1109529933=9, srv1628149721=37, srv1947544082=54, srv1016216226=1, srv1477643094=31, srv8510699=89, srv1435137656=26, srv19445360=53, srv711522768=86, srv468796431=75, srv56273553=83, srv933423432=96, srv1105249884=8, srv2102292564=60, srv315492423=66, srv1122800980=11, srv104552650=2, srv323413399=67, srv432161140=73, srv1589320635=34, srv651326644=84, srv1280020407=17, srv1207059625=13, srv1594566092=35, srv1999237412=56, srv669079844=85, srv1713209126=42, srv1067634110=4, srv1697634846=41, srv327315246=68, srv1295203304=19, srv736710871=88, srv287695384=65, srv1633047795=38, srv904763069=94, srv1225887123=14, srv255019042=62, srv180068182=46, srv561541261=82, srv101312519=0, srv394644503=70, srv99199308=99, srv1455237357=30, srv1484882757=32, srv554768824=80, srv1289089602=18, srv271443089=63, srv1748500844=45, srv2064079453=59, srv932923947=95, srv1341970042=20, srv522933450=78, srv1095167169=7, srv1113959346=10, srv1073630369=5, srv2115468328=61, srv1355564725=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-21 11:59:06,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-21 11:59:06,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-21 11:59:06,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-21 11:59:06,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-21 11:59:06,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-21 11:59:06,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-21 11:59:06,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-21 11:59:06,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-21 11:59:06,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-21 11:59:06,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-21 11:59:06,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-21 11:59:06,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-21 11:59:06,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-21 11:59:06,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-21 11:59:06,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-21 11:59:06,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-21 11:59:06,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-21 11:59:06,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-21 11:59:06,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-21 11:59:06,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-21 11:59:06,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-21 11:59:06,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-21 11:59:06,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-21 11:59:06,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-21 11:59:06,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-21 11:59:06,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-21 11:59:06,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-21 11:59:06,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-21 11:59:06,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-21 11:59:06,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-21 11:59:06,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-21 11:59:06,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-21 11:59:06,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-21 11:59:06,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-21 11:59:06,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-21 11:59:06,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-21 11:59:06,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-21 11:59:06,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-21 11:59:06,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-21 11:59:06,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-21 11:59:06,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-21 11:59:06,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-21 11:59:06,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-21 11:59:06,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-21 11:59:06,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-21 11:59:06,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-21 11:59:06,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-21 11:59:06,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-21 11:59:06,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-21 11:59:06,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-21 11:59:06,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-21 11:59:06,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-21 11:59:06,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-21 11:59:06,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-21 11:59:06,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-21 11:59:06,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-21 11:59:06,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-21 11:59:06,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-21 11:59:06,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-21 11:59:06,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-21 11:59:06,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-21 11:59:06,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-21 11:59:06,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-21 11:59:06,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-21 11:59:06,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-21 11:59:06,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-21 11:59:06,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-21 11:59:06,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-21 11:59:06,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-21 11:59:06,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-21 11:59:06,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-21 11:59:06,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-21 11:59:06,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-21 11:59:06,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-21 11:59:06,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-21 11:59:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-21 11:59:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-21 11:59:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-21 11:59:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-21 11:59:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-21 11:59:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-21 11:59:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-21 11:59:06,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-21 11:59:06,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-21 11:59:06,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-21 11:59:06,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-21 11:59:06,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-21 11:59:06,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-21 11:59:06,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-21 11:59:06,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-21 11:59:06,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-21 11:59:06,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-21 11:59:06,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-21 11:59:06,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-21 11:59:06,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-21 11:59:06,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-21 11:59:06,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-21 11:59:06,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-21 11:59:06,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-21 11:59:06,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-21 11:59:06,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-21 11:59:06,370 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-21 11:59:06,370 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-21 11:59:06,371 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-21 11:59:06,632 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 422 ms to try 5000 different iterations.  Found a solution that moves 4682 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17870133187087422. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039068847688512556, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.050128902304361095, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5202222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039068847688512556, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17358333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24416666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-21 11:59:06,673 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1341970042:82 , srv1355564725:82 , srv1477643094:82 , srv522933450:82 , srv1064298867:83 , srv1605064869:83 , srv1633047795:83 , srv730516044:83 , srv872793267:83 , srv932923947:83 , srv1105249884:84 , srv1349750519:84 , srv1369030154:84 , srv1484882757:84 , srv1873467913:84 , srv1889193518:84 , srv2115468328:84 , srv271443089:84 , srv287695384:84 , srv416754856:84 , srv904763069:84 , srv933423432:84 , srv101312519:85 , srv1067634110:85 , srv1095167169:85 , srv1207059625:85 , srv1455237357:85 , srv1628149721:85 , srv1719189962:85 , srv1748500844:85 , srv2058754668:85 , srv315492423:85 , srv432161140:85 , srv471906454:85 , srv549442543:85 , srv558460815:85 , srv561541261:85 , srv56273553:85 , srv711522768:85 , srv8510699:85 , srv871006360:85 , srv974644567:85 , srv1016216226:86 , srv1080663564:86 , srv1109529933:86 , srv1176357197:86 , srv1280020407:86 , srv1589320635:86 , srv1697634846:86 , srv195911693:86 , srv2031018575:86 , srv327315246:86 , srv651326644:86 , srv888171015:86 , srv991881910:86 , srv1073630369:87 , srv1263289454:87 , srv1350989743:87 , srv1444216803:87 , srv1594566092:87 , srv1947544082:87 , srv2064079453:87 , srv2102292564:87 , srv255019042:87 , srv370908116:87 , srv394644503:87 , srv430875558:87 , srv554768824:87 , srv669079844:87 , srv104552650:88 , srv1113959346:88 , srv12506430:88 , srv1289089602:88 , srv1295203304:88 , srv144917355:88 , srv1692284123:88 , srv19445360:88 , srv438693070:88 , srv468796431:88 , srv736710871:88 , srv894003238:88 , srv1122800980:89 , srv1225887123:89 , srv1582925722:89 , srv1839572166:89 , srv1917075767:89 , srv1657292964:90 , srv1713209126:90 , srv1743056572:90 , srv180068182:90 , srv1812985712:90 , srv282239295:90 , srv478157603:90 , srv1889145682:91 , srv1999237412:91 , srv1435137656:92 , srv323413399:94 , srv1355207830:96 , srv99199308:98 , srv1436364100:437 }
2023-03-21 11:59:06,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-21 11:59:06,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1692284123=40, srv894003238=93, srv1064298867=3, srv438693070=74, srv974644567=97, srv370908116=69, srv1349750519=21, srv144917355=29, srv1657292964=39, srv1889193518=51, srv282239295=64, srv872793267=91, srv888171015=92, srv1369030154=25, srv1889145682=50, srv2058754668=58, srv1350989743=22, srv1582925722=33, srv1444216803=28, srv1917075767=52, srv991881910=98, srv1839572166=48, srv471906454=76, srv12506430=15, srv1873467913=49, srv1263289454=16, srv871006360=90, srv195911693=55, srv549442543=79, srv1436364100=27, srv478157603=77, srv1080663564=6, srv1719189962=43, srv1812985712=47, srv1605064869=36, srv1176357197=12, srv1743056572=44, srv1355207830=23, srv2031018575=57, srv558460815=81, srv730516044=87, srv430875558=72, srv416754856=71, srv1109529933=9, srv1628149721=37, srv1947544082=54, srv1016216226=1, srv1477643094=31, srv8510699=89, srv1435137656=26, srv19445360=53, srv711522768=86, srv468796431=75, srv56273553=83, srv933423432=96, srv1105249884=8, srv2102292564=60, srv315492423=66, srv1122800980=11, srv104552650=2, srv323413399=67, srv432161140=73, srv1589320635=34, srv651326644=84, srv1280020407=17, srv1207059625=13, srv1594566092=35, srv1999237412=56, srv669079844=85, srv1713209126=42, srv1067634110=4, srv1697634846=41, srv327315246=68, srv1295203304=19, srv736710871=88, srv287695384=65, srv1633047795=38, srv904763069=94, srv1225887123=14, srv255019042=62, srv180068182=46, srv561541261=82, srv101312519=0, srv394644503=70, srv99199308=99, srv1455237357=30, srv1484882757=32, srv554768824=80, srv1289089602=18, srv271443089=63, srv1748500844=45, srv2064079453=59, srv932923947=95, srv1341970042=20, srv522933450=78, srv1095167169=7, srv1113959346=10, srv1073630369=5, srv2115468328=61, srv1355564725=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-21 11:59:06,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-21 11:59:06,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-21 11:59:06,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-21 11:59:06,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-21 11:59:06,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-21 11:59:06,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-21 11:59:06,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-21 11:59:06,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-21 11:59:06,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-21 11:59:06,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-21 11:59:06,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-21 11:59:06,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-21 11:59:06,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-21 11:59:06,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-21 11:59:06,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-21 11:59:06,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-21 11:59:06,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-21 11:59:06,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-21 11:59:06,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-21 11:59:06,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-21 11:59:06,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-21 11:59:06,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-21 11:59:06,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-21 11:59:06,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-21 11:59:06,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-21 11:59:06,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-21 11:59:06,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-21 11:59:06,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-21 11:59:06,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-21 11:59:06,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-21 11:59:06,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-21 11:59:06,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-21 11:59:06,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-21 11:59:06,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-21 11:59:06,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-21 11:59:06,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-21 11:59:06,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-21 11:59:06,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-21 11:59:06,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-21 11:59:06,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-21 11:59:06,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-21 11:59:06,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-21 11:59:06,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-21 11:59:06,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-21 11:59:06,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-21 11:59:06,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-21 11:59:06,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-21 11:59:06,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-21 11:59:06,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-21 11:59:06,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-21 11:59:06,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-21 11:59:06,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-21 11:59:06,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-21 11:59:06,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-21 11:59:06,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-21 11:59:06,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-21 11:59:06,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-21 11:59:06,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-21 11:59:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-21 11:59:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-21 11:59:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-21 11:59:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-21 11:59:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-21 11:59:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-21 11:59:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-21 11:59:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-21 11:59:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-21 11:59:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-21 11:59:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-21 11:59:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-21 11:59:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-21 11:59:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-21 11:59:06,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-21 11:59:06,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-21 11:59:06,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-21 11:59:06,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-21 11:59:06,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-21 11:59:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-21 11:59:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-21 11:59:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-21 11:59:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-21 11:59:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-21 11:59:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-21 11:59:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-21 11:59:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-21 11:59:06,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-21 11:59:06,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-21 11:59:06,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-21 11:59:06,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-21 11:59:06,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-21 11:59:06,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-21 11:59:06,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-21 11:59:06,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-21 11:59:06,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-21 11:59:06,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-21 11:59:06,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-21 11:59:06,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-21 11:59:06,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-21 11:59:06,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-21 11:59:06,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-21 11:59:06,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-21 11:59:06,851 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039068847688512556, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.050128902304361095, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039068847688512556, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17358333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24416666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-21 11:59:06,851 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-21 11:59:06,851 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17866854780616634, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039068847688512556, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.050128902304361095, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039068847688512556, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17358333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24416666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-21 11:59:07,078 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 324 ms to try 5000 different iterations.  Found a solution that moves 2631 regions; Going from a computed imbalance of 0.17866854780616634 to a new imbalance of 0.014995523785408691. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021975623902201476); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03204234011499305, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29233333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021975623902201476); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04691666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-21 11:59:07,087 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv99199308:73 , srv1355207830:77 , srv1435137656:79 , srv1889145682:80 , srv104552650:81 , srv1225887123:81 , srv1582925722:81 , srv323413399:81 , srv144917355:82 , srv1713209126:82 , srv180068182:82 , srv1812985712:82 , srv1839572166:82 , srv1999237412:82 , srv1073630369:83 , srv12506430:83 , srv1350989743:83 , srv1692284123:83 , srv1917075767:83 , srv19445360:83 , srv1947544082:83 , srv2102292564:83 , srv394644503:83 , srv438693070:83 , srv991881910:83 , srv1289089602:84 , srv1743056572:84 , srv478157603:84 , srv554768824:84 , srv1067634110:85 , srv1109529933:85 , srv1113959346:85 , srv1122800980:85 , srv1295203304:85 , srv1589320635:85 , srv1657292964:85 , srv1697634846:85 , srv1719189962:85 , srv370908116:85 , srv468796431:85 , srv558460815:85 , srv736710871:85 , srv904763069:85 , srv974644567:85 , srv1176357197:86 , srv1207059625:86 , srv1263289454:86 , srv1444216803:86 , srv1628149721:86 , srv195911693:86 , srv2031018575:86 , srv2064079453:86 , srv282239295:86 , srv430875558:86 , srv561541261:86 , srv651326644:86 , srv888171015:86 , srv894003238:86 , srv932923947:86 , srv933423432:86 , srv101312519:87 , srv1016216226:87 , srv1064298867:87 , srv1105249884:87 , srv1349750519:87 , srv1594566092:87 , srv1748500844:87 , srv255019042:87 , srv416754856:87 , srv432161140:87 , srv522933450:87 , srv56273553:87 , srv730516044:87 , srv871006360:87 , srv1080663564:88 , srv1095167169:88 , srv1280020407:88 , srv1369030154:88 , srv1605064869:88 , srv1873467913:88 , srv2058754668:88 , srv287695384:88 , srv315492423:88 , srv327315246:88 , srv549442543:88 , srv1341970042:89 , srv1455237357:89 , srv1477643094:89 , srv1484882757:89 , srv1633047795:89 , srv271443089:89 , srv669079844:89 , srv711522768:89 , srv8510699:89 , srv1355564725:90 , srv1889193518:90 , srv2115468328:90 , srv872793267:91 , srv471906454:95 , srv1436364100:532 }
2023-03-21 11:59:07,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-21 11:59:07,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1692284123=40, srv894003238=93, srv1064298867=3, srv438693070=74, srv974644567=97, srv370908116=69, srv1349750519=21, srv144917355=29, srv1657292964=39, srv1889193518=51, srv282239295=64, srv872793267=91, srv888171015=92, srv1369030154=25, srv1889145682=50, srv2058754668=58, srv1350989743=22, srv1582925722=33, srv1444216803=28, srv1917075767=52, srv991881910=98, srv1839572166=48, srv471906454=76, srv12506430=15, srv1873467913=49, srv1263289454=16, srv871006360=90, srv195911693=55, srv549442543=79, srv1436364100=27, srv478157603=77, srv1080663564=6, srv1719189962=43, srv1812985712=47, srv1605064869=36, srv1176357197=12, srv1743056572=44, srv1355207830=23, srv2031018575=57, srv558460815=81, srv730516044=87, srv430875558=72, srv416754856=71, srv1109529933=9, srv1628149721=37, srv1947544082=54, srv1016216226=1, srv1477643094=31, srv8510699=89, srv1435137656=26, srv19445360=53, srv711522768=86, srv468796431=75, srv56273553=83, srv933423432=96, srv1105249884=8, srv2102292564=60, srv315492423=66, srv1122800980=11, srv104552650=2, srv323413399=67, srv432161140=73, srv1589320635=34, srv651326644=84, srv1280020407=17, srv1207059625=13, srv1594566092=35, srv1999237412=56, srv669079844=85, srv1713209126=42, srv1067634110=4, srv1697634846=41, srv327315246=68, srv1295203304=19, srv736710871=88, srv287695384=65, srv1633047795=38, srv904763069=94, srv1225887123=14, srv255019042=62, srv180068182=46, srv561541261=82, srv101312519=0, srv394644503=70, srv99199308=99, srv1455237357=30, srv1484882757=32, srv554768824=80, srv1289089602=18, srv271443089=63, srv1748500844=45, srv2064079453=59, srv932923947=95, srv1341970042=20, srv522933450=78, srv1095167169=7, srv1113959346=10, srv1073630369=5, srv2115468328=61, srv1355564725=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-21 11:59:07,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-21 11:59:07,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-21 11:59:07,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-21 11:59:07,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-21 11:59:07,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-21 11:59:07,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-21 11:59:07,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-21 11:59:07,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-21 11:59:07,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-21 11:59:07,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-21 11:59:07,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-21 11:59:07,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-21 11:59:07,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-21 11:59:07,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-21 11:59:07,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-21 11:59:07,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-21 11:59:07,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-21 11:59:07,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-21 11:59:07,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-21 11:59:07,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-21 11:59:07,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-21 11:59:07,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-21 11:59:07,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-21 11:59:07,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-21 11:59:07,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-21 11:59:07,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-21 11:59:07,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-21 11:59:07,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-21 11:59:07,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-21 11:59:07,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-21 11:59:07,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-21 11:59:07,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-21 11:59:07,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-21 11:59:07,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-21 11:59:07,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-21 11:59:07,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-21 11:59:07,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-21 11:59:07,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-21 11:59:07,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-21 11:59:07,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-21 11:59:07,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-21 11:59:07,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-21 11:59:07,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-21 11:59:07,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-21 11:59:07,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-21 11:59:07,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-21 11:59:07,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-21 11:59:07,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-21 11:59:07,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-21 11:59:07,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-21 11:59:07,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-21 11:59:07,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-21 11:59:07,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-21 11:59:07,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-21 11:59:07,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-21 11:59:07,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-21 11:59:07,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-21 11:59:07,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-21 11:59:07,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-21 11:59:07,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-21 11:59:07,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-21 11:59:07,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-21 11:59:07,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-21 11:59:07,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-21 11:59:07,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-21 11:59:07,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-21 11:59:07,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-21 11:59:07,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-21 11:59:07,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-21 11:59:07,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-21 11:59:07,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-21 11:59:07,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-21 11:59:07,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-21 11:59:07,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-21 11:59:07,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-21 11:59:07,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-21 11:59:07,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-21 11:59:07,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-21 11:59:07,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-21 11:59:07,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-21 11:59:07,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-21 11:59:07,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-21 11:59:07,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-21 11:59:07,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-21 11:59:07,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-21 11:59:07,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-21 11:59:07,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-21 11:59:07,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-21 11:59:07,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-21 11:59:07,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-21 11:59:07,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-21 11:59:07,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-21 11:59:07,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-21 11:59:07,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-21 11:59:07,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-21 11:59:07,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-21 11:59:07,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-21 11:59:07,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-21 11:59:07,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-21 11:59:07,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-21 11:59:07,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-21 11:59:07,191 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021975623902201476); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03204234011499305, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021975623902201476); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04691666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-21 11:59:07,191 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-21 11:59:07,192 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014977101129653373, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021975623902201476); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03204234011499305, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021975623902201476); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04691666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-21 11:59:07,395 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 276 ms to try 5000 different iterations.  Found a solution that moves 995 regions; Going from a computed imbalance of 0.014977101129653373 to a new imbalance of 6.898891519717987E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00834473593247659); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007784989441615229); MoveCostFunction : (multiplier=7.0, imbalance=0.11055555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00834473593247659); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-21 11:59:07,397 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv471906454:77 , srv669079844:81 , srv1657292964:82 , srv282239295:82 , srv1016216226:83 , srv1080663564:83 , srv1113959346:83 , srv1355207830:83 , srv1594566092:83 , srv1743056572:83 , srv2115468328:83 , srv323413399:83 , srv478157603:83 , srv8510699:83 , srv872793267:83 , srv1122800980:84 , srv1263289454:84 , srv1280020407:84 , srv1295203304:84 , srv1455237357:84 , srv1484882757:84 , srv1889193518:84 , srv1999237412:84 , srv2064079453:84 , srv255019042:84 , srv327315246:84 , srv430875558:84 , srv549442543:84 , srv736710871:84 , srv888171015:84 , srv894003238:84 , srv101312519:85 , srv1095167169:85 , srv1176357197:85 , srv12506430:85 , srv1355564725:85 , srv1444216803:85 , srv1697634846:85 , srv1748500844:85 , srv180068182:85 , srv1812985712:85 , srv1839572166:85 , srv1917075767:85 , srv19445360:85 , srv195911693:85 , srv2058754668:85 , srv271443089:85 , srv315492423:85 , srv370908116:85 , srv468796431:85 , srv561541261:85 , srv56273553:85 , srv651326644:85 , srv711522768:85 , srv1073630369:86 , srv1105249884:86 , srv1225887123:86 , srv1289089602:86 , srv1341970042:86 , srv1349750519:86 , srv1369030154:86 , srv1435137656:86 , srv144917355:86 , srv1477643094:86 , srv1589320635:86 , srv1605064869:86 , srv1628149721:86 , srv1713209126:86 , srv1873467913:86 , srv2031018575:86 , srv2102292564:86 , srv287695384:86 , srv394644503:86 , srv416754856:86 , srv432161140:86 , srv871006360:86 , srv99199308:86 , srv1064298867:87 , srv1067634110:87 , srv1207059625:87 , srv1582925722:87 , srv1633047795:87 , srv1692284123:87 , srv1889145682:87 , srv1947544082:87 , srv438693070:87 , srv554768824:87 , srv558460815:87 , srv974644567:87 , srv104552650:88 , srv1109529933:88 , srv1350989743:88 , srv1719189962:88 , srv522933450:88 , srv730516044:88 , srv904763069:88 , srv932923947:88 , srv933423432:88 , srv991881910:89 , srv1436364100:563 }
2023-03-21 11:59:07,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-21 11:59:07,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1692284123=40, srv894003238=93, srv1064298867=3, srv438693070=74, srv974644567=97, srv370908116=69, srv1349750519=21, srv144917355=29, srv1657292964=39, srv1889193518=51, srv282239295=64, srv872793267=91, srv888171015=92, srv1369030154=25, srv1889145682=50, srv2058754668=58, srv1350989743=22, srv1582925722=33, srv1444216803=28, srv1917075767=52, srv991881910=98, srv1839572166=48, srv471906454=76, srv12506430=15, srv1873467913=49, srv1263289454=16, srv871006360=90, srv195911693=55, srv549442543=79, srv1436364100=27, srv478157603=77, srv1080663564=6, srv1719189962=43, srv1812985712=47, srv1605064869=36, srv1176357197=12, srv1743056572=44, srv1355207830=23, srv2031018575=57, srv558460815=81, srv730516044=87, srv430875558=72, srv416754856=71, srv1109529933=9, srv1628149721=37, srv1947544082=54, srv1016216226=1, srv1477643094=31, srv8510699=89, srv1435137656=26, srv19445360=53, srv711522768=86, srv468796431=75, srv56273553=83, srv933423432=96, srv1105249884=8, srv2102292564=60, srv315492423=66, srv1122800980=11, srv104552650=2, srv323413399=67, srv432161140=73, srv1589320635=34, srv651326644=84, srv1280020407=17, srv1207059625=13, srv1594566092=35, srv1999237412=56, srv669079844=85, srv1713209126=42, srv1067634110=4, srv1697634846=41, srv327315246=68, srv1295203304=19, srv736710871=88, srv287695384=65, srv1633047795=38, srv904763069=94, srv1225887123=14, srv255019042=62, srv180068182=46, srv561541261=82, srv101312519=0, srv394644503=70, srv99199308=99, srv1455237357=30, srv1484882757=32, srv554768824=80, srv1289089602=18, srv271443089=63, srv1748500844=45, srv2064079453=59, srv932923947=95, srv1341970042=20, srv522933450=78, srv1095167169=7, srv1113959346=10, srv1073630369=5, srv2115468328=61, srv1355564725=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-21 11:59:07,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-21 11:59:07,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-21 11:59:07,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-21 11:59:07,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-21 11:59:07,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-21 11:59:07,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-21 11:59:07,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-21 11:59:07,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-21 11:59:07,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-21 11:59:07,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-21 11:59:07,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-21 11:59:07,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-21 11:59:07,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-21 11:59:07,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-21 11:59:07,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-21 11:59:07,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-21 11:59:07,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-21 11:59:07,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-21 11:59:07,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-21 11:59:07,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-21 11:59:07,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-21 11:59:07,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-21 11:59:07,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-21 11:59:07,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-21 11:59:07,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-21 11:59:07,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-21 11:59:07,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-21 11:59:07,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-21 11:59:07,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-21 11:59:07,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-21 11:59:07,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-21 11:59:07,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-21 11:59:07,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-21 11:59:07,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-21 11:59:07,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-21 11:59:07,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-21 11:59:07,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-21 11:59:07,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-21 11:59:07,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-21 11:59:07,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-21 11:59:07,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-21 11:59:07,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-21 11:59:07,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-21 11:59:07,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-21 11:59:07,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-21 11:59:07,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-21 11:59:07,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-21 11:59:07,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-21 11:59:07,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-21 11:59:07,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-21 11:59:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-21 11:59:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-21 11:59:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-21 11:59:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-21 11:59:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-21 11:59:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-21 11:59:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-21 11:59:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-21 11:59:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-21 11:59:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-21 11:59:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-21 11:59:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-21 11:59:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-21 11:59:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-21 11:59:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-21 11:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-21 11:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-21 11:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-21 11:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-21 11:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-21 11:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-21 11:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-21 11:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-21 11:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-21 11:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-21 11:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-21 11:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-21 11:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-21 11:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-21 11:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-21 11:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-21 11:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-21 11:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-21 11:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-21 11:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-21 11:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-21 11:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-21 11:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-21 11:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-21 11:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-21 11:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-21 11:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-21 11:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-21 11:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-21 11:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-21 11:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-21 11:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-21 11:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-21 11:59:07,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-21 11:59:07,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-21 11:59:07,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-21 11:59:07,516 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00834473593247659); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007784989441615229); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00834473593247659); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-21 11:59:07,517 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-21 11:59:07,517 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.829220130601529E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00834473593247659); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007784989441615229); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00834473593247659); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-21 11:59:07,725 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 295 ms to try 5000 different iterations.  Found a solution that moves 232 regions; Going from a computed imbalance of 6.829220130601529E-4 to a new imbalance of 5.557476018196731E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.025777777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-21 11:59:07,726 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1109529933:83 , srv1873467913:83 , srv711522768:83 , srv1095167169:84 , srv1122800980:84 , srv1207059625:84 , srv1280020407:84 , srv1295203304:84 , srv1349750519:84 , srv1350989743:84 , srv1369030154:84 , srv1455237357:84 , srv1594566092:84 , srv1633047795:84 , srv1657292964:84 , srv1692284123:84 , srv1713209126:84 , srv1719189962:84 , srv1889145682:84 , srv1889193518:84 , srv2031018575:84 , srv2058754668:84 , srv255019042:84 , srv271443089:84 , srv282239295:84 , srv287695384:84 , srv315492423:84 , srv323413399:84 , srv432161140:84 , srv438693070:84 , srv468796431:84 , srv56273553:84 , srv730516044:84 , srv871006360:84 , srv894003238:84 , srv933423432:84 , srv991881910:84 , srv101312519:85 , srv104552650:85 , srv1064298867:85 , srv1080663564:85 , srv1105249884:85 , srv1176357197:85 , srv1263289454:85 , srv1289089602:85 , srv1341970042:85 , srv1355564725:85 , srv1435137656:85 , srv1444216803:85 , srv1477643094:85 , srv1484882757:85 , srv1582925722:85 , srv1589320635:85 , srv1605064869:85 , srv1628149721:85 , srv1743056572:85 , srv1748500844:85 , srv180068182:85 , srv1812985712:85 , srv1917075767:85 , srv1947544082:85 , srv195911693:85 , srv1999237412:85 , srv2064079453:85 , srv2115468328:85 , srv327315246:85 , srv370908116:85 , srv416754856:85 , srv430875558:85 , srv471906454:85 , srv478157603:85 , srv522933450:85 , srv549442543:85 , srv554768824:85 , srv558460815:85 , srv651326644:85 , srv669079844:85 , srv736710871:85 , srv8510699:85 , srv872793267:85 , srv904763069:85 , srv932923947:85 , srv974644567:85 , srv99199308:85 , srv1016216226:86 , srv1067634110:86 , srv1073630369:86 , srv1113959346:86 , srv1225887123:86 , srv12506430:86 , srv1355207830:86 , srv144917355:86 , srv1697634846:86 , srv1839572166:86 , srv19445360:86 , srv2102292564:86 , srv394644503:86 , srv561541261:86 , srv888171015:86 , srv1436364100:610 }
2023-03-21 11:59:07,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-21 11:59:07,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1692284123=40, srv894003238=93, srv1064298867=3, srv438693070=74, srv974644567=97, srv370908116=69, srv1349750519=21, srv144917355=29, srv1657292964=39, srv1889193518=51, srv282239295=64, srv872793267=91, srv888171015=92, srv1369030154=25, srv1889145682=50, srv2058754668=58, srv1350989743=22, srv1582925722=33, srv1444216803=28, srv1917075767=52, srv991881910=98, srv1839572166=48, srv471906454=76, srv12506430=15, srv1873467913=49, srv1263289454=16, srv871006360=90, srv195911693=55, srv549442543=79, srv1436364100=27, srv478157603=77, srv1080663564=6, srv1719189962=43, srv1812985712=47, srv1605064869=36, srv1176357197=12, srv1743056572=44, srv1355207830=23, srv2031018575=57, srv558460815=81, srv730516044=87, srv430875558=72, srv416754856=71, srv1109529933=9, srv1628149721=37, srv1947544082=54, srv1016216226=1, srv1477643094=31, srv8510699=89, srv1435137656=26, srv19445360=53, srv711522768=86, srv468796431=75, srv56273553=83, srv933423432=96, srv1105249884=8, srv2102292564=60, srv315492423=66, srv1122800980=11, srv104552650=2, srv323413399=67, srv432161140=73, srv1589320635=34, srv651326644=84, srv1280020407=17, srv1207059625=13, srv1594566092=35, srv1999237412=56, srv669079844=85, srv1713209126=42, srv1067634110=4, srv1697634846=41, srv327315246=68, srv1295203304=19, srv736710871=88, srv287695384=65, srv1633047795=38, srv904763069=94, srv1225887123=14, srv255019042=62, srv180068182=46, srv561541261=82, srv101312519=0, srv394644503=70, srv99199308=99, srv1455237357=30, srv1484882757=32, srv554768824=80, srv1289089602=18, srv271443089=63, srv1748500844=45, srv2064079453=59, srv932923947=95, srv1341970042=20, srv522933450=78, srv1095167169=7, srv1113959346=10, srv1073630369=5, srv2115468328=61, srv1355564725=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-21 11:59:07,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-21 11:59:07,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-21 11:59:07,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-21 11:59:07,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-21 11:59:07,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-21 11:59:07,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-21 11:59:07,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-21 11:59:07,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-21 11:59:07,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-21 11:59:07,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-21 11:59:07,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-21 11:59:07,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-21 11:59:07,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-21 11:59:07,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-21 11:59:07,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-21 11:59:07,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-21 11:59:07,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-21 11:59:07,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-21 11:59:07,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-21 11:59:07,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-21 11:59:07,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-21 11:59:07,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-21 11:59:07,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-21 11:59:07,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-21 11:59:07,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-21 11:59:07,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-21 11:59:07,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-21 11:59:07,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-21 11:59:07,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-21 11:59:07,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-21 11:59:07,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-21 11:59:07,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-21 11:59:07,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-21 11:59:07,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-21 11:59:07,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-21 11:59:07,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-21 11:59:07,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-21 11:59:07,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-21 11:59:07,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-21 11:59:07,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-21 11:59:07,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-21 11:59:07,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-21 11:59:07,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-21 11:59:07,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-21 11:59:07,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-21 11:59:07,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-21 11:59:07,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-21 11:59:07,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-21 11:59:07,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-21 11:59:07,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-21 11:59:07,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-21 11:59:07,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-21 11:59:07,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-21 11:59:07,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-21 11:59:07,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-21 11:59:07,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-21 11:59:07,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-21 11:59:07,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-21 11:59:07,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-21 11:59:07,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-21 11:59:07,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-21 11:59:07,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-21 11:59:07,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-21 11:59:07,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-21 11:59:07,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-21 11:59:07,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-21 11:59:07,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-21 11:59:07,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-21 11:59:07,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-21 11:59:07,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-21 11:59:07,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-21 11:59:07,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-21 11:59:07,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-21 11:59:07,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-21 11:59:07,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-21 11:59:07,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-21 11:59:07,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-21 11:59:07,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-21 11:59:07,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-21 11:59:07,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-21 11:59:07,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-21 11:59:07,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-21 11:59:07,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-21 11:59:07,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-21 11:59:07,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-21 11:59:07,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-21 11:59:07,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-21 11:59:07,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-21 11:59:07,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-21 11:59:07,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-21 11:59:07,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-21 11:59:07,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-21 11:59:07,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-21 11:59:07,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-21 11:59:07,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-21 11:59:07,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-21 11:59:07,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-21 11:59:07,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-21 11:59:07,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-21 11:59:07,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-21 11:59:07,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-21 11:59:07,844 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-21 11:59:07,845 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-21 11:59:07,845 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.3950261460859924E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-21 11:59:08,004 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 245 ms to try 5000 different iterations.  Found a solution that moves 24 regions; Going from a computed imbalance of 5.3950261460859924E-5 to a new imbalance of 6.566037264108393E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0026666666666666666); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-21 11:59:08,005 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1067634110:83 , srv1289089602:83 , srv1444216803:83 , srv1633047795:83 , srv327315246:83 , srv554768824:83 , srv101312519:84 , srv1016216226:84 , srv104552650:84 , srv1064298867:84 , srv1073630369:84 , srv1080663564:84 , srv1095167169:84 , srv1105249884:84 , srv1109529933:84 , srv1113959346:84 , srv1122800980:84 , srv1176357197:84 , srv1207059625:84 , srv1225887123:84 , srv12506430:84 , srv1263289454:84 , srv1280020407:84 , srv1341970042:84 , srv1350989743:84 , srv1355207830:84 , srv1355564725:84 , srv1369030154:84 , srv1435137656:84 , srv144917355:84 , srv1455237357:84 , srv1477643094:84 , srv1484882757:84 , srv1582925722:84 , srv1589320635:84 , srv1605064869:84 , srv1628149721:84 , srv1692284123:84 , srv1697634846:84 , srv1713209126:84 , srv1719189962:84 , srv1743056572:84 , srv1748500844:84 , srv180068182:84 , srv1812985712:84 , srv1839572166:84 , srv1889145682:84 , srv1889193518:84 , srv1917075767:84 , srv19445360:84 , srv1947544082:84 , srv195911693:84 , srv1999237412:84 , srv2031018575:84 , srv2058754668:84 , srv2064079453:84 , srv2102292564:84 , srv2115468328:84 , srv255019042:84 , srv271443089:84 , srv282239295:84 , srv287695384:84 , srv315492423:84 , srv323413399:84 , srv370908116:84 , srv394644503:84 , srv416754856:84 , srv430875558:84 , srv432161140:84 , srv438693070:84 , srv468796431:84 , srv471906454:84 , srv478157603:84 , srv522933450:84 , srv549442543:84 , srv558460815:84 , srv561541261:84 , srv651326644:84 , srv669079844:84 , srv711522768:84 , srv730516044:84 , srv736710871:84 , srv8510699:84 , srv871006360:84 , srv872793267:84 , srv888171015:84 , srv894003238:84 , srv904763069:84 , srv932923947:84 , srv933423432:84 , srv974644567:84 , srv991881910:84 , srv99199308:84 , srv1295203304:85 , srv1349750519:85 , srv1594566092:85 , srv1657292964:85 , srv1873467913:85 , srv56273553:85 , srv1436364100:684 }
2023-03-21 11:59:08,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-21 11:59:08,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1692284123=40, srv894003238=93, srv1064298867=3, srv438693070=74, srv974644567=97, srv370908116=69, srv1349750519=21, srv144917355=29, srv1657292964=39, srv1889193518=51, srv282239295=64, srv872793267=91, srv888171015=92, srv1369030154=25, srv1889145682=50, srv2058754668=58, srv1350989743=22, srv1582925722=33, srv1444216803=28, srv1917075767=52, srv991881910=98, srv1839572166=48, srv471906454=76, srv12506430=15, srv1873467913=49, srv1263289454=16, srv871006360=90, srv195911693=55, srv549442543=79, srv1436364100=27, srv478157603=77, srv1080663564=6, srv1719189962=43, srv1812985712=47, srv1605064869=36, srv1176357197=12, srv1743056572=44, srv1355207830=23, srv2031018575=57, srv558460815=81, srv730516044=87, srv430875558=72, srv416754856=71, srv1109529933=9, srv1628149721=37, srv1947544082=54, srv1016216226=1, srv1477643094=31, srv8510699=89, srv1435137656=26, srv19445360=53, srv711522768=86, srv468796431=75, srv56273553=83, srv933423432=96, srv1105249884=8, srv2102292564=60, srv315492423=66, srv1122800980=11, srv104552650=2, srv323413399=67, srv432161140=73, srv1589320635=34, srv651326644=84, srv1280020407=17, srv1207059625=13, srv1594566092=35, srv1999237412=56, srv669079844=85, srv1713209126=42, srv1067634110=4, srv1697634846=41, srv327315246=68, srv1295203304=19, srv736710871=88, srv287695384=65, srv1633047795=38, srv904763069=94, srv1225887123=14, srv255019042=62, srv180068182=46, srv561541261=82, srv101312519=0, srv394644503=70, srv99199308=99, srv1455237357=30, srv1484882757=32, srv554768824=80, srv1289089602=18, srv271443089=63, srv1748500844=45, srv2064079453=59, srv932923947=95, srv1341970042=20, srv522933450=78, srv1095167169=7, srv1113959346=10, srv1073630369=5, srv2115468328=61, srv1355564725=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-21 11:59:08,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-21 11:59:08,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-21 11:59:08,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-21 11:59:08,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-21 11:59:08,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-21 11:59:08,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-21 11:59:08,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-21 11:59:08,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-21 11:59:08,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-21 11:59:08,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-21 11:59:08,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-21 11:59:08,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-21 11:59:08,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-21 11:59:08,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-21 11:59:08,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-21 11:59:08,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-21 11:59:08,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-21 11:59:08,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-21 11:59:08,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-21 11:59:08,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-21 11:59:08,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-21 11:59:08,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-21 11:59:08,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-21 11:59:08,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-21 11:59:08,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-21 11:59:08,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-21 11:59:08,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-21 11:59:08,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-21 11:59:08,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-21 11:59:08,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-21 11:59:08,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-21 11:59:08,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-21 11:59:08,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-21 11:59:08,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-21 11:59:08,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-21 11:59:08,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-21 11:59:08,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-21 11:59:08,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-21 11:59:08,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-21 11:59:08,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-21 11:59:08,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-21 11:59:08,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-21 11:59:08,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-21 11:59:08,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-21 11:59:08,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-21 11:59:08,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-21 11:59:08,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-21 11:59:08,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-21 11:59:08,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-21 11:59:08,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-21 11:59:08,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-21 11:59:08,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-21 11:59:08,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-21 11:59:08,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-21 11:59:08,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-21 11:59:08,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-21 11:59:08,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-21 11:59:08,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-21 11:59:08,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-21 11:59:08,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-21 11:59:08,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-21 11:59:08,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-21 11:59:08,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-21 11:59:08,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-21 11:59:08,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-21 11:59:08,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-21 11:59:08,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-21 11:59:08,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-21 11:59:08,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-21 11:59:08,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-21 11:59:08,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-21 11:59:08,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-21 11:59:08,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-21 11:59:08,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-21 11:59:08,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-21 11:59:08,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-21 11:59:08,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-21 11:59:08,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-21 11:59:08,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-21 11:59:08,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-21 11:59:08,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-21 11:59:08,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-21 11:59:08,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-21 11:59:08,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-21 11:59:08,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-21 11:59:08,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-21 11:59:08,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-21 11:59:08,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-21 11:59:08,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-21 11:59:08,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-21 11:59:08,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-21 11:59:08,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-21 11:59:08,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-21 11:59:08,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-21 11:59:08,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-21 11:59:08,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-21 11:59:08,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-21 11:59:08,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-21 11:59:08,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-21 11:59:08,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-21 11:59:08,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-21 11:59:08,114 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.397985672269699E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-21 11:59:08,115 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1067634110:83 , srv1289089602:83 , srv1444216803:83 , srv1633047795:83 , srv327315246:83 , srv554768824:83 , srv101312519:84 , srv1016216226:84 , srv104552650:84 , srv1064298867:84 , srv1073630369:84 , srv1080663564:84 , srv1095167169:84 , srv1105249884:84 , srv1109529933:84 , srv1113959346:84 , srv1122800980:84 , srv1176357197:84 , srv1207059625:84 , srv1225887123:84 , srv12506430:84 , srv1263289454:84 , srv1280020407:84 , srv1341970042:84 , srv1350989743:84 , srv1355207830:84 , srv1355564725:84 , srv1369030154:84 , srv1435137656:84 , srv144917355:84 , srv1455237357:84 , srv1477643094:84 , srv1484882757:84 , srv1582925722:84 , srv1589320635:84 , srv1605064869:84 , srv1628149721:84 , srv1692284123:84 , srv1697634846:84 , srv1713209126:84 , srv1719189962:84 , srv1743056572:84 , srv1748500844:84 , srv180068182:84 , srv1812985712:84 , srv1839572166:84 , srv1889145682:84 , srv1889193518:84 , srv1917075767:84 , srv19445360:84 , srv1947544082:84 , srv195911693:84 , srv1999237412:84 , srv2031018575:84 , srv2058754668:84 , srv2064079453:84 , srv2102292564:84 , srv2115468328:84 , srv255019042:84 , srv271443089:84 , srv282239295:84 , srv287695384:84 , srv315492423:84 , srv323413399:84 , srv370908116:84 , srv394644503:84 , srv416754856:84 , srv430875558:84 , srv432161140:84 , srv438693070:84 , srv468796431:84 , srv471906454:84 , srv478157603:84 , srv522933450:84 , srv549442543:84 , srv558460815:84 , srv561541261:84 , srv651326644:84 , srv669079844:84 , srv711522768:84 , srv730516044:84 , srv736710871:84 , srv8510699:84 , srv871006360:84 , srv872793267:84 , srv888171015:84 , srv894003238:84 , srv904763069:84 , srv932923947:84 , srv933423432:84 , srv974644567:84 , srv991881910:84 , srv99199308:84 , srv1295203304:85 , srv1349750519:85 , srv1594566092:85 , srv1657292964:85 , srv1873467913:85 , srv56273553:85 , srv1436364100:684 }
2023-03-21 11:59:08,153 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=367 (was 367), ProcessCount=166 (was 166), AvailableMemoryMB=9739 (was 10245)
2023-03-21 11:59:08,163 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=367, ProcessCount=166, AvailableMemoryMB=9739
2023-03-21 11:59:08,163 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-21 11:59:08,164 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-03-21 11:59:08,176 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1017625925:84 , srv1173258317:84 , srv1358172098:84 , srv1371926097:84 , srv1397032517:84 , srv1410769899:84 , srv1568870210:84 , srv1663905734:84 , srv167765017:84 , srv1709598326:84 , srv1720615602:84 , srv1727011968:84 , srv1909599924:84 , srv1943985610:84 , srv1987730512:84 , srv2000975229:84 , srv2007955370:84 , srv2011027238:84 , srv203816908:84 , srv260474782:84 , srv374218359:84 , srv520623998:84 , srv543207668:84 , srv605184874:84 , srv696028105:84 , srv810832334:84 , srv82298997:84 , srv869462641:84 , srv982378825:84 , srv1683233996:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-03-21 11:59:08,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-21 11:59:08,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv167765017=8, srv1720615602=11, srv1727011968=12, srv1987730512=15, srv1371926097=3, srv1709598326=10, srv1943985610=14, srv1173258317=1, srv2007955370=17, srv1017625925=0, srv1663905734=7, srv82298997=27, srv520623998=22, srv1397032517=4, srv1568870210=6, srv696028105=25, srv1410769899=5, srv982378825=29, srv2000975229=16, srv869462641=28, srv1683233996=9, srv543207668=23, srv2011027238=18, srv203816908=19, srv810832334=26, srv605184874=24, srv374218359=21, srv1358172098=2, srv260474782=20, srv1909599924=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-21 11:59:08,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-21 11:59:08,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-21 11:59:08,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-21 11:59:08,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-21 11:59:08,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-21 11:59:08,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-21 11:59:08,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-21 11:59:08,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-21 11:59:08,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-21 11:59:08,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-21 11:59:08,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-21 11:59:08,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-21 11:59:08,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-21 11:59:08,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-21 11:59:08,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-21 11:59:08,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-21 11:59:08,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-21 11:59:08,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-21 11:59:08,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-21 11:59:08,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-21 11:59:08,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-21 11:59:08,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-21 11:59:08,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-21 11:59:08,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-21 11:59:08,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-21 11:59:08,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-21 11:59:08,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-21 11:59:08,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-21 11:59:08,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-21 11:59:08,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-21 11:59:08,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-21 11:59:08,221 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0957558123502542, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-21 11:59:08,222 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912494766702281, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0957558123502542, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-03-21 12:01:08,202 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9525540 different iterations.  Found a solution that moves 1827 regions; Going from a computed imbalance of 0.9912494766702281 to a new imbalance of 4.264309142906872E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6766666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-21 12:01:08,207 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1017625925:90 , srv1173258317:90 , srv1358172098:90 , srv1371926097:90 , srv1397032517:90 , srv1410769899:90 , srv1568870210:90 , srv1663905734:90 , srv167765017:90 , srv1683233996:90 , srv1709598326:90 , srv1720615602:90 , srv1727011968:90 , srv1909599924:90 , srv1943985610:90 , srv1987730512:90 , srv2000975229:90 , srv2007955370:90 , srv2011027238:90 , srv203816908:90 , srv260474782:90 , srv374218359:90 , srv520623998:90 , srv543207668:90 , srv605184874:90 , srv696028105:90 , srv810832334:90 , srv82298997:90 , srv869462641:90 , srv982378825:90 }
2023-03-21 12:01:08,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-21 12:01:08,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv167765017=8, srv1720615602=11, srv1727011968=12, srv1987730512=15, srv1371926097=3, srv1709598326=10, srv1943985610=14, srv1173258317=1, srv2007955370=17, srv1017625925=0, srv1663905734=7, srv82298997=27, srv520623998=22, srv1397032517=4, srv1568870210=6, srv696028105=25, srv1410769899=5, srv982378825=29, srv2000975229=16, srv869462641=28, srv1683233996=9, srv543207668=23, srv2011027238=18, srv203816908=19, srv810832334=26, srv605184874=24, srv374218359=21, srv1358172098=2, srv260474782=20, srv1909599924=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-21 12:01:08,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-21 12:01:08,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-21 12:01:08,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-21 12:01:08,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-21 12:01:08,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-21 12:01:08,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-21 12:01:08,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-21 12:01:08,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-21 12:01:08,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-21 12:01:08,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-21 12:01:08,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-21 12:01:08,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-21 12:01:08,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-21 12:01:08,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-21 12:01:08,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-21 12:01:08,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-21 12:01:08,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-21 12:01:08,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-21 12:01:08,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-21 12:01:08,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-21 12:01:08,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-21 12:01:08,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-21 12:01:08,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-21 12:01:08,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-21 12:01:08,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-21 12:01:08,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-21 12:01:08,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-21 12:01:08,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-21 12:01:08,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-21 12:01:08,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-21 12:01:08,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-21 12:01:08,221 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-21 12:01:08,221 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1017625925:90 , srv1173258317:90 , srv1358172098:90 , srv1371926097:90 , srv1397032517:90 , srv1410769899:90 , srv1568870210:90 , srv1663905734:90 , srv167765017:90 , srv1683233996:90 , srv1709598326:90 , srv1720615602:90 , srv1727011968:90 , srv1909599924:90 , srv1943985610:90 , srv1987730512:90 , srv2000975229:90 , srv2007955370:90 , srv2011027238:90 , srv203816908:90 , srv260474782:90 , srv374218359:90 , srv520623998:90 , srv543207668:90 , srv605184874:90 , srv696028105:90 , srv810832334:90 , srv82298997:90 , srv869462641:90 , srv982378825:90 }
2023-03-21 12:01:08,230 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=173 (was 367), ProcessCount=162 (was 166), AvailableMemoryMB=10322 (was 9739) - AvailableMemoryMB LEAK? -