2023-09-05 07:53:12,842 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-09-05 07:53:13,175 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-09-05 07:53:13,294 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-09-05 07:53:13,364 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-05 07:53:13,372 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-09-05 07:53:13,387 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=329, ProcessCount=170, AvailableMemoryMB=9970
2023-09-05 07:53:13,389 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-05 07:53:13,390 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-09-05 07:53:13,535 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1002377042:84 , srv1009134000:84 , srv103243152:84 , srv1053835125:84 , srv1075324679:84 , srv1081091352:84 , srv1090380560:84 , srv1113542340:84 , srv1116802961:84 , srv1163790990:84 , srv1171766495:84 , srv1172738590:84 , srv1187992806:84 , srv1220587568:84 , srv1222060474:84 , srv1222556427:84 , srv1250386244:84 , srv1270346597:84 , srv1286074314:84 , srv1296310004:84 , srv1348692275:84 , srv1355956905:84 , srv1366032583:84 , srv1394848196:84 , srv1394881176:84 , srv1443475602:84 , srv1449894282:84 , srv1461004258:84 , srv1468096326:84 , srv1517184802:84 , srv151936354:84 , srv1558163944:84 , srv1558187304:84 , srv1569255028:84 , srv1575377080:84 , srv1576696857:84 , srv1585132047:84 , srv1587413808:84 , srv1600889556:84 , srv1657180754:84 , srv1664287562:84 , srv1682748287:84 , srv16986136:84 , srv1701741604:84 , srv1719596718:84 , srv1721150960:84 , srv1724247219:84 , srv1726712737:84 , srv174046076:84 , srv1761624114:84 , srv1770780929:84 , srv1773969508:84 , srv1777772000:84 , srv1781503702:84 , srv1800461418:84 , srv1838719134:84 , srv1879593777:84 , srv1901717197:84 , srv1904847203:84 , srv1911565348:84 , srv1936055506:84 , srv193947562:84 , srv1945300918:84 , srv1957556256:84 , srv2005110244:84 , srv203742922:84 , srv2053263425:84 , srv2084357080:84 , srv2095611745:84 , srv2099540760:84 , srv2146097978:84 , srv377598823:84 , srv388993712:84 , srv399051651:84 , srv451625627:84 , srv470407504:84 , srv484311721:84 , srv493194416:84 , srv49406100:84 , srv527985048:84 , srv545083445:84 , srv550377784:84 , srv550826930:84 , srv563311441:84 , srv566926815:84 , srv629443784:84 , srv639017061:84 , srv678949215:84 , srv702634871:84 , srv816620894:84 , srv841216858:84 , srv846667939:84 , srv86558280:84 , srv875106065:84 , srv904556757:84 , srv928900391:84 , srv932320305:84 , srv934372728:84 , srv998156206:84 , srv887321445:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-09-05 07:53:13,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-05 07:53:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2146097978=70, srv2005110244=64, srv566926815=84, srv932320305=97, srv1172738590=11, srv451625627=74, srv550826930=82, srv1879593777=56, srv1911565348=59, srv639017061=86, srv1250386244=16, srv1600889556=38, srv1838719134=55, srv550377784=81, srv1558187304=32, srv527985048=79, srv1777772000=52, srv1443475602=25, srv399051651=73, srv1270346597=17, srv998156206=99, srv1945300918=62, srv678949215=87, srv1773969508=51, srv16986136=42, srv1296310004=19, srv1682748287=41, srv49406100=78, srv1075324679=4, srv1761624114=49, srv377598823=71, srv1957556256=63, srv1770780929=50, srv1904847203=58, srv702634871=88, srv2053263425=66, srv846667939=91, srv1090380560=6, srv1664287562=40, srv1575377080=34, srv484311721=76, srv103243152=2, srv1901717197=57, srv1113542340=7, srv1569255028=33, srv1009134000=1, srv816620894=89, srv470407504=75, srv1171766495=10, srv1394848196=23, srv1394881176=24, srv1724247219=46, srv1116802961=8, srv1558163944=31, srv1222556427=15, srv563311441=83, srv1286074314=18, srv151936354=30, srv1587413808=37, srv1936055506=60, srv887321445=94, srv545083445=80, srv1726712737=47, srv1348692275=20, srv1781503702=53, srv1657180754=39, srv2095611745=68, srv1449894282=26, srv193947562=61, srv493194416=77, srv1355956905=21, srv1468096326=28, srv1187992806=12, srv203742922=65, srv86558280=92, srv2084357080=67, srv904556757=95, srv1222060474=14, srv928900391=96, srv1220587568=13, srv875106065=93, srv1721150960=45, srv1002377042=0, srv1576696857=35, srv174046076=48, srv1517184802=29, srv1701741604=43, srv629443784=85, srv841216858=90, srv1366032583=22, srv1081091352=5, srv2099540760=69, srv1163790990=9, srv1585132047=36, srv1800461418=54, srv934372728=98, srv1053835125=3, srv1461004258=27, srv388993712=72, srv1719596718=44} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-05 07:53:13,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-05 07:53:13,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-05 07:53:13,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-05 07:53:13,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-05 07:53:13,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-05 07:53:13,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-05 07:53:13,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-05 07:53:13,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-05 07:53:13,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-05 07:53:13,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-05 07:53:13,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-05 07:53:13,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-05 07:53:13,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-05 07:53:13,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-05 07:53:13,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-05 07:53:13,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-05 07:53:13,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-05 07:53:13,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-05 07:53:13,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-05 07:53:13,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-05 07:53:13,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-05 07:53:13,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-05 07:53:13,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-05 07:53:13,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-05 07:53:13,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-05 07:53:13,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-05 07:53:13,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-05 07:53:13,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-05 07:53:13,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-05 07:53:13,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-05 07:53:13,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-05 07:53:13,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-05 07:53:13,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-05 07:53:13,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-05 07:53:13,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-05 07:53:13,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-05 07:53:13,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-05 07:53:13,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-05 07:53:13,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-05 07:53:13,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-05 07:53:13,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-05 07:53:13,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-05 07:53:13,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-05 07:53:13,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-05 07:53:13,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-05 07:53:13,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-05 07:53:13,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-05 07:53:13,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-05 07:53:13,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-05 07:53:13,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-05 07:53:13,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-05 07:53:13,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-05 07:53:13,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-05 07:53:13,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-05 07:53:13,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-05 07:53:13,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-05 07:53:13,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-05 07:53:13,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-05 07:53:13,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-05 07:53:13,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-05 07:53:13,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-05 07:53:13,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-05 07:53:13,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-05 07:53:13,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-05 07:53:13,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-05 07:53:13,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-05 07:53:13,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-05 07:53:13,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-05 07:53:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-05 07:53:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-05 07:53:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-05 07:53:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-05 07:53:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-05 07:53:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-05 07:53:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-05 07:53:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-05 07:53:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-05 07:53:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-05 07:53:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-05 07:53:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-05 07:53:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-05 07:53:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-05 07:53:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-05 07:53:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-05 07:53:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-05 07:53:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-05 07:53:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-05 07:53:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-05 07:53:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-05 07:53:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-05 07:53:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-05 07:53:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-05 07:53:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-05 07:53:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-05 07:53:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-05 07:53:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-05 07:53:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-05 07:53:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-05 07:53:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-05 07:53:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-05 07:53:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-05 07:53:14,117 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-05 07:53:14,117 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-05 07:53:14,118 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-05 07:53:14,882 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1215 ms to try 5000 different iterations.  Found a solution that moves 4660 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18536290771037875. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04052160225617512, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056837729521807635, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5177777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04052160225617512, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18108333333333335, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-05 07:53:14,905 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1355956905:81 , srv1009134000:82 , srv1222556427:82 , srv193947562:82 , srv846667939:82 , srv1075324679:83 , srv1461004258:83 , srv1664287562:83 , srv1777772000:83 , srv1957556256:83 , srv545083445:83 , srv103243152:84 , srv1116802961:84 , srv1286074314:84 , srv1701741604:84 , srv1761624114:84 , srv1773969508:84 , srv1901717197:84 , srv203742922:84 , srv550377784:84 , srv678949215:84 , srv998156206:84 , srv1113542340:85 , srv1172738590:85 , srv1187992806:85 , srv1270346597:85 , srv1569255028:85 , srv16986136:85 , srv1724247219:85 , srv1911565348:85 , srv2005110244:85 , srv2084357080:85 , srv86558280:85 , srv934372728:85 , srv1002377042:86 , srv1090380560:86 , srv1220587568:86 , srv1222060474:86 , srv1296310004:86 , srv1366032583:86 , srv1394848196:86 , srv1443475602:86 , srv1468096326:86 , srv1558163944:86 , srv1585132047:86 , srv1600889556:86 , srv174046076:86 , srv1800461418:86 , srv1936055506:86 , srv1945300918:86 , srv2095611745:86 , srv2146097978:86 , srv377598823:86 , srv493194416:86 , srv527985048:86 , srv629443784:86 , srv932320305:86 , srv1081091352:87 , srv1250386244:87 , srv1348692275:87 , srv1575377080:87 , srv1576696857:87 , srv1682748287:87 , srv1721150960:87 , srv388993712:87 , srv399051651:87 , srv451625627:87 , srv470407504:87 , srv49406100:87 , srv550826930:87 , srv563311441:87 , srv702634871:87 , srv841216858:87 , srv928900391:87 , srv1163790990:88 , srv1449894282:88 , srv1517184802:88 , srv1587413808:88 , srv1719596718:88 , srv1726712737:88 , srv1770780929:88 , srv1838719134:88 , srv1879593777:88 , srv1904847203:88 , srv484311721:88 , srv816620894:88 , srv151936354:89 , srv2053263425:89 , srv566926815:89 , srv639017061:89 , srv1053835125:90 , srv1394881176:90 , srv1558187304:90 , srv1657180754:90 , srv2099540760:90 , srv1171766495:91 , srv875106065:91 , srv1781503702:97 , srv904556757:101 , srv887321445:450 }
2023-09-05 07:53:14,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-05 07:53:14,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2146097978=70, srv2005110244=64, srv566926815=84, srv932320305=97, srv1172738590=11, srv451625627=74, srv550826930=82, srv1879593777=56, srv1911565348=59, srv639017061=86, srv1250386244=16, srv1600889556=38, srv1838719134=55, srv550377784=81, srv1558187304=32, srv527985048=79, srv1777772000=52, srv1443475602=25, srv399051651=73, srv1270346597=17, srv998156206=99, srv1945300918=62, srv678949215=87, srv1773969508=51, srv16986136=42, srv1296310004=19, srv1682748287=41, srv49406100=78, srv1075324679=4, srv1761624114=49, srv377598823=71, srv1957556256=63, srv1770780929=50, srv1904847203=58, srv702634871=88, srv2053263425=66, srv846667939=91, srv1090380560=6, srv1664287562=40, srv1575377080=34, srv484311721=76, srv103243152=2, srv1901717197=57, srv1113542340=7, srv1569255028=33, srv1009134000=1, srv816620894=89, srv470407504=75, srv1171766495=10, srv1394848196=23, srv1394881176=24, srv1724247219=46, srv1116802961=8, srv1558163944=31, srv1222556427=15, srv563311441=83, srv1286074314=18, srv151936354=30, srv1587413808=37, srv1936055506=60, srv887321445=94, srv545083445=80, srv1726712737=47, srv1348692275=20, srv1781503702=53, srv1657180754=39, srv2095611745=68, srv1449894282=26, srv193947562=61, srv493194416=77, srv1355956905=21, srv1468096326=28, srv1187992806=12, srv203742922=65, srv86558280=92, srv2084357080=67, srv904556757=95, srv1222060474=14, srv928900391=96, srv1220587568=13, srv875106065=93, srv1721150960=45, srv1002377042=0, srv1576696857=35, srv174046076=48, srv1517184802=29, srv1701741604=43, srv629443784=85, srv841216858=90, srv1366032583=22, srv1081091352=5, srv2099540760=69, srv1163790990=9, srv1585132047=36, srv1800461418=54, srv934372728=98, srv1053835125=3, srv1461004258=27, srv388993712=72, srv1719596718=44} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-05 07:53:14,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-05 07:53:14,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-05 07:53:14,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-05 07:53:14,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-05 07:53:14,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-05 07:53:14,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-05 07:53:14,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-05 07:53:14,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-05 07:53:14,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-05 07:53:14,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-05 07:53:14,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-05 07:53:14,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-05 07:53:14,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-05 07:53:14,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-05 07:53:14,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-05 07:53:14,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-05 07:53:14,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-05 07:53:14,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-05 07:53:14,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-05 07:53:14,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-05 07:53:14,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-05 07:53:14,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-05 07:53:14,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-05 07:53:14,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-05 07:53:14,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-05 07:53:14,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-05 07:53:14,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-05 07:53:14,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-05 07:53:14,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-05 07:53:14,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-05 07:53:14,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-05 07:53:14,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-05 07:53:14,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-05 07:53:14,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-05 07:53:14,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-05 07:53:14,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-05 07:53:14,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-05 07:53:14,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-05 07:53:14,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-05 07:53:14,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-05 07:53:14,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-05 07:53:14,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-05 07:53:14,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-05 07:53:14,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-05 07:53:14,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-05 07:53:14,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-05 07:53:14,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-05 07:53:14,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-05 07:53:14,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-05 07:53:14,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-05 07:53:14,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-05 07:53:14,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-05 07:53:14,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-05 07:53:14,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-05 07:53:14,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-05 07:53:14,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-05 07:53:14,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-05 07:53:14,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-05 07:53:14,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-05 07:53:14,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-05 07:53:14,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-05 07:53:14,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-05 07:53:14,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-05 07:53:14,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-05 07:53:14,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-05 07:53:14,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-05 07:53:14,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-05 07:53:14,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-05 07:53:14,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-05 07:53:14,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-05 07:53:14,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-05 07:53:14,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-05 07:53:14,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-05 07:53:14,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-05 07:53:14,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-05 07:53:14,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-05 07:53:14,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-05 07:53:14,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-05 07:53:14,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-05 07:53:14,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-05 07:53:14,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-05 07:53:14,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-05 07:53:14,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-05 07:53:14,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-05 07:53:14,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-05 07:53:14,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-05 07:53:14,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-05 07:53:14,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-05 07:53:14,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-05 07:53:14,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-05 07:53:14,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-05 07:53:14,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-05 07:53:14,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-05 07:53:14,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-05 07:53:14,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-05 07:53:14,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-05 07:53:14,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-05 07:53:14,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-05 07:53:14,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-05 07:53:14,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-05 07:53:14,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-05 07:53:15,147 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04052160225617512, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056837729521807635, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04052160225617512, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18108333333333335, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-05 07:53:15,148 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-05 07:53:15,148 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18533027769296337, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04052160225617512, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056837729521807635, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04052160225617512, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18108333333333335, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-05 07:53:15,533 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 574 ms to try 5000 different iterations.  Found a solution that moves 2695 regions; Going from a computed imbalance of 0.18533027769296337 to a new imbalance of 0.014813320981503963. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023412563895228305); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034928177068248016, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29944444444444446, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023412563895228305); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04466666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-05 07:53:15,545 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv904556757:70 , srv1171766495:79 , srv1781503702:79 , srv1558187304:81 , srv639017061:81 , srv1163790990:82 , srv1449894282:82 , srv151936354:82 , srv1838719134:82 , srv1904847203:82 , srv2053263425:82 , srv470407504:82 , srv484311721:82 , srv1053835125:83 , srv1394881176:83 , srv1575377080:83 , srv1657180754:83 , srv1770780929:83 , srv388993712:83 , srv451625627:83 , srv875106065:83 , srv1002377042:84 , srv1081091352:84 , srv1576696857:84 , srv1726712737:84 , srv2099540760:84 , srv49406100:84 , srv566926815:84 , srv928900391:84 , srv932320305:84 , srv1090380560:85 , srv1222060474:85 , srv1443475602:85 , srv1558163944:85 , srv1682748287:85 , srv1719596718:85 , srv1721150960:85 , srv1724247219:85 , srv1800461418:85 , srv1879593777:85 , srv1945300918:85 , srv2084357080:85 , srv2095611745:85 , srv2146097978:85 , srv527985048:85 , srv563311441:85 , srv702634871:85 , srv841216858:85 , srv86558280:85 , srv934372728:85 , srv1220587568:86 , srv1348692275:86 , srv1366032583:86 , srv1517184802:86 , srv1569255028:86 , srv16986136:86 , srv174046076:86 , srv377598823:86 , srv399051651:86 , srv493194416:86 , srv550826930:86 , srv103243152:87 , srv1113542340:87 , srv1172738590:87 , srv1270346597:87 , srv1394848196:87 , srv1468096326:87 , srv1585132047:87 , srv1587413808:87 , srv1600889556:87 , srv1761624114:87 , srv1773969508:87 , srv1901717197:87 , srv203742922:87 , srv545083445:87 , srv629443784:87 , srv1187992806:88 , srv1250386244:88 , srv1286074314:88 , srv1296310004:88 , srv1664287562:88 , srv1777772000:88 , srv1911565348:88 , srv1936055506:88 , srv678949215:88 , srv1116802961:89 , srv1222556427:89 , srv1701741604:89 , srv2005110244:89 , srv550377784:89 , srv1009134000:90 , srv1461004258:90 , srv193947562:90 , srv1957556256:90 , srv846667939:90 , srv998156206:90 , srv1075324679:91 , srv1355956905:91 , srv816620894:92 , srv887321445:532 }
2023-09-05 07:53:15,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-05 07:53:15,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2146097978=70, srv2005110244=64, srv566926815=84, srv932320305=97, srv1172738590=11, srv451625627=74, srv550826930=82, srv1879593777=56, srv1911565348=59, srv639017061=86, srv1250386244=16, srv1600889556=38, srv1838719134=55, srv550377784=81, srv1558187304=32, srv527985048=79, srv1777772000=52, srv1443475602=25, srv399051651=73, srv1270346597=17, srv998156206=99, srv1945300918=62, srv678949215=87, srv1773969508=51, srv16986136=42, srv1296310004=19, srv1682748287=41, srv49406100=78, srv1075324679=4, srv1761624114=49, srv377598823=71, srv1957556256=63, srv1770780929=50, srv1904847203=58, srv702634871=88, srv2053263425=66, srv846667939=91, srv1090380560=6, srv1664287562=40, srv1575377080=34, srv484311721=76, srv103243152=2, srv1901717197=57, srv1113542340=7, srv1569255028=33, srv1009134000=1, srv816620894=89, srv470407504=75, srv1171766495=10, srv1394848196=23, srv1394881176=24, srv1724247219=46, srv1116802961=8, srv1558163944=31, srv1222556427=15, srv563311441=83, srv1286074314=18, srv151936354=30, srv1587413808=37, srv1936055506=60, srv887321445=94, srv545083445=80, srv1726712737=47, srv1348692275=20, srv1781503702=53, srv1657180754=39, srv2095611745=68, srv1449894282=26, srv193947562=61, srv493194416=77, srv1355956905=21, srv1468096326=28, srv1187992806=12, srv203742922=65, srv86558280=92, srv2084357080=67, srv904556757=95, srv1222060474=14, srv928900391=96, srv1220587568=13, srv875106065=93, srv1721150960=45, srv1002377042=0, srv1576696857=35, srv174046076=48, srv1517184802=29, srv1701741604=43, srv629443784=85, srv841216858=90, srv1366032583=22, srv1081091352=5, srv2099540760=69, srv1163790990=9, srv1585132047=36, srv1800461418=54, srv934372728=98, srv1053835125=3, srv1461004258=27, srv388993712=72, srv1719596718=44} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-05 07:53:15,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-05 07:53:15,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-05 07:53:15,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-05 07:53:15,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-05 07:53:15,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-05 07:53:15,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-05 07:53:15,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-05 07:53:15,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-05 07:53:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-05 07:53:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-05 07:53:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-05 07:53:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-05 07:53:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-05 07:53:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-05 07:53:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-05 07:53:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-05 07:53:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-05 07:53:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-05 07:53:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-05 07:53:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-05 07:53:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-05 07:53:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-05 07:53:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-05 07:53:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-05 07:53:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-05 07:53:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-05 07:53:15,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-05 07:53:15,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-05 07:53:15,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-05 07:53:15,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-05 07:53:15,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-05 07:53:15,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-05 07:53:15,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-05 07:53:15,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-05 07:53:15,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-05 07:53:15,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-05 07:53:15,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-05 07:53:15,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-05 07:53:15,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-05 07:53:15,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-05 07:53:15,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-05 07:53:15,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-05 07:53:15,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-05 07:53:15,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-05 07:53:15,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-05 07:53:15,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-05 07:53:15,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-05 07:53:15,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-05 07:53:15,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-05 07:53:15,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-05 07:53:15,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-05 07:53:15,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-05 07:53:15,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-05 07:53:15,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-05 07:53:15,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-05 07:53:15,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-05 07:53:15,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-05 07:53:15,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-05 07:53:15,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-05 07:53:15,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-05 07:53:15,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-05 07:53:15,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-05 07:53:15,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-05 07:53:15,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-05 07:53:15,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-05 07:53:15,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-05 07:53:15,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-05 07:53:15,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-05 07:53:15,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-05 07:53:15,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-05 07:53:15,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-05 07:53:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-05 07:53:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-05 07:53:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-05 07:53:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-05 07:53:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-05 07:53:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-05 07:53:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-05 07:53:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-05 07:53:15,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-05 07:53:15,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-05 07:53:15,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-05 07:53:15,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-05 07:53:15,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-05 07:53:15,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-05 07:53:15,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-05 07:53:15,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-05 07:53:15,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-05 07:53:15,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-05 07:53:15,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-05 07:53:15,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-05 07:53:15,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-05 07:53:15,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-05 07:53:15,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-05 07:53:15,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-05 07:53:15,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-05 07:53:15,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-05 07:53:15,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-05 07:53:15,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-05 07:53:15,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-05 07:53:15,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-05 07:53:15,745 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023412563895228305); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034928177068248016, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023412563895228305); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04466666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-05 07:53:15,745 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-05 07:53:15,745 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01479445018817041, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023412563895228305); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034928177068248016, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023412563895228305); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04466666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-05 07:53:16,065 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 457 ms to try 5000 different iterations.  Found a solution that moves 1003 regions; Going from a computed imbalance of 0.01479445018817041 to a new imbalance of 9.048670909172565E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005768100330189269); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008287754140107479); MoveCostFunction : (multiplier=7.0, imbalance=0.11144444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005768100330189269); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-05 07:53:16,066 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv816620894:79 , srv1587413808:82 , srv1781503702:82 , srv1250386244:83 , srv1296310004:83 , srv1911565348:83 , srv1053835125:84 , srv1075324679:84 , srv1116802961:84 , srv1468096326:84 , srv1517184802:84 , srv1600889556:84 , srv1701741604:84 , srv1719596718:84 , srv1879593777:84 , srv1936055506:84 , srv1957556256:84 , srv2005110244:84 , srv2099540760:84 , srv629443784:84 , srv875106065:84 , srv998156206:84 , srv1009134000:85 , srv1113542340:85 , srv1172738590:85 , srv1187992806:85 , srv1286074314:85 , srv1348692275:85 , srv1355956905:85 , srv1366032583:85 , srv1394848196:85 , srv1461004258:85 , srv1585132047:85 , srv1657180754:85 , srv16986136:85 , srv1726712737:85 , srv377598823:85 , srv399051651:85 , srv493194416:85 , srv550377784:85 , srv550826930:85 , srv563311441:85 , srv566926815:85 , srv678949215:85 , srv841216858:85 , srv103243152:86 , srv1081091352:86 , srv1090380560:86 , srv1220587568:86 , srv1222060474:86 , srv1270346597:86 , srv1394881176:86 , srv151936354:86 , srv1558187304:86 , srv1576696857:86 , srv1664287562:86 , srv1682748287:86 , srv1721150960:86 , srv174046076:86 , srv1773969508:86 , srv1777772000:86 , srv1901717197:86 , srv193947562:86 , srv1945300918:86 , srv203742922:86 , srv2053263425:86 , srv2084357080:86 , srv2095611745:86 , srv49406100:86 , srv527985048:86 , srv846667939:86 , srv904556757:86 , srv1002377042:87 , srv1163790990:87 , srv1222556427:87 , srv1443475602:87 , srv1449894282:87 , srv1569255028:87 , srv1575377080:87 , srv1724247219:87 , srv1770780929:87 , srv1800461418:87 , srv2146097978:87 , srv484311721:87 , srv639017061:87 , srv702634871:87 , srv928900391:87 , srv932320305:87 , srv1171766495:88 , srv1558163944:88 , srv1761624114:88 , srv1838719134:88 , srv1904847203:88 , srv388993712:88 , srv451625627:88 , srv470407504:88 , srv545083445:88 , srv86558280:88 , srv934372728:88 , srv887321445:527 }
2023-09-05 07:53:16,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-05 07:53:16,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2146097978=70, srv2005110244=64, srv566926815=84, srv932320305=97, srv1172738590=11, srv451625627=74, srv550826930=82, srv1879593777=56, srv1911565348=59, srv639017061=86, srv1250386244=16, srv1600889556=38, srv1838719134=55, srv550377784=81, srv1558187304=32, srv527985048=79, srv1777772000=52, srv1443475602=25, srv399051651=73, srv1270346597=17, srv998156206=99, srv1945300918=62, srv678949215=87, srv1773969508=51, srv16986136=42, srv1296310004=19, srv1682748287=41, srv49406100=78, srv1075324679=4, srv1761624114=49, srv377598823=71, srv1957556256=63, srv1770780929=50, srv1904847203=58, srv702634871=88, srv2053263425=66, srv846667939=91, srv1090380560=6, srv1664287562=40, srv1575377080=34, srv484311721=76, srv103243152=2, srv1901717197=57, srv1113542340=7, srv1569255028=33, srv1009134000=1, srv816620894=89, srv470407504=75, srv1171766495=10, srv1394848196=23, srv1394881176=24, srv1724247219=46, srv1116802961=8, srv1558163944=31, srv1222556427=15, srv563311441=83, srv1286074314=18, srv151936354=30, srv1587413808=37, srv1936055506=60, srv887321445=94, srv545083445=80, srv1726712737=47, srv1348692275=20, srv1781503702=53, srv1657180754=39, srv2095611745=68, srv1449894282=26, srv193947562=61, srv493194416=77, srv1355956905=21, srv1468096326=28, srv1187992806=12, srv203742922=65, srv86558280=92, srv2084357080=67, srv904556757=95, srv1222060474=14, srv928900391=96, srv1220587568=13, srv875106065=93, srv1721150960=45, srv1002377042=0, srv1576696857=35, srv174046076=48, srv1517184802=29, srv1701741604=43, srv629443784=85, srv841216858=90, srv1366032583=22, srv1081091352=5, srv2099540760=69, srv1163790990=9, srv1585132047=36, srv1800461418=54, srv934372728=98, srv1053835125=3, srv1461004258=27, srv388993712=72, srv1719596718=44} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-05 07:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-05 07:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-05 07:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-05 07:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-05 07:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-05 07:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-05 07:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-05 07:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-05 07:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-05 07:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-05 07:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-05 07:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-05 07:53:16,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-05 07:53:16,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-05 07:53:16,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-05 07:53:16,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-05 07:53:16,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-05 07:53:16,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-05 07:53:16,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-05 07:53:16,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-05 07:53:16,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-05 07:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-05 07:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-05 07:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-05 07:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-05 07:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-05 07:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-05 07:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-05 07:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-05 07:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-05 07:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-05 07:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-05 07:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-05 07:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-05 07:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-05 07:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-05 07:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-05 07:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-05 07:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-05 07:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-05 07:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-05 07:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-05 07:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-05 07:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-05 07:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-05 07:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-05 07:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-05 07:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-05 07:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-05 07:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-05 07:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-05 07:53:16,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-05 07:53:16,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-05 07:53:16,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-05 07:53:16,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-05 07:53:16,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-05 07:53:16,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-05 07:53:16,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-05 07:53:16,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-05 07:53:16,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-05 07:53:16,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-05 07:53:16,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-05 07:53:16,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-05 07:53:16,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-05 07:53:16,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-05 07:53:16,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-05 07:53:16,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-05 07:53:16,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-05 07:53:16,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-05 07:53:16,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-05 07:53:16,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-05 07:53:16,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-05 07:53:16,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-05 07:53:16,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-05 07:53:16,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-05 07:53:16,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-05 07:53:16,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-05 07:53:16,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-05 07:53:16,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-05 07:53:16,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-05 07:53:16,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-05 07:53:16,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-05 07:53:16,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-05 07:53:16,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-05 07:53:16,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-05 07:53:16,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-05 07:53:16,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-05 07:53:16,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-05 07:53:16,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-05 07:53:16,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-05 07:53:16,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-05 07:53:16,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-05 07:53:16,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-05 07:53:16,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-05 07:53:16,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-05 07:53:16,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-05 07:53:16,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-05 07:53:16,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-05 07:53:16,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-05 07:53:16,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-05 07:53:16,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-05 07:53:16,213 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005768100330189269); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008287754140107479); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005768100330189269); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-05 07:53:16,213 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-05 07:53:16,214 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.978439348083312E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005768100330189269); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008287754140107479); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005768100330189269); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-05 07:53:16,598 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 495 ms to try 5000 different iterations.  Found a solution that moves 249 regions; Going from a computed imbalance of 8.978439348083312E-4 to a new imbalance of 3.5067801364534784E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.027666666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-05 07:53:16,600 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1394881176:83 , srv1558163944:83 , srv1761624114:83 , srv1800461418:83 , srv545083445:83 , srv702634871:83 , srv816620894:83 , srv1075324679:84 , srv1171766495:84 , srv1187992806:84 , srv1220587568:84 , srv1222556427:84 , srv1250386244:84 , srv1270346597:84 , srv1348692275:84 , srv1394848196:84 , srv1443475602:84 , srv1449894282:84 , srv1461004258:84 , srv1517184802:84 , srv1569255028:84 , srv1585132047:84 , srv1657180754:84 , srv1682748287:84 , srv1721150960:84 , srv174046076:84 , srv1770780929:84 , srv1781503702:84 , srv1838719134:84 , srv1904847203:84 , srv1936055506:84 , srv193947562:84 , srv2005110244:84 , srv2146097978:84 , srv388993712:84 , srv399051651:84 , srv451625627:84 , srv550826930:84 , srv566926815:84 , srv846667939:84 , srv86558280:84 , srv875106065:84 , srv904556757:84 , srv928900391:84 , srv934372728:84 , srv998156206:84 , srv1002377042:85 , srv1009134000:85 , srv103243152:85 , srv1053835125:85 , srv1081091352:85 , srv1090380560:85 , srv1116802961:85 , srv1163790990:85 , srv1172738590:85 , srv1222060474:85 , srv1286074314:85 , srv1296310004:85 , srv1355956905:85 , srv1366032583:85 , srv1468096326:85 , srv151936354:85 , srv1558187304:85 , srv1575377080:85 , srv1587413808:85 , srv1600889556:85 , srv1664287562:85 , srv1701741604:85 , srv1719596718:85 , srv1724247219:85 , srv1726712737:85 , srv1773969508:85 , srv1777772000:85 , srv1879593777:85 , srv1901717197:85 , srv1945300918:85 , srv1957556256:85 , srv203742922:85 , srv2053263425:85 , srv2095611745:85 , srv2099540760:85 , srv377598823:85 , srv470407504:85 , srv484311721:85 , srv493194416:85 , srv49406100:85 , srv527985048:85 , srv550377784:85 , srv563311441:85 , srv629443784:85 , srv639017061:85 , srv678949215:85 , srv841216858:85 , srv932320305:85 , srv1113542340:86 , srv1576696857:86 , srv16986136:86 , srv1911565348:86 , srv2084357080:86 , srv887321445:633 }
2023-09-05 07:53:16,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-05 07:53:16,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2146097978=70, srv2005110244=64, srv566926815=84, srv932320305=97, srv1172738590=11, srv451625627=74, srv550826930=82, srv1879593777=56, srv1911565348=59, srv639017061=86, srv1250386244=16, srv1600889556=38, srv1838719134=55, srv550377784=81, srv1558187304=32, srv527985048=79, srv1777772000=52, srv1443475602=25, srv399051651=73, srv1270346597=17, srv998156206=99, srv1945300918=62, srv678949215=87, srv1773969508=51, srv16986136=42, srv1296310004=19, srv1682748287=41, srv49406100=78, srv1075324679=4, srv1761624114=49, srv377598823=71, srv1957556256=63, srv1770780929=50, srv1904847203=58, srv702634871=88, srv2053263425=66, srv846667939=91, srv1090380560=6, srv1664287562=40, srv1575377080=34, srv484311721=76, srv103243152=2, srv1901717197=57, srv1113542340=7, srv1569255028=33, srv1009134000=1, srv816620894=89, srv470407504=75, srv1171766495=10, srv1394848196=23, srv1394881176=24, srv1724247219=46, srv1116802961=8, srv1558163944=31, srv1222556427=15, srv563311441=83, srv1286074314=18, srv151936354=30, srv1587413808=37, srv1936055506=60, srv887321445=94, srv545083445=80, srv1726712737=47, srv1348692275=20, srv1781503702=53, srv1657180754=39, srv2095611745=68, srv1449894282=26, srv193947562=61, srv493194416=77, srv1355956905=21, srv1468096326=28, srv1187992806=12, srv203742922=65, srv86558280=92, srv2084357080=67, srv904556757=95, srv1222060474=14, srv928900391=96, srv1220587568=13, srv875106065=93, srv1721150960=45, srv1002377042=0, srv1576696857=35, srv174046076=48, srv1517184802=29, srv1701741604=43, srv629443784=85, srv841216858=90, srv1366032583=22, srv1081091352=5, srv2099540760=69, srv1163790990=9, srv1585132047=36, srv1800461418=54, srv934372728=98, srv1053835125=3, srv1461004258=27, srv388993712=72, srv1719596718=44} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-05 07:53:16,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-05 07:53:16,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-05 07:53:16,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-05 07:53:16,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-05 07:53:16,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-05 07:53:16,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-05 07:53:16,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-05 07:53:16,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-05 07:53:16,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-05 07:53:16,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-05 07:53:16,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-05 07:53:16,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-05 07:53:16,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-05 07:53:16,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-05 07:53:16,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-05 07:53:16,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-05 07:53:16,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-05 07:53:16,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-05 07:53:16,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-05 07:53:16,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-05 07:53:16,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-05 07:53:16,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-05 07:53:16,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-05 07:53:16,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-05 07:53:16,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-05 07:53:16,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-05 07:53:16,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-05 07:53:16,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-05 07:53:16,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-05 07:53:16,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-05 07:53:16,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-05 07:53:16,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-05 07:53:16,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-05 07:53:16,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-05 07:53:16,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-05 07:53:16,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-05 07:53:16,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-05 07:53:16,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-05 07:53:16,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-05 07:53:16,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-05 07:53:16,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-05 07:53:16,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-05 07:53:16,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-05 07:53:16,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-05 07:53:16,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-05 07:53:16,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-05 07:53:16,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-05 07:53:16,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-05 07:53:16,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-05 07:53:16,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-05 07:53:16,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-05 07:53:16,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-05 07:53:16,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-05 07:53:16,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-05 07:53:16,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-05 07:53:16,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-05 07:53:16,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-05 07:53:16,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-05 07:53:16,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-05 07:53:16,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-05 07:53:16,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-05 07:53:16,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-05 07:53:16,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-05 07:53:16,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-05 07:53:16,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-05 07:53:16,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-05 07:53:16,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-05 07:53:16,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-05 07:53:16,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-05 07:53:16,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-05 07:53:16,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-05 07:53:16,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-05 07:53:16,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-05 07:53:16,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-05 07:53:16,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-05 07:53:16,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-05 07:53:16,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-05 07:53:16,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-05 07:53:16,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-05 07:53:16,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-05 07:53:16,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-05 07:53:16,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-05 07:53:16,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-05 07:53:16,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-05 07:53:16,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-05 07:53:16,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-05 07:53:16,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-05 07:53:16,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-05 07:53:16,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-05 07:53:16,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-05 07:53:16,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-05 07:53:16,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-05 07:53:16,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-05 07:53:16,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-05 07:53:16,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-05 07:53:16,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-05 07:53:16,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-05 07:53:16,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-05 07:53:16,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-05 07:53:16,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-05 07:53:16,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-05 07:53:16,767 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-05 07:53:16,767 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-05 07:53:16,767 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.332426609920833E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-05 07:53:17,043 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 389 ms to try 5000 different iterations.  Found a solution that moves 20 regions; Going from a computed imbalance of 3.332426609920833E-5 to a new imbalance of 6.88411204703086E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0022222222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-05 07:53:17,044 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1113542340:83 , srv1576696857:83 , srv2099540760:83 , srv550377784:83 , srv1002377042:84 , srv1009134000:84 , srv103243152:84 , srv1053835125:84 , srv1075324679:84 , srv1081091352:84 , srv1090380560:84 , srv1116802961:84 , srv1163790990:84 , srv1171766495:84 , srv1172738590:84 , srv1187992806:84 , srv1220587568:84 , srv1222060474:84 , srv1222556427:84 , srv1250386244:84 , srv1270346597:84 , srv1286074314:84 , srv1296310004:84 , srv1348692275:84 , srv1355956905:84 , srv1366032583:84 , srv1394848196:84 , srv1394881176:84 , srv1443475602:84 , srv1461004258:84 , srv1468096326:84 , srv1517184802:84 , srv151936354:84 , srv1558163944:84 , srv1558187304:84 , srv1569255028:84 , srv1575377080:84 , srv1585132047:84 , srv1587413808:84 , srv1600889556:84 , srv1657180754:84 , srv1664287562:84 , srv1682748287:84 , srv16986136:84 , srv1701741604:84 , srv1719596718:84 , srv1721150960:84 , srv1724247219:84 , srv1726712737:84 , srv174046076:84 , srv1761624114:84 , srv1770780929:84 , srv1773969508:84 , srv1777772000:84 , srv1781503702:84 , srv1838719134:84 , srv1879593777:84 , srv1901717197:84 , srv1904847203:84 , srv1911565348:84 , srv1936055506:84 , srv193947562:84 , srv1945300918:84 , srv1957556256:84 , srv2005110244:84 , srv203742922:84 , srv2053263425:84 , srv2084357080:84 , srv2095611745:84 , srv2146097978:84 , srv377598823:84 , srv388993712:84 , srv399051651:84 , srv451625627:84 , srv470407504:84 , srv484311721:84 , srv493194416:84 , srv49406100:84 , srv527985048:84 , srv550826930:84 , srv563311441:84 , srv566926815:84 , srv629443784:84 , srv639017061:84 , srv678949215:84 , srv702634871:84 , srv816620894:84 , srv841216858:84 , srv846667939:84 , srv86558280:84 , srv875106065:84 , srv928900391:84 , srv932320305:84 , srv934372728:84 , srv998156206:84 , srv1449894282:85 , srv1800461418:85 , srv545083445:85 , srv904556757:85 , srv887321445:684 }
2023-09-05 07:53:17,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-05 07:53:17,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2146097978=70, srv2005110244=64, srv566926815=84, srv932320305=97, srv1172738590=11, srv451625627=74, srv550826930=82, srv1879593777=56, srv1911565348=59, srv639017061=86, srv1250386244=16, srv1600889556=38, srv1838719134=55, srv550377784=81, srv1558187304=32, srv527985048=79, srv1777772000=52, srv1443475602=25, srv399051651=73, srv1270346597=17, srv998156206=99, srv1945300918=62, srv678949215=87, srv1773969508=51, srv16986136=42, srv1296310004=19, srv1682748287=41, srv49406100=78, srv1075324679=4, srv1761624114=49, srv377598823=71, srv1957556256=63, srv1770780929=50, srv1904847203=58, srv702634871=88, srv2053263425=66, srv846667939=91, srv1090380560=6, srv1664287562=40, srv1575377080=34, srv484311721=76, srv103243152=2, srv1901717197=57, srv1113542340=7, srv1569255028=33, srv1009134000=1, srv816620894=89, srv470407504=75, srv1171766495=10, srv1394848196=23, srv1394881176=24, srv1724247219=46, srv1116802961=8, srv1558163944=31, srv1222556427=15, srv563311441=83, srv1286074314=18, srv151936354=30, srv1587413808=37, srv1936055506=60, srv887321445=94, srv545083445=80, srv1726712737=47, srv1348692275=20, srv1781503702=53, srv1657180754=39, srv2095611745=68, srv1449894282=26, srv193947562=61, srv493194416=77, srv1355956905=21, srv1468096326=28, srv1187992806=12, srv203742922=65, srv86558280=92, srv2084357080=67, srv904556757=95, srv1222060474=14, srv928900391=96, srv1220587568=13, srv875106065=93, srv1721150960=45, srv1002377042=0, srv1576696857=35, srv174046076=48, srv1517184802=29, srv1701741604=43, srv629443784=85, srv841216858=90, srv1366032583=22, srv1081091352=5, srv2099540760=69, srv1163790990=9, srv1585132047=36, srv1800461418=54, srv934372728=98, srv1053835125=3, srv1461004258=27, srv388993712=72, srv1719596718=44} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-05 07:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-05 07:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-05 07:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-05 07:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-05 07:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-05 07:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-05 07:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-05 07:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-05 07:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-05 07:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-05 07:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-05 07:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-05 07:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-05 07:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-05 07:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-05 07:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-05 07:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-05 07:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-05 07:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-05 07:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-05 07:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-05 07:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-05 07:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-05 07:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-05 07:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-05 07:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-05 07:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-05 07:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-05 07:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-05 07:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-05 07:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-05 07:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-05 07:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-05 07:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-05 07:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-05 07:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-05 07:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-05 07:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-05 07:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-05 07:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-05 07:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-05 07:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-05 07:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-05 07:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-05 07:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-05 07:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-05 07:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-05 07:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-05 07:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-05 07:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-05 07:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-05 07:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-05 07:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-05 07:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-05 07:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-05 07:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-05 07:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-05 07:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-05 07:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-05 07:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-05 07:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-05 07:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-05 07:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-05 07:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-05 07:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-05 07:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-05 07:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-05 07:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-05 07:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-05 07:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-05 07:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-05 07:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-05 07:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-05 07:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-05 07:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-05 07:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-05 07:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-05 07:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-05 07:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-05 07:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-05 07:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-05 07:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-05 07:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-05 07:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-05 07:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-05 07:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-05 07:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-05 07:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-05 07:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-05 07:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-05 07:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-05 07:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-05 07:53:17,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-05 07:53:17,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-05 07:53:17,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-05 07:53:17,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-05 07:53:17,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-05 07:53:17,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-05 07:53:17,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-05 07:53:17,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-05 07:53:17,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-05 07:53:17,201 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-05 07:53:17,202 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1113542340:83 , srv1576696857:83 , srv2099540760:83 , srv550377784:83 , srv1002377042:84 , srv1009134000:84 , srv103243152:84 , srv1053835125:84 , srv1075324679:84 , srv1081091352:84 , srv1090380560:84 , srv1116802961:84 , srv1163790990:84 , srv1171766495:84 , srv1172738590:84 , srv1187992806:84 , srv1220587568:84 , srv1222060474:84 , srv1222556427:84 , srv1250386244:84 , srv1270346597:84 , srv1286074314:84 , srv1296310004:84 , srv1348692275:84 , srv1355956905:84 , srv1366032583:84 , srv1394848196:84 , srv1394881176:84 , srv1443475602:84 , srv1461004258:84 , srv1468096326:84 , srv1517184802:84 , srv151936354:84 , srv1558163944:84 , srv1558187304:84 , srv1569255028:84 , srv1575377080:84 , srv1585132047:84 , srv1587413808:84 , srv1600889556:84 , srv1657180754:84 , srv1664287562:84 , srv1682748287:84 , srv16986136:84 , srv1701741604:84 , srv1719596718:84 , srv1721150960:84 , srv1724247219:84 , srv1726712737:84 , srv174046076:84 , srv1761624114:84 , srv1770780929:84 , srv1773969508:84 , srv1777772000:84 , srv1781503702:84 , srv1838719134:84 , srv1879593777:84 , srv1901717197:84 , srv1904847203:84 , srv1911565348:84 , srv1936055506:84 , srv193947562:84 , srv1945300918:84 , srv1957556256:84 , srv2005110244:84 , srv203742922:84 , srv2053263425:84 , srv2084357080:84 , srv2095611745:84 , srv2146097978:84 , srv377598823:84 , srv388993712:84 , srv399051651:84 , srv451625627:84 , srv470407504:84 , srv484311721:84 , srv493194416:84 , srv49406100:84 , srv527985048:84 , srv550826930:84 , srv563311441:84 , srv566926815:84 , srv629443784:84 , srv639017061:84 , srv678949215:84 , srv702634871:84 , srv816620894:84 , srv841216858:84 , srv846667939:84 , srv86558280:84 , srv875106065:84 , srv928900391:84 , srv932320305:84 , srv934372728:84 , srv998156206:84 , srv1449894282:85 , srv1800461418:85 , srv545083445:85 , srv904556757:85 , srv887321445:684 }
2023-09-05 07:53:17,227 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=343 (was 329) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 170), AvailableMemoryMB=9111 (was 9970)
2023-09-05 07:53:17,236 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=343, ProcessCount=169, AvailableMemoryMB=9109
2023-09-05 07:53:17,237 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-05 07:53:17,237 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-09-05 07:53:17,250 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1104463987:84 , srv1122584111:84 , srv1127749294:84 , srv1136497384:84 , srv1200790810:84 , srv1310963459:84 , srv1496520389:84 , srv1559191718:84 , srv1669884102:84 , srv1672352032:84 , srv1695082678:84 , srv180341325:84 , srv1826587707:84 , srv1938550277:84 , srv2034407412:84 , srv229691786:84 , srv231509650:84 , srv379090905:84 , srv650834597:84 , srv658259701:84 , srv675714733:84 , srv753819532:84 , srv823453383:84 , srv849834882:84 , srv871721012:84 , srv894454316:84 , srv925477736:84 , srv982119283:84 , srv992417611:84 , srv1866324747:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-09-05 07:53:17,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-05 07:53:17,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1669884102=8, srv1496520389=6, srv1122584111=1, srv871721012=25, srv982119283=28, srv1104463987=0, srv925477736=27, srv180341325=11, srv1866324747=13, srv1938550277=14, srv1127749294=2, srv1559191718=7, srv675714733=21, srv658259701=20, srv1200790810=4, srv1695082678=10, srv992417611=29, srv823453383=23, srv849834882=24, srv379090905=18, srv229691786=16, srv231509650=17, srv1310963459=5, srv1672352032=9, srv1136497384=3, srv2034407412=15, srv753819532=22, srv650834597=19, srv894454316=26, srv1826587707=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-05 07:53:17,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-05 07:53:17,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-05 07:53:17,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-05 07:53:17,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-05 07:53:17,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-05 07:53:17,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-05 07:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-05 07:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-05 07:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-05 07:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-05 07:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-05 07:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-05 07:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-05 07:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-05 07:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-05 07:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-05 07:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-05 07:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-05 07:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-05 07:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-05 07:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-05 07:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-05 07:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-05 07:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-05 07:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-05 07:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-05 07:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-05 07:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-05 07:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-05 07:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-05 07:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-09-05 07:53:17,298 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1591719143524948, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-05 07:53:17,298 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912694588768963, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1591719143524948, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-09-05 07:55:17,273 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6296375 different iterations.  Found a solution that moves 1858 regions; Going from a computed imbalance of 0.9912694588768963 to a new imbalance of 4.336664689392977E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6881481481481482, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-05 07:55:17,280 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1104463987:90 , srv1122584111:90 , srv1127749294:90 , srv1136497384:90 , srv1200790810:90 , srv1310963459:90 , srv1496520389:90 , srv1559191718:90 , srv1669884102:90 , srv1672352032:90 , srv1695082678:90 , srv180341325:90 , srv1826587707:90 , srv1866324747:90 , srv1938550277:90 , srv2034407412:90 , srv229691786:90 , srv231509650:90 , srv379090905:90 , srv650834597:90 , srv658259701:90 , srv675714733:90 , srv753819532:90 , srv823453383:90 , srv849834882:90 , srv871721012:90 , srv894454316:90 , srv925477736:90 , srv982119283:90 , srv992417611:90 }
2023-09-05 07:55:17,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-05 07:55:17,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1669884102=8, srv1496520389=6, srv1122584111=1, srv871721012=25, srv982119283=28, srv1104463987=0, srv925477736=27, srv180341325=11, srv1866324747=13, srv1938550277=14, srv1127749294=2, srv1559191718=7, srv675714733=21, srv658259701=20, srv1200790810=4, srv1695082678=10, srv992417611=29, srv823453383=23, srv849834882=24, srv379090905=18, srv229691786=16, srv231509650=17, srv1310963459=5, srv1672352032=9, srv1136497384=3, srv2034407412=15, srv753819532=22, srv650834597=19, srv894454316=26, srv1826587707=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-05 07:55:17,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-05 07:55:17,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-05 07:55:17,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-05 07:55:17,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-05 07:55:17,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-05 07:55:17,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-05 07:55:17,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-05 07:55:17,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-05 07:55:17,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-05 07:55:17,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-05 07:55:17,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-05 07:55:17,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-05 07:55:17,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-05 07:55:17,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-05 07:55:17,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-05 07:55:17,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-05 07:55:17,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-05 07:55:17,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-05 07:55:17,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-05 07:55:17,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-05 07:55:17,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-05 07:55:17,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-05 07:55:17,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-05 07:55:17,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-05 07:55:17,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-05 07:55:17,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-05 07:55:17,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-05 07:55:17,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-05 07:55:17,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-05 07:55:17,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-05 07:55:17,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-09-05 07:55:17,302 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-05 07:55:17,302 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1104463987:90 , srv1122584111:90 , srv1127749294:90 , srv1136497384:90 , srv1200790810:90 , srv1310963459:90 , srv1496520389:90 , srv1559191718:90 , srv1669884102:90 , srv1672352032:90 , srv1695082678:90 , srv180341325:90 , srv1826587707:90 , srv1866324747:90 , srv1938550277:90 , srv2034407412:90 , srv229691786:90 , srv231509650:90 , srv379090905:90 , srv650834597:90 , srv658259701:90 , srv675714733:90 , srv753819532:90 , srv823453383:90 , srv849834882:90 , srv871721012:90 , srv894454316:90 , srv925477736:90 , srv982119283:90 , srv992417611:90 }
2023-09-05 07:55:17,321 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=202 (was 343), ProcessCount=173 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=9232 (was 9109) - AvailableMemoryMB LEAK? -