2023-10-29 16:56:48,515 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-10-29 16:56:48,745 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-10-29 16:56:48,831 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-10-29 16:56:48,883 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-10-29 16:56:48,889 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-10-29 16:56:48,899 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=199, ProcessCount=167, AvailableMemoryMB=3713
2023-10-29 16:56:48,901 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-10-29 16:56:48,901 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-10-29 16:56:49,013 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1025268702:84 , srv1095418944:84 , srv1189766933:84 , srv1221527917:84 , srv1233897777:84 , srv1236697272:84 , srv1241295901:84 , srv1242033227:84 , srv1242257397:84 , srv1307764452:84 , srv1310593570:84 , srv1320055458:84 , srv135259914:84 , srv1413695696:84 , srv1428734975:84 , srv1432392259:84 , srv1451830400:84 , srv1456883687:84 , srv1471623976:84 , srv1548333681:84 , srv1548376490:84 , srv1551225241:84 , srv1567681071:84 , srv156834643:84 , srv1583028129:84 , srv1594650827:84 , srv1651719847:84 , srv1655679930:84 , srv1657574578:84 , srv1663715326:84 , srv1712654658:84 , srv1714174684:84 , srv1725813674:84 , srv1797619806:84 , srv1820558871:84 , srv1821744337:84 , srv1828794778:84 , srv1838672580:84 , srv184524761:84 , srv1859768577:84 , srv1863124635:84 , srv1869121598:84 , srv1871465644:84 , srv1879996812:84 , srv1883738373:84 , srv1883748492:84 , srv1896972956:84 , srv189962989:84 , srv1939078858:84 , srv1940753930:84 , srv1964812364:84 , srv1993945947:84 , srv200233494:84 , srv2002508388:84 , srv2003765788:84 , srv2009973658:84 , srv2011918826:84 , srv2032365925:84 , srv203274528:84 , srv207008887:84 , srv2082846104:84 , srv2092767495:84 , srv2114147894:84 , srv2129354068:84 , srv225015921:84 , srv257029627:84 , srv274987847:84 , srv27690449:84 , srv287601486:84 , srv335851010:84 , srv344013959:84 , srv3543161:84 , srv36843099:84 , srv410123882:84 , srv423546651:84 , srv439016998:84 , srv470427668:84 , srv476978945:84 , srv498990239:84 , srv510080498:84 , srv551365263:84 , srv557080841:84 , srv582756833:84 , srv638757071:84 , srv650044590:84 , srv674616612:84 , srv678036496:84 , srv680369267:84 , srv706728199:84 , srv716751973:84 , srv75600221:84 , srv760690288:84 , srv833223800:84 , srv859295191:84 , srv893852567:84 , srv89584074:84 , srv925428825:84 , srv948163969:84 , srv970354070:84 , srv1491357564:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-10-29 16:56:49,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-29 16:56:49,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2129354068=64, srv1413695696=13, srv1451830400=16, srv925428825=97, srv470427668=77, srv638757071=84, srv557080841=82, srv760690288=92, srv2032365925=58, srv2003765788=55, srv1236697272=5, srv1797619806=34, srv1095418944=1, srv510080498=80, srv1307764452=9, srv75600221=91, srv1320055458=11, srv2011918826=57, srv650044590=85, srv1548376490=21, srv1883738373=45, srv203274528=59, srv2009973658=56, srv3543161=72, srv1993945947=52, srv1838672580=38, srv184524761=39, srv1863124635=41, srv674616612=86, srv200233494=53, srv678036496=87, srv1869121598=42, srv1583028129=25, srv1491357564=19, srv1551225241=22, srv498990239=79, srv1594650827=26, srv706728199=89, srv1428734975=14, srv2092767495=62, srv948163969=98, srv135259914=12, srv893852567=95, srv1663715326=30, srv1548333681=20, srv1725813674=33, srv1828794778=37, srv1896972956=47, srv1651719847=27, srv287601486=69, srv27690449=68, srv1025268702=0, srv1940753930=50, srv274987847=67, srv1820558871=35, srv1310593570=10, srv1821744337=36, srv410123882=74, srv1456883687=17, srv1964812364=51, srv439016998=76, srv476978945=78, srv1189766933=2, srv2002508388=54, srv970354070=99, srv2082846104=61, srv859295191=94, srv1241295901=6, srv89584074=96, srv1233897777=4, srv1432392259=15, srv1871465644=43, srv1567681071=23, srv225015921=65, srv1939078858=49, srv423546651=75, srv833223800=93, srv36843099=73, srv335851010=70, srv551365263=81, srv1859768577=40, srv1657574578=29, srv1242257397=8, srv156834643=24, srv1655679930=28, srv1879996812=44, srv1883748492=46, srv207008887=60, srv257029627=66, srv1221527917=3, srv582756833=83, srv1714174684=32, srv716751973=90, srv680369267=88, srv1242033227=7, srv1471623976=18, srv189962989=48, srv344013959=71, srv1712654658=31, srv2114147894=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-29 16:56:49,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-29 16:56:49,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-29 16:56:49,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-29 16:56:49,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-29 16:56:49,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-29 16:56:49,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-29 16:56:49,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-29 16:56:49,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-29 16:56:49,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-29 16:56:49,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-29 16:56:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-29 16:56:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-29 16:56:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-29 16:56:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-29 16:56:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-29 16:56:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-29 16:56:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-29 16:56:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-29 16:56:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-29 16:56:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-29 16:56:49,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-29 16:56:49,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-29 16:56:49,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-29 16:56:49,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-29 16:56:49,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-29 16:56:49,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-29 16:56:49,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-29 16:56:49,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-29 16:56:49,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-29 16:56:49,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-29 16:56:49,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-29 16:56:49,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-29 16:56:49,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-29 16:56:49,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-29 16:56:49,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-29 16:56:49,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-29 16:56:49,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-29 16:56:49,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-29 16:56:49,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-29 16:56:49,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-29 16:56:49,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-29 16:56:49,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-29 16:56:49,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-29 16:56:49,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-29 16:56:49,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-29 16:56:49,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-29 16:56:49,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-29 16:56:49,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-29 16:56:49,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-29 16:56:49,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-29 16:56:49,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-29 16:56:49,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-29 16:56:49,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-29 16:56:49,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-29 16:56:49,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-29 16:56:49,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-29 16:56:49,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-29 16:56:49,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-29 16:56:49,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-29 16:56:49,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-29 16:56:49,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-29 16:56:49,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-29 16:56:49,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-29 16:56:49,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-29 16:56:49,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-29 16:56:49,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-29 16:56:49,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-29 16:56:49,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-29 16:56:49,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-29 16:56:49,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-29 16:56:49,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-29 16:56:49,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-29 16:56:49,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-29 16:56:49,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-29 16:56:49,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-29 16:56:49,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-29 16:56:49,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-29 16:56:49,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-29 16:56:49,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-29 16:56:49,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-29 16:56:49,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-29 16:56:49,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-29 16:56:49,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-29 16:56:49,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-29 16:56:49,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-29 16:56:49,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-29 16:56:49,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-29 16:56:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-29 16:56:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-29 16:56:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-29 16:56:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-29 16:56:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-29 16:56:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-29 16:56:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-29 16:56:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-29 16:56:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-29 16:56:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-29 16:56:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-29 16:56:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-29 16:56:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-29 16:56:49,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-29 16:56:49,347 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-29 16:56:49,347 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-29 16:56:49,347 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-29 16:56:49,595 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 479 ms to try 5000 different iterations.  Found a solution that moves 4686 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1816645459645713. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04217079980453458, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06009345508919151, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5206666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04217079980453458, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17666666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24558333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-29 16:56:49,605 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1567681071:81 , srv1456883687:82 , srv1310593570:83 , srv1428734975:83 , srv1548333681:83 , srv1655679930:83 , srv189962989:83 , srv200233494:83 , srv207008887:83 , srv439016998:83 , srv1221527917:84 , srv1242257397:84 , srv1583028129:84 , srv1714174684:84 , srv1797619806:84 , srv1939078858:84 , srv2009973658:84 , srv335851010:84 , srv557080841:84 , srv760690288:84 , srv893852567:84 , srv1025268702:85 , srv1095418944:85 , srv1189766933:85 , srv1241295901:85 , srv1307764452:85 , srv1471623976:85 , srv156834643:85 , srv1651719847:85 , srv1657574578:85 , srv1821744337:85 , srv1838672580:85 , srv184524761:85 , srv1896972956:85 , srv2003765788:85 , srv2092767495:85 , srv225015921:85 , srv274987847:85 , srv287601486:85 , srv476978945:85 , srv678036496:85 , srv716751973:85 , srv1320055458:86 , srv1451830400:86 , srv1725813674:86 , srv1828794778:86 , srv1869121598:86 , srv1871465644:86 , srv1940753930:86 , srv2011918826:86 , srv203274528:86 , srv27690449:86 , srv410123882:86 , srv551365263:86 , srv680369267:86 , srv75600221:86 , srv89584074:86 , srv948163969:86 , srv1233897777:87 , srv1236697272:87 , srv1242033227:87 , srv135259914:87 , srv1432392259:87 , srv1548376490:87 , srv1551225241:87 , srv1663715326:87 , srv1859768577:87 , srv1883738373:87 , srv1964812364:87 , srv2002508388:87 , srv2032365925:87 , srv344013959:87 , srv423546651:87 , srv498990239:87 , srv925428825:87 , srv970354070:87 , srv1413695696:88 , srv1594650827:88 , srv1883748492:88 , srv2082846104:88 , srv3543161:88 , srv36843099:88 , srv638757071:88 , srv650044590:88 , srv706728199:88 , srv833223800:88 , srv859295191:88 , srv1712654658:89 , srv2114147894:89 , srv470427668:89 , srv510080498:89 , srv1879996812:90 , srv257029627:90 , srv582756833:90 , srv1820558871:91 , srv674616612:91 , srv1863124635:92 , srv1993945947:92 , srv2129354068:97 , srv1491357564:465 }
2023-10-29 16:56:49,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-29 16:56:49,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2129354068=64, srv1413695696=13, srv1451830400=16, srv925428825=97, srv470427668=77, srv638757071=84, srv557080841=82, srv760690288=92, srv2032365925=58, srv2003765788=55, srv1236697272=5, srv1797619806=34, srv1095418944=1, srv510080498=80, srv1307764452=9, srv75600221=91, srv1320055458=11, srv2011918826=57, srv650044590=85, srv1548376490=21, srv1883738373=45, srv203274528=59, srv2009973658=56, srv3543161=72, srv1993945947=52, srv1838672580=38, srv184524761=39, srv1863124635=41, srv674616612=86, srv200233494=53, srv678036496=87, srv1869121598=42, srv1583028129=25, srv1491357564=19, srv1551225241=22, srv498990239=79, srv1594650827=26, srv706728199=89, srv1428734975=14, srv2092767495=62, srv948163969=98, srv135259914=12, srv893852567=95, srv1663715326=30, srv1548333681=20, srv1725813674=33, srv1828794778=37, srv1896972956=47, srv1651719847=27, srv287601486=69, srv27690449=68, srv1025268702=0, srv1940753930=50, srv274987847=67, srv1820558871=35, srv1310593570=10, srv1821744337=36, srv410123882=74, srv1456883687=17, srv1964812364=51, srv439016998=76, srv476978945=78, srv1189766933=2, srv2002508388=54, srv970354070=99, srv2082846104=61, srv859295191=94, srv1241295901=6, srv89584074=96, srv1233897777=4, srv1432392259=15, srv1871465644=43, srv1567681071=23, srv225015921=65, srv1939078858=49, srv423546651=75, srv833223800=93, srv36843099=73, srv335851010=70, srv551365263=81, srv1859768577=40, srv1657574578=29, srv1242257397=8, srv156834643=24, srv1655679930=28, srv1879996812=44, srv1883748492=46, srv207008887=60, srv257029627=66, srv1221527917=3, srv582756833=83, srv1714174684=32, srv716751973=90, srv680369267=88, srv1242033227=7, srv1471623976=18, srv189962989=48, srv344013959=71, srv1712654658=31, srv2114147894=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-29 16:56:49,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-29 16:56:49,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-29 16:56:49,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-29 16:56:49,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-29 16:56:49,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-29 16:56:49,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-29 16:56:49,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-29 16:56:49,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-29 16:56:49,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-29 16:56:49,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-29 16:56:49,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-29 16:56:49,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-29 16:56:49,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-29 16:56:49,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-29 16:56:49,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-29 16:56:49,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-29 16:56:49,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-29 16:56:49,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-29 16:56:49,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-29 16:56:49,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-29 16:56:49,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-29 16:56:49,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-29 16:56:49,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-29 16:56:49,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-29 16:56:49,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-29 16:56:49,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-29 16:56:49,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-29 16:56:49,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-29 16:56:49,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-29 16:56:49,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-29 16:56:49,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-29 16:56:49,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-29 16:56:49,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-29 16:56:49,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-29 16:56:49,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-29 16:56:49,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-29 16:56:49,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-29 16:56:49,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-29 16:56:49,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-29 16:56:49,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-29 16:56:49,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-29 16:56:49,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-29 16:56:49,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-29 16:56:49,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-29 16:56:49,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-29 16:56:49,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-29 16:56:49,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-29 16:56:49,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-29 16:56:49,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-29 16:56:49,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-29 16:56:49,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-29 16:56:49,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-29 16:56:49,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-29 16:56:49,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-29 16:56:49,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-29 16:56:49,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-29 16:56:49,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-29 16:56:49,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-29 16:56:49,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-29 16:56:49,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-29 16:56:49,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-29 16:56:49,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-29 16:56:49,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-29 16:56:49,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-29 16:56:49,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-29 16:56:49,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-29 16:56:49,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-29 16:56:49,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-29 16:56:49,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-29 16:56:49,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-29 16:56:49,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-29 16:56:49,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-29 16:56:49,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-29 16:56:49,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-29 16:56:49,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-29 16:56:49,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-29 16:56:49,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-29 16:56:49,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-29 16:56:49,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-29 16:56:49,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-29 16:56:49,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-29 16:56:49,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-29 16:56:49,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-29 16:56:49,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-29 16:56:49,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-29 16:56:49,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-29 16:56:49,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-29 16:56:49,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-29 16:56:49,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-29 16:56:49,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-29 16:56:49,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-29 16:56:49,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-29 16:56:49,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-29 16:56:49,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-29 16:56:49,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-29 16:56:49,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-29 16:56:49,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-29 16:56:49,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-29 16:56:49,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-29 16:56:49,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-29 16:56:49,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-29 16:56:49,735 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04217079980453458, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06009345508919151, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04217079980453458, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17666666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24558333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-29 16:56:49,735 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-29 16:56:49,736 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1816317338912648, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04217079980453458, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06009345508919151, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04217079980453458, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17666666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24558333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-29 16:56:49,910 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 266 ms to try 5000 different iterations.  Found a solution that moves 2683 regions; Going from a computed imbalance of 0.1816317338912648 to a new imbalance of 0.016080705683503383. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02485774466357158); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03995508140076632, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2981111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02485774466357158); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04591666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-29 16:56:49,920 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2129354068:73 , srv1863124635:79 , srv1820558871:80 , srv470427668:80 , srv1879996812:81 , srv582756833:81 , srv706728199:81 , srv859295191:81 , srv1712654658:82 , srv257029627:82 , srv674616612:82 , srv833223800:82 , srv1551225241:83 , srv1594650827:83 , srv2114147894:83 , srv423546651:83 , srv680369267:83 , srv1236697272:84 , srv1869121598:84 , srv1883738373:84 , srv1993945947:84 , srv2002508388:84 , srv203274528:84 , srv344013959:84 , srv650044590:84 , srv678036496:84 , srv1025268702:85 , srv1241295901:85 , srv1242033227:85 , srv135259914:85 , srv1413695696:85 , srv1432392259:85 , srv1548376490:85 , srv156834643:85 , srv1663715326:85 , srv1838672580:85 , srv184524761:85 , srv1871465644:85 , srv1883748492:85 , srv1964812364:85 , srv2003765788:85 , srv2082846104:85 , srv2092767495:85 , srv3543161:85 , srv36843099:85 , srv476978945:85 , srv970354070:85 , srv1189766933:86 , srv1233897777:86 , srv1307764452:86 , srv1651719847:86 , srv1859768577:86 , srv1939078858:86 , srv27690449:86 , srv287601486:86 , srv498990239:86 , srv510080498:86 , srv551365263:86 , srv638757071:86 , srv716751973:86 , srv75600221:86 , srv89584074:86 , srv925428825:86 , srv1221527917:87 , srv1242257397:87 , srv1320055458:87 , srv1428734975:87 , srv1451830400:87 , srv1471623976:87 , srv1548333681:87 , srv1725813674:87 , srv1828794778:87 , srv1896972956:87 , srv2011918826:87 , srv2032365925:87 , srv225015921:87 , srv948163969:87 , srv1095418944:88 , srv1583028129:88 , srv1655679930:88 , srv1657574578:88 , srv1714174684:88 , srv1821744337:88 , srv1940753930:88 , srv2009973658:88 , srv274987847:88 , srv335851010:88 , srv410123882:88 , srv557080841:88 , srv893852567:88 , srv1310593570:89 , srv189962989:89 , srv207008887:89 , srv760690288:89 , srv1567681071:90 , srv1797619806:90 , srv439016998:91 , srv200233494:92 , srv1456883687:93 , srv1491357564:530 }
2023-10-29 16:56:49,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-29 16:56:49,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2129354068=64, srv1413695696=13, srv1451830400=16, srv925428825=97, srv470427668=77, srv638757071=84, srv557080841=82, srv760690288=92, srv2032365925=58, srv2003765788=55, srv1236697272=5, srv1797619806=34, srv1095418944=1, srv510080498=80, srv1307764452=9, srv75600221=91, srv1320055458=11, srv2011918826=57, srv650044590=85, srv1548376490=21, srv1883738373=45, srv203274528=59, srv2009973658=56, srv3543161=72, srv1993945947=52, srv1838672580=38, srv184524761=39, srv1863124635=41, srv674616612=86, srv200233494=53, srv678036496=87, srv1869121598=42, srv1583028129=25, srv1491357564=19, srv1551225241=22, srv498990239=79, srv1594650827=26, srv706728199=89, srv1428734975=14, srv2092767495=62, srv948163969=98, srv135259914=12, srv893852567=95, srv1663715326=30, srv1548333681=20, srv1725813674=33, srv1828794778=37, srv1896972956=47, srv1651719847=27, srv287601486=69, srv27690449=68, srv1025268702=0, srv1940753930=50, srv274987847=67, srv1820558871=35, srv1310593570=10, srv1821744337=36, srv410123882=74, srv1456883687=17, srv1964812364=51, srv439016998=76, srv476978945=78, srv1189766933=2, srv2002508388=54, srv970354070=99, srv2082846104=61, srv859295191=94, srv1241295901=6, srv89584074=96, srv1233897777=4, srv1432392259=15, srv1871465644=43, srv1567681071=23, srv225015921=65, srv1939078858=49, srv423546651=75, srv833223800=93, srv36843099=73, srv335851010=70, srv551365263=81, srv1859768577=40, srv1657574578=29, srv1242257397=8, srv156834643=24, srv1655679930=28, srv1879996812=44, srv1883748492=46, srv207008887=60, srv257029627=66, srv1221527917=3, srv582756833=83, srv1714174684=32, srv716751973=90, srv680369267=88, srv1242033227=7, srv1471623976=18, srv189962989=48, srv344013959=71, srv1712654658=31, srv2114147894=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-29 16:56:49,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-29 16:56:49,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-29 16:56:49,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-29 16:56:49,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-29 16:56:49,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-29 16:56:49,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-29 16:56:49,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-29 16:56:49,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-29 16:56:49,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-29 16:56:49,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-29 16:56:49,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-29 16:56:49,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-29 16:56:49,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-29 16:56:49,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-29 16:56:49,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-29 16:56:49,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-29 16:56:49,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-29 16:56:49,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-29 16:56:49,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-29 16:56:49,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-29 16:56:49,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-29 16:56:49,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-29 16:56:49,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-29 16:56:49,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-29 16:56:49,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-29 16:56:49,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-29 16:56:49,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-29 16:56:49,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-29 16:56:49,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-29 16:56:49,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-29 16:56:49,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-29 16:56:49,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-29 16:56:49,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-29 16:56:49,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-29 16:56:49,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-29 16:56:49,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-29 16:56:49,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-29 16:56:49,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-29 16:56:49,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-29 16:56:49,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-29 16:56:49,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-29 16:56:49,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-29 16:56:49,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-29 16:56:49,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-29 16:56:49,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-29 16:56:49,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-29 16:56:49,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-29 16:56:49,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-29 16:56:49,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-29 16:56:49,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-29 16:56:49,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-29 16:56:49,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-29 16:56:49,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-29 16:56:49,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-29 16:56:49,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-29 16:56:49,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-29 16:56:49,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-29 16:56:49,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-29 16:56:49,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-29 16:56:49,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-29 16:56:49,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-29 16:56:49,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-29 16:56:49,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-29 16:56:49,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-29 16:56:49,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-29 16:56:49,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-29 16:56:49,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-29 16:56:49,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-29 16:56:49,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-29 16:56:49,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-29 16:56:49,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-29 16:56:49,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-29 16:56:49,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-29 16:56:49,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-29 16:56:49,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-29 16:56:49,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-29 16:56:49,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-29 16:56:49,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-29 16:56:49,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-29 16:56:49,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-29 16:56:49,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-29 16:56:49,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-29 16:56:49,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-29 16:56:49,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-29 16:56:49,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-29 16:56:49,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-29 16:56:49,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-29 16:56:49,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-29 16:56:49,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-29 16:56:49,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-29 16:56:49,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-29 16:56:49,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-29 16:56:49,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-29 16:56:49,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-29 16:56:49,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-29 16:56:49,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-29 16:56:49,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-29 16:56:49,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-29 16:56:49,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-29 16:56:49,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-29 16:56:49,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-29 16:56:50,030 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02485774466357158); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03995508140076632, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02485774466357158); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04591666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-29 16:56:50,030 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-29 16:56:50,030 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01606191891596575, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02485774466357158); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03995508140076632, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02485774466357158); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04591666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-29 16:56:50,191 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 234 ms to try 5000 different iterations.  Found a solution that moves 1029 regions; Going from a computed imbalance of 0.01606191891596575 to a new imbalance of 5.524674335583239E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007094396742872501); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008541179053175637); MoveCostFunction : (multiplier=7.0, imbalance=0.11433333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007094396742872501); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-29 16:56:50,194 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1993945947:82 , srv200233494:82 , srv510080498:82 , srv1456883687:83 , srv1797619806:83 , srv410123882:83 , srv439016998:83 , srv1320055458:84 , srv1413695696:84 , srv1451830400:84 , srv1663715326:84 , srv1828794778:84 , srv1883748492:84 , srv1940753930:84 , srv2032365925:84 , srv274987847:84 , srv3543161:84 , srv36843099:84 , srv498990239:84 , srv638757071:84 , srv674616612:84 , srv760690288:84 , srv948163969:84 , srv1095418944:85 , srv1233897777:85 , srv1310593570:85 , srv1432392259:85 , srv1548376490:85 , srv1567681071:85 , srv1657574578:85 , srv1714174684:85 , srv1725813674:85 , srv1821744337:85 , srv1859768577:85 , srv1896972956:85 , srv189962989:85 , srv2009973658:85 , srv2011918826:85 , srv2082846104:85 , srv225015921:85 , srv27690449:85 , srv335851010:85 , srv557080841:85 , srv893852567:85 , srv89584074:85 , srv925428825:85 , srv970354070:85 , srv1221527917:86 , srv1242033227:86 , srv1242257397:86 , srv1307764452:86 , srv135259914:86 , srv1471623976:86 , srv1583028129:86 , srv1594650827:86 , srv1651719847:86 , srv1655679930:86 , srv1712654658:86 , srv1820558871:86 , srv1863124635:86 , srv1871465644:86 , srv1964812364:86 , srv2002508388:86 , srv207008887:86 , srv2114147894:86 , srv257029627:86 , srv287601486:86 , srv344013959:86 , srv551365263:86 , srv582756833:86 , srv650044590:86 , srv716751973:86 , srv75600221:86 , srv1189766933:87 , srv1236697272:87 , srv1548333681:87 , srv1551225241:87 , srv1838672580:87 , srv184524761:87 , srv1869121598:87 , srv1879996812:87 , srv1883738373:87 , srv1939078858:87 , srv2003765788:87 , srv203274528:87 , srv2092767495:87 , srv476978945:87 , srv1025268702:88 , srv1241295901:88 , srv1428734975:88 , srv156834643:88 , srv2129354068:88 , srv423546651:88 , srv678036496:88 , srv706728199:88 , srv833223800:88 , srv859295191:88 , srv470427668:89 , srv680369267:89 , srv1491357564:526 }
2023-10-29 16:56:50,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-29 16:56:50,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2129354068=64, srv1413695696=13, srv1451830400=16, srv925428825=97, srv470427668=77, srv638757071=84, srv557080841=82, srv760690288=92, srv2032365925=58, srv2003765788=55, srv1236697272=5, srv1797619806=34, srv1095418944=1, srv510080498=80, srv1307764452=9, srv75600221=91, srv1320055458=11, srv2011918826=57, srv650044590=85, srv1548376490=21, srv1883738373=45, srv203274528=59, srv2009973658=56, srv3543161=72, srv1993945947=52, srv1838672580=38, srv184524761=39, srv1863124635=41, srv674616612=86, srv200233494=53, srv678036496=87, srv1869121598=42, srv1583028129=25, srv1491357564=19, srv1551225241=22, srv498990239=79, srv1594650827=26, srv706728199=89, srv1428734975=14, srv2092767495=62, srv948163969=98, srv135259914=12, srv893852567=95, srv1663715326=30, srv1548333681=20, srv1725813674=33, srv1828794778=37, srv1896972956=47, srv1651719847=27, srv287601486=69, srv27690449=68, srv1025268702=0, srv1940753930=50, srv274987847=67, srv1820558871=35, srv1310593570=10, srv1821744337=36, srv410123882=74, srv1456883687=17, srv1964812364=51, srv439016998=76, srv476978945=78, srv1189766933=2, srv2002508388=54, srv970354070=99, srv2082846104=61, srv859295191=94, srv1241295901=6, srv89584074=96, srv1233897777=4, srv1432392259=15, srv1871465644=43, srv1567681071=23, srv225015921=65, srv1939078858=49, srv423546651=75, srv833223800=93, srv36843099=73, srv335851010=70, srv551365263=81, srv1859768577=40, srv1657574578=29, srv1242257397=8, srv156834643=24, srv1655679930=28, srv1879996812=44, srv1883748492=46, srv207008887=60, srv257029627=66, srv1221527917=3, srv582756833=83, srv1714174684=32, srv716751973=90, srv680369267=88, srv1242033227=7, srv1471623976=18, srv189962989=48, srv344013959=71, srv1712654658=31, srv2114147894=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-29 16:56:50,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-29 16:56:50,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-29 16:56:50,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-29 16:56:50,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-29 16:56:50,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-29 16:56:50,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-29 16:56:50,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-29 16:56:50,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-29 16:56:50,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-29 16:56:50,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-29 16:56:50,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-29 16:56:50,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-29 16:56:50,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-29 16:56:50,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-29 16:56:50,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-29 16:56:50,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-29 16:56:50,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-29 16:56:50,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-29 16:56:50,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-29 16:56:50,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-29 16:56:50,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-29 16:56:50,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-29 16:56:50,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-29 16:56:50,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-29 16:56:50,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-29 16:56:50,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-29 16:56:50,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-29 16:56:50,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-29 16:56:50,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-29 16:56:50,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-29 16:56:50,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-29 16:56:50,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-29 16:56:50,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-29 16:56:50,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-29 16:56:50,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-29 16:56:50,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-29 16:56:50,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-29 16:56:50,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-29 16:56:50,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-29 16:56:50,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-29 16:56:50,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-29 16:56:50,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-29 16:56:50,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-29 16:56:50,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-29 16:56:50,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-29 16:56:50,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-29 16:56:50,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-29 16:56:50,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-29 16:56:50,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-29 16:56:50,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-29 16:56:50,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-29 16:56:50,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-29 16:56:50,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-29 16:56:50,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-29 16:56:50,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-29 16:56:50,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-29 16:56:50,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-29 16:56:50,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-29 16:56:50,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-29 16:56:50,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-29 16:56:50,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-29 16:56:50,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-29 16:56:50,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-29 16:56:50,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-29 16:56:50,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-29 16:56:50,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-29 16:56:50,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-29 16:56:50,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-29 16:56:50,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-29 16:56:50,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-29 16:56:50,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-29 16:56:50,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-29 16:56:50,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-29 16:56:50,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-29 16:56:50,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-29 16:56:50,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-29 16:56:50,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-29 16:56:50,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-29 16:56:50,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-29 16:56:50,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-29 16:56:50,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-29 16:56:50,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-29 16:56:50,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-29 16:56:50,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-29 16:56:50,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-29 16:56:50,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-29 16:56:50,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-29 16:56:50,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-29 16:56:50,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-29 16:56:50,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-29 16:56:50,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-29 16:56:50,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-29 16:56:50,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-29 16:56:50,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-29 16:56:50,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-29 16:56:50,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-29 16:56:50,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-29 16:56:50,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-29 16:56:50,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-29 16:56:50,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-29 16:56:50,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-29 16:56:50,299 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007094396742872501); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008541179053175637); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007094396742872501); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-29 16:56:50,299 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-29 16:56:50,299 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.452622215582399E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007094396742872501); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008541179053175637); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007094396742872501); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-29 16:56:50,522 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 299 ms to try 5000 different iterations.  Found a solution that moves 268 regions; Going from a computed imbalance of 5.452622215582399E-4 to a new imbalance of 2.188871466331757E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.029777777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-29 16:56:50,523 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv423546651:83 , srv1025268702:84 , srv1095418944:84 , srv1189766933:84 , srv1221527917:84 , srv1236697272:84 , srv1241295901:84 , srv1242033227:84 , srv1320055458:84 , srv135259914:84 , srv1428734975:84 , srv1456883687:84 , srv1471623976:84 , srv1548376490:84 , srv156834643:84 , srv1583028129:84 , srv1657574578:84 , srv1712654658:84 , srv1725813674:84 , srv1821744337:84 , srv184524761:84 , srv1879996812:84 , srv1883738373:84 , srv1940753930:84 , srv1964812364:84 , srv207008887:84 , srv2092767495:84 , srv2114147894:84 , srv2129354068:84 , srv257029627:84 , srv274987847:84 , srv470427668:84 , srv498990239:84 , srv551365263:84 , srv650044590:84 , srv680369267:84 , srv75600221:84 , srv833223800:84 , srv1233897777:85 , srv1242257397:85 , srv1307764452:85 , srv1310593570:85 , srv1413695696:85 , srv1432392259:85 , srv1451830400:85 , srv1548333681:85 , srv1551225241:85 , srv1594650827:85 , srv1651719847:85 , srv1655679930:85 , srv1714174684:85 , srv1797619806:85 , srv1820558871:85 , srv1828794778:85 , srv1838672580:85 , srv1859768577:85 , srv1863124635:85 , srv1869121598:85 , srv1871465644:85 , srv1883748492:85 , srv1896972956:85 , srv189962989:85 , srv1939078858:85 , srv1993945947:85 , srv2002508388:85 , srv2003765788:85 , srv2009973658:85 , srv2011918826:85 , srv2032365925:85 , srv203274528:85 , srv2082846104:85 , srv225015921:85 , srv27690449:85 , srv287601486:85 , srv335851010:85 , srv344013959:85 , srv3543161:85 , srv36843099:85 , srv410123882:85 , srv439016998:85 , srv476978945:85 , srv510080498:85 , srv557080841:85 , srv582756833:85 , srv638757071:85 , srv674616612:85 , srv678036496:85 , srv706728199:85 , srv716751973:85 , srv760690288:85 , srv859295191:85 , srv893852567:85 , srv89584074:85 , srv925428825:85 , srv948163969:85 , srv970354070:85 , srv1567681071:86 , srv1663715326:86 , srv200233494:86 , srv1491357564:621 }
2023-10-29 16:56:50,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-29 16:56:50,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2129354068=64, srv1413695696=13, srv1451830400=16, srv925428825=97, srv470427668=77, srv638757071=84, srv557080841=82, srv760690288=92, srv2032365925=58, srv2003765788=55, srv1236697272=5, srv1797619806=34, srv1095418944=1, srv510080498=80, srv1307764452=9, srv75600221=91, srv1320055458=11, srv2011918826=57, srv650044590=85, srv1548376490=21, srv1883738373=45, srv203274528=59, srv2009973658=56, srv3543161=72, srv1993945947=52, srv1838672580=38, srv184524761=39, srv1863124635=41, srv674616612=86, srv200233494=53, srv678036496=87, srv1869121598=42, srv1583028129=25, srv1491357564=19, srv1551225241=22, srv498990239=79, srv1594650827=26, srv706728199=89, srv1428734975=14, srv2092767495=62, srv948163969=98, srv135259914=12, srv893852567=95, srv1663715326=30, srv1548333681=20, srv1725813674=33, srv1828794778=37, srv1896972956=47, srv1651719847=27, srv287601486=69, srv27690449=68, srv1025268702=0, srv1940753930=50, srv274987847=67, srv1820558871=35, srv1310593570=10, srv1821744337=36, srv410123882=74, srv1456883687=17, srv1964812364=51, srv439016998=76, srv476978945=78, srv1189766933=2, srv2002508388=54, srv970354070=99, srv2082846104=61, srv859295191=94, srv1241295901=6, srv89584074=96, srv1233897777=4, srv1432392259=15, srv1871465644=43, srv1567681071=23, srv225015921=65, srv1939078858=49, srv423546651=75, srv833223800=93, srv36843099=73, srv335851010=70, srv551365263=81, srv1859768577=40, srv1657574578=29, srv1242257397=8, srv156834643=24, srv1655679930=28, srv1879996812=44, srv1883748492=46, srv207008887=60, srv257029627=66, srv1221527917=3, srv582756833=83, srv1714174684=32, srv716751973=90, srv680369267=88, srv1242033227=7, srv1471623976=18, srv189962989=48, srv344013959=71, srv1712654658=31, srv2114147894=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-29 16:56:50,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-29 16:56:50,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-29 16:56:50,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-29 16:56:50,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-29 16:56:50,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-29 16:56:50,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-29 16:56:50,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-29 16:56:50,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-29 16:56:50,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-29 16:56:50,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-29 16:56:50,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-29 16:56:50,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-29 16:56:50,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-29 16:56:50,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-29 16:56:50,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-29 16:56:50,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-29 16:56:50,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-29 16:56:50,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-29 16:56:50,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-29 16:56:50,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-29 16:56:50,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-29 16:56:50,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-29 16:56:50,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-29 16:56:50,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-29 16:56:50,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-29 16:56:50,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-29 16:56:50,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-29 16:56:50,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-29 16:56:50,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-29 16:56:50,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-29 16:56:50,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-29 16:56:50,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-29 16:56:50,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-29 16:56:50,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-29 16:56:50,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-29 16:56:50,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-29 16:56:50,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-29 16:56:50,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-29 16:56:50,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-29 16:56:50,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-29 16:56:50,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-29 16:56:50,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-29 16:56:50,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-29 16:56:50,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-29 16:56:50,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-29 16:56:50,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-29 16:56:50,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-29 16:56:50,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-29 16:56:50,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-29 16:56:50,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-29 16:56:50,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-29 16:56:50,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-29 16:56:50,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-29 16:56:50,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-29 16:56:50,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-29 16:56:50,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-29 16:56:50,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-29 16:56:50,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-29 16:56:50,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-29 16:56:50,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-29 16:56:50,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-29 16:56:50,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-29 16:56:50,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-29 16:56:50,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-29 16:56:50,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-29 16:56:50,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-29 16:56:50,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-29 16:56:50,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-29 16:56:50,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-29 16:56:50,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-29 16:56:50,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-29 16:56:50,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-29 16:56:50,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-29 16:56:50,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-29 16:56:50,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-29 16:56:50,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-29 16:56:50,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-29 16:56:50,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-29 16:56:50,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-29 16:56:50,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-29 16:56:50,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-29 16:56:50,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-29 16:56:50,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-29 16:56:50,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-29 16:56:50,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-29 16:56:50,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-29 16:56:50,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-29 16:56:50,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-29 16:56:50,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-29 16:56:50,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-29 16:56:50,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-29 16:56:50,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-29 16:56:50,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-29 16:56:50,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-29 16:56:50,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-29 16:56:50,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-29 16:56:50,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-29 16:56:50,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-29 16:56:50,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-29 16:56:50,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-29 16:56:50,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-29 16:56:50,632 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=2.001213855445215E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-29 16:56:50,632 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv423546651:83 , srv1025268702:84 , srv1095418944:84 , srv1189766933:84 , srv1221527917:84 , srv1236697272:84 , srv1241295901:84 , srv1242033227:84 , srv1320055458:84 , srv135259914:84 , srv1428734975:84 , srv1456883687:84 , srv1471623976:84 , srv1548376490:84 , srv156834643:84 , srv1583028129:84 , srv1657574578:84 , srv1712654658:84 , srv1725813674:84 , srv1821744337:84 , srv184524761:84 , srv1879996812:84 , srv1883738373:84 , srv1940753930:84 , srv1964812364:84 , srv207008887:84 , srv2092767495:84 , srv2114147894:84 , srv2129354068:84 , srv257029627:84 , srv274987847:84 , srv470427668:84 , srv498990239:84 , srv551365263:84 , srv650044590:84 , srv680369267:84 , srv75600221:84 , srv833223800:84 , srv1233897777:85 , srv1242257397:85 , srv1307764452:85 , srv1310593570:85 , srv1413695696:85 , srv1432392259:85 , srv1451830400:85 , srv1548333681:85 , srv1551225241:85 , srv1594650827:85 , srv1651719847:85 , srv1655679930:85 , srv1714174684:85 , srv1797619806:85 , srv1820558871:85 , srv1828794778:85 , srv1838672580:85 , srv1859768577:85 , srv1863124635:85 , srv1869121598:85 , srv1871465644:85 , srv1883748492:85 , srv1896972956:85 , srv189962989:85 , srv1939078858:85 , srv1993945947:85 , srv2002508388:85 , srv2003765788:85 , srv2009973658:85 , srv2011918826:85 , srv2032365925:85 , srv203274528:85 , srv2082846104:85 , srv225015921:85 , srv27690449:85 , srv287601486:85 , srv335851010:85 , srv344013959:85 , srv3543161:85 , srv36843099:85 , srv410123882:85 , srv439016998:85 , srv476978945:85 , srv510080498:85 , srv557080841:85 , srv582756833:85 , srv638757071:85 , srv674616612:85 , srv678036496:85 , srv706728199:85 , srv716751973:85 , srv760690288:85 , srv859295191:85 , srv893852567:85 , srv89584074:85 , srv925428825:85 , srv948163969:85 , srv970354070:85 , srv1567681071:86 , srv1663715326:86 , srv200233494:86 , srv1491357564:621 }
2023-10-29 16:56:50,655 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=199 (was 199), ProcessCount=169 (was 167) - ProcessCount LEAK? -, AvailableMemoryMB=3242 (was 3713)
2023-10-29 16:56:50,663 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=199, ProcessCount=169, AvailableMemoryMB=3241
2023-10-29 16:56:50,663 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-10-29 16:56:50,663 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-10-29 16:56:50,671 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1005605019:84 , srv1074815491:84 , srv1246109717:84 , srv1328708771:84 , srv1466997883:84 , srv1478735937:84 , srv1720064002:84 , srv175222469:84 , srv1765828303:84 , srv1768794722:84 , srv1768989755:84 , srv1875273747:84 , srv1975413442:84 , srv2008877082:84 , srv224699086:84 , srv256207353:84 , srv389448027:84 , srv400797264:84 , srv470788191:84 , srv53922989:84 , srv574892819:84 , srv632038951:84 , srv661654799:84 , srv688482926:84 , srv713652953:84 , srv727791860:84 , srv927686582:84 , srv929331631:84 , srv98478303:84 , srv1433859987:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-10-29 16:56:50,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-29 16:56:50,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv389448027=17, srv1246109717=2, srv1478735937=6, srv1768989755=11, srv224699086=15, srv1875273747=12, srv632038951=22, srv98478303=29, srv727791860=26, srv1328708771=3, srv1005605019=0, srv1768794722=10, srv470788191=19, srv661654799=23, srv1720064002=7, srv2008877082=14, srv688482926=24, srv53922989=20, srv927686582=27, srv1074815491=1, srv1975413442=13, srv929331631=28, srv1765828303=9, srv256207353=16, srv175222469=8, srv574892819=21, srv1433859987=4, srv713652953=25, srv400797264=18, srv1466997883=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-29 16:56:50,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-29 16:56:50,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-29 16:56:50,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-29 16:56:50,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-29 16:56:50,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-29 16:56:50,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-29 16:56:50,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-29 16:56:50,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-29 16:56:50,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-29 16:56:50,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-29 16:56:50,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-29 16:56:50,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-29 16:56:50,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-29 16:56:50,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-29 16:56:50,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-29 16:56:50,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-29 16:56:50,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-29 16:56:50,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-29 16:56:50,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-29 16:56:50,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-29 16:56:50,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-29 16:56:50,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-29 16:56:50,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-29 16:56:50,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-29 16:56:50,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-29 16:56:50,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-29 16:56:50,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-29 16:56:50,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-29 16:56:50,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-29 16:56:50,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-29 16:56:50,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-10-29 16:56:50,697 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0934639231003784, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-29 16:56:50,697 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912487545034092, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0934639231003784, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-10-29 16:58:50,684 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9119986 different iterations.  Found a solution that moves 1846 regions; Going from a computed imbalance of 0.9912487545034092 to a new imbalance of 4.308656090753194E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6837037037037037, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-29 16:58:50,691 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1005605019:90 , srv1074815491:90 , srv1246109717:90 , srv1328708771:90 , srv1433859987:90 , srv1466997883:90 , srv1478735937:90 , srv1720064002:90 , srv175222469:90 , srv1765828303:90 , srv1768794722:90 , srv1768989755:90 , srv1875273747:90 , srv1975413442:90 , srv2008877082:90 , srv224699086:90 , srv256207353:90 , srv389448027:90 , srv400797264:90 , srv470788191:90 , srv53922989:90 , srv574892819:90 , srv632038951:90 , srv661654799:90 , srv688482926:90 , srv713652953:90 , srv727791860:90 , srv927686582:90 , srv929331631:90 , srv98478303:90 }
2023-10-29 16:58:50,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-29 16:58:50,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv389448027=17, srv1246109717=2, srv1478735937=6, srv1768989755=11, srv224699086=15, srv1875273747=12, srv632038951=22, srv98478303=29, srv727791860=26, srv1328708771=3, srv1005605019=0, srv1768794722=10, srv470788191=19, srv661654799=23, srv1720064002=7, srv2008877082=14, srv688482926=24, srv53922989=20, srv927686582=27, srv1074815491=1, srv1975413442=13, srv929331631=28, srv1765828303=9, srv256207353=16, srv175222469=8, srv574892819=21, srv1433859987=4, srv713652953=25, srv400797264=18, srv1466997883=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-29 16:58:50,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-29 16:58:50,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-29 16:58:50,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-29 16:58:50,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-29 16:58:50,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-29 16:58:50,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-29 16:58:50,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-29 16:58:50,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-29 16:58:50,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-29 16:58:50,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-29 16:58:50,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-29 16:58:50,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-29 16:58:50,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-29 16:58:50,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-29 16:58:50,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-29 16:58:50,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-29 16:58:50,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-29 16:58:50,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-29 16:58:50,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-29 16:58:50,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-29 16:58:50,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-29 16:58:50,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-29 16:58:50,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-29 16:58:50,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-29 16:58:50,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-29 16:58:50,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-29 16:58:50,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-29 16:58:50,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-29 16:58:50,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-29 16:58:50,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-29 16:58:50,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-10-29 16:58:50,716 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-29 16:58:50,716 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1005605019:90 , srv1074815491:90 , srv1246109717:90 , srv1328708771:90 , srv1433859987:90 , srv1466997883:90 , srv1478735937:90 , srv1720064002:90 , srv175222469:90 , srv1765828303:90 , srv1768794722:90 , srv1768989755:90 , srv1875273747:90 , srv1975413442:90 , srv2008877082:90 , srv224699086:90 , srv256207353:90 , srv389448027:90 , srv400797264:90 , srv470788191:90 , srv53922989:90 , srv574892819:90 , srv632038951:90 , srv661654799:90 , srv688482926:90 , srv713652953:90 , srv727791860:90 , srv927686582:90 , srv929331631:90 , srv98478303:90 }
2023-10-29 16:58:50,729 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=240 (was 199) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 169), AvailableMemoryMB=3420 (was 3241) - AvailableMemoryMB LEAK? -