2023-08-12 15:53:07,289 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-08-12 15:53:07,604 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-08-12 15:53:07,716 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-08-12 15:53:07,784 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-08-12 15:53:07,791 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-08-12 15:53:07,804 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=298, ProcessCount=173, AvailableMemoryMB=22406
2023-08-12 15:53:07,806 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-08-12 15:53:07,806 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-08-12 15:53:07,965 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1036153927:84 , srv1042532612:84 , srv1044358957:84 , srv1064560104:84 , srv1089498800:84 , srv1150811298:84 , srv1183110483:84 , srv1189690258:84 , srv1208712267:84 , srv1210745328:84 , srv1222886863:84 , srv1240761548:84 , srv1256415057:84 , srv1266184604:84 , srv126839322:84 , srv1268956207:84 , srv1282330618:84 , srv130851095:84 , srv1308889564:84 , srv1318406716:84 , srv1327527695:84 , srv1358217068:84 , srv1373785382:84 , srv1399188200:84 , srv1468208660:84 , srv1469154361:84 , srv1588769050:84 , srv1601245979:84 , srv1611809699:84 , srv1631194627:84 , srv1654737074:84 , srv1659927426:84 , srv1674048010:84 , srv1709093535:84 , srv1717932978:84 , srv1736917852:84 , srv176705867:84 , srv1771092406:84 , srv1785270353:84 , srv1803400830:84 , srv1820495104:84 , srv1874576781:84 , srv1886883097:84 , srv1911681175:84 , srv1928153479:84 , srv1939074696:84 , srv1946291933:84 , srv1992787866:84 , srv2001779046:84 , srv2057196944:84 , srv2074098535:84 , srv2083734177:84 , srv210650271:84 , srv2112677457:84 , srv2126858794:84 , srv221551599:84 , srv228028721:84 , srv240824119:84 , srv245860647:84 , srv256884433:84 , srv289490941:84 , srv312637736:84 , srv368233274:84 , srv417388467:84 , srv425433026:84 , srv431094109:84 , srv435558945:84 , srv438807315:84 , srv451171557:84 , srv451741432:84 , srv453417307:84 , srv458742762:84 , srv472123417:84 , srv474049556:84 , srv501007149:84 , srv518779992:84 , srv530756191:84 , srv537386916:84 , srv575686866:84 , srv580984552:84 , srv599914082:84 , srv60834475:84 , srv641720926:84 , srv648251510:84 , srv652445159:84 , srv728575386:84 , srv752052393:84 , srv763035038:84 , srv866089667:84 , srv870007275:84 , srv872835888:84 , srv892897458:84 , srv913667369:84 , srv940958428:84 , srv946835167:84 , srv947620637:84 , srv969235016:84 , srv978010862:84 , srv980344294:84 , srv843230722:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-08-12 15:53:07,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-12 15:53:07,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv245860647=58, srv518779992=75, srv1042532612=1, srv1654737074=30, srv1631194627=29, srv940958428=94, srv1601245979=27, srv256884433=59, srv1911681175=43, srv60834475=81, srv866089667=89, srv1210745328=9, srv580984552=79, srv1183110483=6, srv530756191=76, srv1611809699=28, srv641720926=82, srv1282330618=16, srv425433026=64, srv752052393=86, srv126839322=14, srv980344294=99, srv1208712267=8, srv1256415057=12, srv1946291933=46, srv451741432=69, srv1939074696=45, srv2057196944=49, srv1820495104=40, srv1044358957=2, srv417388467=63, srv575686866=78, srv1468208660=24, srv438807315=67, srv2083734177=51, srv1717932978=34, srv1308889564=18, srv872835888=91, srv1089498800=4, srv1318406716=19, srv2126858794=54, srv892897458=92, srv1266184604=13, srv1189690258=7, srv1674048010=32, srv453417307=70, srv1064560104=3, srv451171557=68, srv1736917852=35, srv1240761548=11, srv946835167=95, srv2112677457=53, srv435558945=66, srv763035038=87, srv1588769050=26, srv1150811298=5, srv1874576781=41, srv652445159=84, srv1268956207=15, srv474049556=73, srv472123417=72, srv228028721=56, srv176705867=36, srv1036153927=0, srv1659927426=31, srv870007275=90, srv130851095=17, srv978010862=98, srv501007149=74, srv221551599=55, srv1992787866=47, srv368233274=62, srv1709093535=33, srv1771092406=37, srv458742762=71, srv2001779046=48, srv312637736=61, srv431094109=65, srv537386916=77, srv728575386=85, srv1785270353=38, srv1469154361=25, srv913667369=93, srv599914082=80, srv240824119=57, srv210650271=52, srv969235016=97, srv1399188200=23, srv1886883097=42, srv843230722=88, srv2074098535=50, srv947620637=96, srv1222886863=10, srv1928153479=44, srv289490941=60, srv1803400830=39, srv648251510=83, srv1373785382=22, srv1327527695=20, srv1358217068=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-12 15:53:08,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-12 15:53:08,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-12 15:53:08,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-12 15:53:08,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-12 15:53:08,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-12 15:53:08,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-12 15:53:08,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-12 15:53:08,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-12 15:53:08,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-12 15:53:08,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-12 15:53:08,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-12 15:53:08,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-12 15:53:08,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-12 15:53:08,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-12 15:53:08,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-12 15:53:08,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-12 15:53:08,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-12 15:53:08,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-12 15:53:08,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-12 15:53:08,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-12 15:53:08,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-12 15:53:08,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-12 15:53:08,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-12 15:53:08,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-12 15:53:08,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-12 15:53:08,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-12 15:53:08,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-12 15:53:08,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-12 15:53:08,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-12 15:53:08,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-12 15:53:08,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-12 15:53:08,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-12 15:53:08,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-12 15:53:08,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-12 15:53:08,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-12 15:53:08,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-12 15:53:08,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-12 15:53:08,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-12 15:53:08,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-12 15:53:08,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-12 15:53:08,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-12 15:53:08,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-12 15:53:08,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-12 15:53:08,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-12 15:53:08,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-12 15:53:08,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-12 15:53:08,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-12 15:53:08,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-12 15:53:08,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-12 15:53:08,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-12 15:53:08,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-12 15:53:08,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-12 15:53:08,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-12 15:53:08,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-12 15:53:08,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-12 15:53:08,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-12 15:53:08,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-12 15:53:08,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-12 15:53:08,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-12 15:53:08,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-12 15:53:08,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-12 15:53:08,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-12 15:53:08,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-12 15:53:08,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-12 15:53:08,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-12 15:53:08,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-12 15:53:08,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-12 15:53:08,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-12 15:53:08,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-12 15:53:08,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-12 15:53:08,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-12 15:53:08,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-12 15:53:08,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-12 15:53:08,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-12 15:53:08,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-12 15:53:08,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-12 15:53:08,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-12 15:53:08,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-12 15:53:08,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-12 15:53:08,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-12 15:53:08,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-12 15:53:08,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-12 15:53:08,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-12 15:53:08,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-12 15:53:08,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-12 15:53:08,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-12 15:53:08,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-12 15:53:08,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-12 15:53:08,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-12 15:53:08,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-12 15:53:08,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-12 15:53:08,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-12 15:53:08,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-12 15:53:08,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-12 15:53:08,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-12 15:53:08,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-12 15:53:08,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-12 15:53:08,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-12 15:53:08,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-12 15:53:08,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-12 15:53:08,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-12 15:53:08,405 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-12 15:53:08,406 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-08-12 15:53:08,406 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-08-12 15:53:08,821 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 707 ms to try 5000 different iterations.  Found a solution that moves 4671 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18291977462095074. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03959862218767201, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05672308540187511, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.519, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03959862218767201, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17833333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24316666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-12 15:53:08,833 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1674048010:82 , srv1089498800:83 , srv1256415057:83 , srv1318406716:83 , srv1631194627:83 , srv1659927426:83 , srv2074098535:83 , srv946835167:83 , srv1042532612:84 , srv126839322:84 , srv1588769050:84 , srv1611809699:84 , srv1736917852:84 , srv1803400830:84 , srv1911681175:84 , srv599914082:84 , srv641720926:84 , srv728575386:84 , srv866089667:84 , srv870007275:84 , srv1208712267:85 , srv1210745328:85 , srv1266184604:85 , srv1282330618:85 , srv130851095:85 , srv1327527695:85 , srv1358217068:85 , srv1373785382:85 , srv1717932978:85 , srv1785270353:85 , srv1874576781:85 , srv1928153479:85 , srv2001779046:85 , srv2083734177:85 , srv221551599:85 , srv256884433:85 , srv425433026:85 , srv752052393:85 , srv763035038:85 , srv980344294:85 , srv1183110483:86 , srv1189690258:86 , srv1222886863:86 , srv1601245979:86 , srv1886883097:86 , srv2057196944:86 , srv2112677457:86 , srv240824119:86 , srv417388467:86 , srv438807315:86 , srv453417307:86 , srv458742762:86 , srv537386916:86 , srv652445159:86 , srv940958428:86 , srv1036153927:87 , srv1150811298:87 , srv1308889564:87 , srv1469154361:87 , srv176705867:87 , srv1820495104:87 , srv1992787866:87 , srv2126858794:87 , srv431094109:87 , srv435558945:87 , srv451741432:87 , srv472123417:87 , srv501007149:87 , srv518779992:87 , srv580984552:87 , srv648251510:87 , srv872835888:87 , srv892897458:87 , srv913667369:87 , srv1044358957:88 , srv1654737074:88 , srv1946291933:88 , srv210650271:88 , srv289490941:88 , srv474049556:88 , srv530756191:88 , srv60834475:88 , srv978010862:88 , srv1709093535:89 , srv245860647:89 , srv312637736:89 , srv1268956207:90 , srv1468208660:90 , srv368233274:90 , srv451171557:90 , srv575686866:90 , srv947620637:90 , srv969235016:90 , srv1771092406:91 , srv228028721:91 , srv1064560104:92 , srv1399188200:93 , srv1939074696:93 , srv1240761548:95 , srv843230722:442 }
2023-08-12 15:53:08,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-12 15:53:08,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv245860647=58, srv518779992=75, srv1042532612=1, srv1654737074=30, srv1631194627=29, srv940958428=94, srv1601245979=27, srv256884433=59, srv1911681175=43, srv60834475=81, srv866089667=89, srv1210745328=9, srv580984552=79, srv1183110483=6, srv530756191=76, srv1611809699=28, srv641720926=82, srv1282330618=16, srv425433026=64, srv752052393=86, srv126839322=14, srv980344294=99, srv1208712267=8, srv1256415057=12, srv1946291933=46, srv451741432=69, srv1939074696=45, srv2057196944=49, srv1820495104=40, srv1044358957=2, srv417388467=63, srv575686866=78, srv1468208660=24, srv438807315=67, srv2083734177=51, srv1717932978=34, srv1308889564=18, srv872835888=91, srv1089498800=4, srv1318406716=19, srv2126858794=54, srv892897458=92, srv1266184604=13, srv1189690258=7, srv1674048010=32, srv453417307=70, srv1064560104=3, srv451171557=68, srv1736917852=35, srv1240761548=11, srv946835167=95, srv2112677457=53, srv435558945=66, srv763035038=87, srv1588769050=26, srv1150811298=5, srv1874576781=41, srv652445159=84, srv1268956207=15, srv474049556=73, srv472123417=72, srv228028721=56, srv176705867=36, srv1036153927=0, srv1659927426=31, srv870007275=90, srv130851095=17, srv978010862=98, srv501007149=74, srv221551599=55, srv1992787866=47, srv368233274=62, srv1709093535=33, srv1771092406=37, srv458742762=71, srv2001779046=48, srv312637736=61, srv431094109=65, srv537386916=77, srv728575386=85, srv1785270353=38, srv1469154361=25, srv913667369=93, srv599914082=80, srv240824119=57, srv210650271=52, srv969235016=97, srv1399188200=23, srv1886883097=42, srv843230722=88, srv2074098535=50, srv947620637=96, srv1222886863=10, srv1928153479=44, srv289490941=60, srv1803400830=39, srv648251510=83, srv1373785382=22, srv1327527695=20, srv1358217068=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-12 15:53:08,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-12 15:53:08,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-12 15:53:08,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-12 15:53:08,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-12 15:53:08,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-12 15:53:08,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-12 15:53:08,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-12 15:53:08,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-12 15:53:08,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-12 15:53:08,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-12 15:53:08,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-12 15:53:08,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-12 15:53:08,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-12 15:53:08,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-12 15:53:08,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-12 15:53:08,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-12 15:53:08,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-12 15:53:08,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-12 15:53:08,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-12 15:53:08,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-12 15:53:08,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-12 15:53:08,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-12 15:53:08,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-12 15:53:08,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-12 15:53:08,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-12 15:53:08,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-12 15:53:08,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-12 15:53:08,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-12 15:53:08,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-12 15:53:08,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-12 15:53:08,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-12 15:53:08,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-12 15:53:08,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-12 15:53:08,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-12 15:53:08,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-12 15:53:08,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-12 15:53:08,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-12 15:53:08,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-12 15:53:08,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-12 15:53:08,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-12 15:53:08,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-12 15:53:08,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-12 15:53:08,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-12 15:53:08,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-12 15:53:08,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-12 15:53:08,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-12 15:53:08,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-12 15:53:08,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-12 15:53:08,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-12 15:53:08,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-12 15:53:08,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-12 15:53:08,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-12 15:53:08,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-12 15:53:08,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-12 15:53:08,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-12 15:53:08,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-12 15:53:08,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-12 15:53:08,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-12 15:53:08,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-12 15:53:08,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-12 15:53:08,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-12 15:53:08,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-12 15:53:08,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-12 15:53:08,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-12 15:53:08,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-12 15:53:08,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-12 15:53:08,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-12 15:53:08,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-12 15:53:08,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-12 15:53:08,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-12 15:53:08,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-12 15:53:08,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-12 15:53:08,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-12 15:53:08,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-12 15:53:08,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-12 15:53:08,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-12 15:53:08,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-12 15:53:08,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-12 15:53:08,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-12 15:53:08,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-12 15:53:08,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-12 15:53:08,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-12 15:53:08,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-12 15:53:08,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-12 15:53:08,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-12 15:53:08,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-12 15:53:08,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-12 15:53:08,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-12 15:53:08,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-12 15:53:08,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-12 15:53:08,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-12 15:53:08,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-12 15:53:08,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-12 15:53:08,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-12 15:53:08,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-12 15:53:08,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-12 15:53:08,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-12 15:53:08,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-12 15:53:08,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-12 15:53:08,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-12 15:53:08,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-12 15:53:09,029 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03959862218767201, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05672308540187511, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03959862218767201, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17833333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24316666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-12 15:53:09,029 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-08-12 15:53:09,030 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18288706757988915, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03959862218767201, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05672308540187511, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03959862218767201, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17833333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24316666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-08-12 15:53:09,415 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 530 ms to try 5000 different iterations.  Found a solution that moves 2665 regions; Going from a computed imbalance of 0.18288706757988915 to a new imbalance of 0.016465181393598014. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021187468973462454); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032656426622879806, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2961111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021187468973462454); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04658333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-12 15:53:09,425 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1240761548:76 , srv1268956207:80 , srv1399188200:80 , srv1939074696:80 , srv575686866:80 , srv947620637:80 , srv1468208660:81 , srv368233274:81 , srv451171557:81 , srv1064560104:82 , srv1771092406:82 , srv312637736:82 , srv1150811298:83 , srv245860647:83 , srv431094109:83 , srv453417307:83 , srv969235016:83 , srv1044358957:84 , srv1654737074:84 , srv210650271:84 , srv228028721:84 , srv417388467:84 , srv458742762:84 , srv60834475:84 , srv892897458:84 , srv978010862:84 , srv1036153927:85 , srv1189690258:85 , srv1282330618:85 , srv1308889564:85 , srv1327527695:85 , srv1469154361:85 , srv176705867:85 , srv1886883097:85 , srv2001779046:85 , srv2126858794:85 , srv289490941:85 , srv435558945:85 , srv451741432:85 , srv518779992:85 , srv530756191:85 , srv580984552:85 , srv913667369:85 , srv1042532612:86 , srv1183110483:86 , srv1358217068:86 , srv1611809699:86 , srv1631194627:86 , srv1803400830:86 , srv1992787866:86 , srv2083734177:86 , srv221551599:86 , srv501007149:86 , srv648251510:86 , srv752052393:86 , srv763035038:86 , srv940958428:86 , srv980344294:86 , srv1089498800:87 , srv1208712267:87 , srv1210745328:87 , srv1222886863:87 , srv1256415057:87 , srv1318406716:87 , srv1373785382:87 , srv1588769050:87 , srv1601245979:87 , srv1659927426:87 , srv1709093535:87 , srv1717932978:87 , srv1785270353:87 , srv1946291933:87 , srv2057196944:87 , srv2074098535:87 , srv2112677457:87 , srv425433026:87 , srv438807315:87 , srv472123417:87 , srv474049556:87 , srv537386916:87 , srv870007275:87 , srv126839322:88 , srv1820495104:88 , srv1928153479:88 , srv599914082:88 , srv652445159:88 , srv866089667:88 , srv872835888:88 , srv130851095:89 , srv1736917852:89 , srv1874576781:89 , srv1911681175:89 , srv240824119:89 , srv946835167:89 , srv256884433:90 , srv1674048010:91 , srv641720926:91 , srv728575386:92 , srv1266184604:94 , srv843230722:520 }
2023-08-12 15:53:09,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-12 15:53:09,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv245860647=58, srv518779992=75, srv1042532612=1, srv1654737074=30, srv1631194627=29, srv940958428=94, srv1601245979=27, srv256884433=59, srv1911681175=43, srv60834475=81, srv866089667=89, srv1210745328=9, srv580984552=79, srv1183110483=6, srv530756191=76, srv1611809699=28, srv641720926=82, srv1282330618=16, srv425433026=64, srv752052393=86, srv126839322=14, srv980344294=99, srv1208712267=8, srv1256415057=12, srv1946291933=46, srv451741432=69, srv1939074696=45, srv2057196944=49, srv1820495104=40, srv1044358957=2, srv417388467=63, srv575686866=78, srv1468208660=24, srv438807315=67, srv2083734177=51, srv1717932978=34, srv1308889564=18, srv872835888=91, srv1089498800=4, srv1318406716=19, srv2126858794=54, srv892897458=92, srv1266184604=13, srv1189690258=7, srv1674048010=32, srv453417307=70, srv1064560104=3, srv451171557=68, srv1736917852=35, srv1240761548=11, srv946835167=95, srv2112677457=53, srv435558945=66, srv763035038=87, srv1588769050=26, srv1150811298=5, srv1874576781=41, srv652445159=84, srv1268956207=15, srv474049556=73, srv472123417=72, srv228028721=56, srv176705867=36, srv1036153927=0, srv1659927426=31, srv870007275=90, srv130851095=17, srv978010862=98, srv501007149=74, srv221551599=55, srv1992787866=47, srv368233274=62, srv1709093535=33, srv1771092406=37, srv458742762=71, srv2001779046=48, srv312637736=61, srv431094109=65, srv537386916=77, srv728575386=85, srv1785270353=38, srv1469154361=25, srv913667369=93, srv599914082=80, srv240824119=57, srv210650271=52, srv969235016=97, srv1399188200=23, srv1886883097=42, srv843230722=88, srv2074098535=50, srv947620637=96, srv1222886863=10, srv1928153479=44, srv289490941=60, srv1803400830=39, srv648251510=83, srv1373785382=22, srv1327527695=20, srv1358217068=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-12 15:53:09,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-12 15:53:09,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-12 15:53:09,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-12 15:53:09,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-12 15:53:09,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-12 15:53:09,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-12 15:53:09,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-12 15:53:09,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-12 15:53:09,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-12 15:53:09,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-12 15:53:09,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-12 15:53:09,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-12 15:53:09,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-12 15:53:09,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-12 15:53:09,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-12 15:53:09,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-12 15:53:09,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-12 15:53:09,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-12 15:53:09,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-12 15:53:09,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-12 15:53:09,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-12 15:53:09,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-12 15:53:09,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-12 15:53:09,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-12 15:53:09,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-12 15:53:09,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-12 15:53:09,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-12 15:53:09,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-12 15:53:09,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-12 15:53:09,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-12 15:53:09,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-12 15:53:09,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-12 15:53:09,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-12 15:53:09,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-12 15:53:09,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-12 15:53:09,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-12 15:53:09,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-12 15:53:09,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-12 15:53:09,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-12 15:53:09,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-12 15:53:09,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-12 15:53:09,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-12 15:53:09,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-12 15:53:09,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-12 15:53:09,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-12 15:53:09,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-12 15:53:09,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-12 15:53:09,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-12 15:53:09,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-12 15:53:09,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-12 15:53:09,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-12 15:53:09,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-12 15:53:09,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-12 15:53:09,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-12 15:53:09,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-12 15:53:09,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-12 15:53:09,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-12 15:53:09,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-12 15:53:09,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-12 15:53:09,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-12 15:53:09,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-12 15:53:09,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-12 15:53:09,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-12 15:53:09,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-12 15:53:09,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-12 15:53:09,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-12 15:53:09,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-12 15:53:09,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-12 15:53:09,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-12 15:53:09,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-12 15:53:09,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-12 15:53:09,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-12 15:53:09,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-12 15:53:09,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-12 15:53:09,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-12 15:53:09,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-12 15:53:09,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-12 15:53:09,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-12 15:53:09,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-12 15:53:09,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-12 15:53:09,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-12 15:53:09,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-12 15:53:09,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-12 15:53:09,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-12 15:53:09,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-12 15:53:09,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-12 15:53:09,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-12 15:53:09,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-12 15:53:09,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-12 15:53:09,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-12 15:53:09,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-12 15:53:09,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-12 15:53:09,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-12 15:53:09,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-12 15:53:09,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-12 15:53:09,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-12 15:53:09,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-12 15:53:09,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-12 15:53:09,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-12 15:53:09,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-12 15:53:09,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-12 15:53:09,614 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021187468973462454); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032656426622879806, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021187468973462454); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04658333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-12 15:53:09,614 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-08-12 15:53:09,614 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01644652066475426, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021187468973462454); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032656426622879806, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021187468973462454); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04658333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-08-12 15:53:09,889 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 376 ms to try 5000 different iterations.  Found a solution that moves 1033 regions; Going from a computed imbalance of 0.01644652066475426 to a new imbalance of 3.7578395742559185E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0048676203012797846); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006666666666666666); MoveCostFunction : (multiplier=7.0, imbalance=0.11477777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0048676203012797846); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0026666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-12 15:53:09,892 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1266184604:80 , srv1709093535:81 , srv1946291933:82 , srv728575386:82 , srv1064560104:83 , srv1820495104:83 , srv228028721:83 , srv240824119:83 , srv256884433:83 , srv474049556:83 , srv641720926:83 , srv652445159:83 , srv872835888:83 , srv130851095:84 , srv1601245979:84 , srv1674048010:84 , srv1736917852:84 , srv1771092406:84 , srv1874576781:84 , srv1939074696:84 , srv2057196944:84 , srv472123417:84 , srv501007149:84 , srv530756191:84 , srv648251510:84 , srv1036153927:85 , srv1208712267:85 , srv1210745328:85 , srv1222886863:85 , srv126839322:85 , srv1373785382:85 , srv1399188200:85 , srv1654737074:85 , srv176705867:85 , srv1911681175:85 , srv1928153479:85 , srv2112677457:85 , srv2126858794:85 , srv245860647:85 , srv289490941:85 , srv425433026:85 , srv438807315:85 , srv451741432:85 , srv537386916:85 , srv599914082:85 , srv940958428:85 , srv969235016:85 , srv978010862:85 , srv1044358957:86 , srv1189690258:86 , srv1308889564:86 , srv1358217068:86 , srv1468208660:86 , srv1469154361:86 , srv1588769050:86 , srv1717932978:86 , srv1785270353:86 , srv1886883097:86 , srv1992787866:86 , srv210650271:86 , srv221551599:86 , srv435558945:86 , srv518779992:86 , srv580984552:86 , srv60834475:86 , srv752052393:86 , srv763035038:86 , srv866089667:86 , srv913667369:86 , srv946835167:86 , srv1089498800:87 , srv1150811298:87 , srv1183110483:87 , srv1240761548:87 , srv1256415057:87 , srv1268956207:87 , srv1611809699:87 , srv1659927426:87 , srv1803400830:87 , srv2001779046:87 , srv2083734177:87 , srv312637736:87 , srv368233274:87 , srv417388467:87 , srv431094109:87 , srv451171557:87 , srv870007275:87 , srv892897458:87 , srv947620637:87 , srv980344294:87 , srv1042532612:88 , srv1282330618:88 , srv1318406716:88 , srv1327527695:88 , srv1631194627:88 , srv2074098535:88 , srv575686866:88 , srv453417307:89 , srv458742762:89 , srv843230722:539 }
2023-08-12 15:53:09,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-12 15:53:09,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv245860647=58, srv518779992=75, srv1042532612=1, srv1654737074=30, srv1631194627=29, srv940958428=94, srv1601245979=27, srv256884433=59, srv1911681175=43, srv60834475=81, srv866089667=89, srv1210745328=9, srv580984552=79, srv1183110483=6, srv530756191=76, srv1611809699=28, srv641720926=82, srv1282330618=16, srv425433026=64, srv752052393=86, srv126839322=14, srv980344294=99, srv1208712267=8, srv1256415057=12, srv1946291933=46, srv451741432=69, srv1939074696=45, srv2057196944=49, srv1820495104=40, srv1044358957=2, srv417388467=63, srv575686866=78, srv1468208660=24, srv438807315=67, srv2083734177=51, srv1717932978=34, srv1308889564=18, srv872835888=91, srv1089498800=4, srv1318406716=19, srv2126858794=54, srv892897458=92, srv1266184604=13, srv1189690258=7, srv1674048010=32, srv453417307=70, srv1064560104=3, srv451171557=68, srv1736917852=35, srv1240761548=11, srv946835167=95, srv2112677457=53, srv435558945=66, srv763035038=87, srv1588769050=26, srv1150811298=5, srv1874576781=41, srv652445159=84, srv1268956207=15, srv474049556=73, srv472123417=72, srv228028721=56, srv176705867=36, srv1036153927=0, srv1659927426=31, srv870007275=90, srv130851095=17, srv978010862=98, srv501007149=74, srv221551599=55, srv1992787866=47, srv368233274=62, srv1709093535=33, srv1771092406=37, srv458742762=71, srv2001779046=48, srv312637736=61, srv431094109=65, srv537386916=77, srv728575386=85, srv1785270353=38, srv1469154361=25, srv913667369=93, srv599914082=80, srv240824119=57, srv210650271=52, srv969235016=97, srv1399188200=23, srv1886883097=42, srv843230722=88, srv2074098535=50, srv947620637=96, srv1222886863=10, srv1928153479=44, srv289490941=60, srv1803400830=39, srv648251510=83, srv1373785382=22, srv1327527695=20, srv1358217068=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-12 15:53:09,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-12 15:53:09,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-12 15:53:09,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-12 15:53:09,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-12 15:53:09,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-12 15:53:09,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-12 15:53:09,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-12 15:53:09,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-12 15:53:09,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-12 15:53:09,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-12 15:53:09,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-12 15:53:09,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-12 15:53:09,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-12 15:53:09,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-12 15:53:09,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-12 15:53:09,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-12 15:53:09,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-12 15:53:09,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-12 15:53:09,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-12 15:53:09,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-12 15:53:09,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-12 15:53:09,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-12 15:53:09,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-12 15:53:09,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-12 15:53:09,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-12 15:53:09,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-12 15:53:09,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-12 15:53:09,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-12 15:53:09,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-12 15:53:09,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-12 15:53:09,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-12 15:53:09,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-12 15:53:09,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-12 15:53:09,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-12 15:53:09,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-12 15:53:09,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-12 15:53:09,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-12 15:53:09,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-12 15:53:09,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-12 15:53:09,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-12 15:53:09,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-12 15:53:09,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-12 15:53:09,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-12 15:53:09,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-12 15:53:09,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-12 15:53:09,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-12 15:53:09,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-12 15:53:09,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-12 15:53:09,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-12 15:53:09,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-12 15:53:09,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-12 15:53:09,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-12 15:53:09,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-12 15:53:09,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-12 15:53:09,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-12 15:53:09,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-12 15:53:09,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-12 15:53:09,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-12 15:53:09,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-12 15:53:09,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-12 15:53:09,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-12 15:53:09,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-12 15:53:09,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-12 15:53:09,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-12 15:53:09,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-12 15:53:09,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-12 15:53:09,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-12 15:53:09,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-12 15:53:09,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-12 15:53:09,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-12 15:53:09,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-12 15:53:09,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-12 15:53:09,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-12 15:53:09,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-12 15:53:09,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-12 15:53:09,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-12 15:53:09,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-12 15:53:09,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-12 15:53:09,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-12 15:53:09,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-12 15:53:09,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-12 15:53:09,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-12 15:53:09,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-12 15:53:09,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-12 15:53:09,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-12 15:53:09,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-12 15:53:09,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-12 15:53:09,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-12 15:53:09,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-12 15:53:09,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-12 15:53:09,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-12 15:53:09,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-12 15:53:09,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-12 15:53:09,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-12 15:53:09,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-12 15:53:09,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-12 15:53:09,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-12 15:53:09,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-12 15:53:09,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-12 15:53:09,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-12 15:53:09,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-12 15:53:10,075 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0048676203012797846); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006666666666666666); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0048676203012797846); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0026666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-12 15:53:10,075 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-08-12 15:53:10,076 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.6855073682686795E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0048676203012797846); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006666666666666666); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0048676203012797846); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0026666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-08-12 15:53:10,592 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 650 ms to try 5000 different iterations.  Found a solution that moves 222 regions; Going from a computed imbalance of 3.6855073682686795E-4 to a new imbalance of 1.2884543557210005E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.024666666666666667); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-12 15:53:10,609 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1266184604:83 , srv1282330618:83 , srv1327527695:83 , srv1928153479:83 , srv1992787866:83 , srv289490941:83 , srv458742762:83 , srv1042532612:84 , srv1044358957:84 , srv1089498800:84 , srv1183110483:84 , srv1240761548:84 , srv130851095:84 , srv1308889564:84 , srv1318406716:84 , srv1399188200:84 , srv1469154361:84 , srv1717932978:84 , srv176705867:84 , srv1785270353:84 , srv1820495104:84 , srv1874576781:84 , srv1911681175:84 , srv2001779046:84 , srv2074098535:84 , srv2083734177:84 , srv210650271:84 , srv2112677457:84 , srv228028721:84 , srv240824119:84 , srv256884433:84 , srv312637736:84 , srv368233274:84 , srv431094109:84 , srv435558945:84 , srv438807315:84 , srv451171557:84 , srv453417307:84 , srv472123417:84 , srv474049556:84 , srv518779992:84 , srv537386916:84 , srv575686866:84 , srv580984552:84 , srv60834475:84 , srv652445159:84 , srv728575386:84 , srv866089667:84 , srv870007275:84 , srv892897458:84 , srv913667369:84 , srv946835167:84 , srv980344294:84 , srv1036153927:85 , srv1064560104:85 , srv1189690258:85 , srv1208712267:85 , srv1210745328:85 , srv1222886863:85 , srv1256415057:85 , srv126839322:85 , srv1358217068:85 , srv1373785382:85 , srv1468208660:85 , srv1588769050:85 , srv1611809699:85 , srv1631194627:85 , srv1654737074:85 , srv1659927426:85 , srv1674048010:85 , srv1709093535:85 , srv1736917852:85 , srv1771092406:85 , srv1803400830:85 , srv1886883097:85 , srv1939074696:85 , srv1946291933:85 , srv2126858794:85 , srv221551599:85 , srv245860647:85 , srv417388467:85 , srv425433026:85 , srv451741432:85 , srv501007149:85 , srv530756191:85 , srv599914082:85 , srv641720926:85 , srv648251510:85 , srv752052393:85 , srv763035038:85 , srv872835888:85 , srv940958428:85 , srv947620637:85 , srv969235016:85 , srv978010862:85 , srv1150811298:86 , srv1268956207:86 , srv1601245979:86 , srv2057196944:86 , srv843230722:641 }
2023-08-12 15:53:10,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-12 15:53:10,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv245860647=58, srv518779992=75, srv1042532612=1, srv1654737074=30, srv1631194627=29, srv940958428=94, srv1601245979=27, srv256884433=59, srv1911681175=43, srv60834475=81, srv866089667=89, srv1210745328=9, srv580984552=79, srv1183110483=6, srv530756191=76, srv1611809699=28, srv641720926=82, srv1282330618=16, srv425433026=64, srv752052393=86, srv126839322=14, srv980344294=99, srv1208712267=8, srv1256415057=12, srv1946291933=46, srv451741432=69, srv1939074696=45, srv2057196944=49, srv1820495104=40, srv1044358957=2, srv417388467=63, srv575686866=78, srv1468208660=24, srv438807315=67, srv2083734177=51, srv1717932978=34, srv1308889564=18, srv872835888=91, srv1089498800=4, srv1318406716=19, srv2126858794=54, srv892897458=92, srv1266184604=13, srv1189690258=7, srv1674048010=32, srv453417307=70, srv1064560104=3, srv451171557=68, srv1736917852=35, srv1240761548=11, srv946835167=95, srv2112677457=53, srv435558945=66, srv763035038=87, srv1588769050=26, srv1150811298=5, srv1874576781=41, srv652445159=84, srv1268956207=15, srv474049556=73, srv472123417=72, srv228028721=56, srv176705867=36, srv1036153927=0, srv1659927426=31, srv870007275=90, srv130851095=17, srv978010862=98, srv501007149=74, srv221551599=55, srv1992787866=47, srv368233274=62, srv1709093535=33, srv1771092406=37, srv458742762=71, srv2001779046=48, srv312637736=61, srv431094109=65, srv537386916=77, srv728575386=85, srv1785270353=38, srv1469154361=25, srv913667369=93, srv599914082=80, srv240824119=57, srv210650271=52, srv969235016=97, srv1399188200=23, srv1886883097=42, srv843230722=88, srv2074098535=50, srv947620637=96, srv1222886863=10, srv1928153479=44, srv289490941=60, srv1803400830=39, srv648251510=83, srv1373785382=22, srv1327527695=20, srv1358217068=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-12 15:53:10,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-12 15:53:10,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-12 15:53:10,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-12 15:53:10,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-12 15:53:10,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-12 15:53:10,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-12 15:53:10,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-12 15:53:10,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-12 15:53:10,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-12 15:53:10,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-12 15:53:10,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-12 15:53:10,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-12 15:53:10,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-12 15:53:10,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-12 15:53:10,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-12 15:53:10,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-12 15:53:10,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-12 15:53:10,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-12 15:53:10,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-12 15:53:10,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-12 15:53:10,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-12 15:53:10,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-12 15:53:10,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-12 15:53:10,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-12 15:53:10,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-12 15:53:10,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-12 15:53:10,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-12 15:53:10,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-12 15:53:10,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-12 15:53:10,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-12 15:53:10,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-12 15:53:10,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-12 15:53:10,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-12 15:53:10,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-12 15:53:10,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-12 15:53:10,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-12 15:53:10,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-12 15:53:10,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-12 15:53:10,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-12 15:53:10,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-12 15:53:10,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-12 15:53:10,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-12 15:53:10,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-12 15:53:10,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-12 15:53:10,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-12 15:53:10,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-12 15:53:10,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-12 15:53:10,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-12 15:53:10,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-12 15:53:10,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-12 15:53:10,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-12 15:53:10,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-12 15:53:10,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-12 15:53:10,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-12 15:53:10,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-12 15:53:10,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-12 15:53:10,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-12 15:53:10,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-12 15:53:10,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-12 15:53:10,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-12 15:53:10,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-12 15:53:10,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-12 15:53:10,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-12 15:53:10,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-12 15:53:10,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-12 15:53:10,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-12 15:53:10,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-12 15:53:10,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-12 15:53:10,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-12 15:53:10,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-12 15:53:10,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-12 15:53:10,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-12 15:53:10,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-12 15:53:10,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-12 15:53:10,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-12 15:53:10,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-12 15:53:10,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-12 15:53:10,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-12 15:53:10,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-12 15:53:10,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-12 15:53:10,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-12 15:53:10,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-12 15:53:10,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-12 15:53:10,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-12 15:53:10,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-12 15:53:10,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-12 15:53:10,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-12 15:53:10,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-12 15:53:10,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-12 15:53:10,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-12 15:53:10,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-12 15:53:10,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-12 15:53:10,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-12 15:53:10,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-12 15:53:10,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-12 15:53:10,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-12 15:53:10,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-12 15:53:10,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-12 15:53:10,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-12 15:53:10,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-12 15:53:10,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-12 15:53:10,794 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.133006633270208E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-12 15:53:10,794 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1266184604:83 , srv1282330618:83 , srv1327527695:83 , srv1928153479:83 , srv1992787866:83 , srv289490941:83 , srv458742762:83 , srv1042532612:84 , srv1044358957:84 , srv1089498800:84 , srv1183110483:84 , srv1240761548:84 , srv130851095:84 , srv1308889564:84 , srv1318406716:84 , srv1399188200:84 , srv1469154361:84 , srv1717932978:84 , srv176705867:84 , srv1785270353:84 , srv1820495104:84 , srv1874576781:84 , srv1911681175:84 , srv2001779046:84 , srv2074098535:84 , srv2083734177:84 , srv210650271:84 , srv2112677457:84 , srv228028721:84 , srv240824119:84 , srv256884433:84 , srv312637736:84 , srv368233274:84 , srv431094109:84 , srv435558945:84 , srv438807315:84 , srv451171557:84 , srv453417307:84 , srv472123417:84 , srv474049556:84 , srv518779992:84 , srv537386916:84 , srv575686866:84 , srv580984552:84 , srv60834475:84 , srv652445159:84 , srv728575386:84 , srv866089667:84 , srv870007275:84 , srv892897458:84 , srv913667369:84 , srv946835167:84 , srv980344294:84 , srv1036153927:85 , srv1064560104:85 , srv1189690258:85 , srv1208712267:85 , srv1210745328:85 , srv1222886863:85 , srv1256415057:85 , srv126839322:85 , srv1358217068:85 , srv1373785382:85 , srv1468208660:85 , srv1588769050:85 , srv1611809699:85 , srv1631194627:85 , srv1654737074:85 , srv1659927426:85 , srv1674048010:85 , srv1709093535:85 , srv1736917852:85 , srv1771092406:85 , srv1803400830:85 , srv1886883097:85 , srv1939074696:85 , srv1946291933:85 , srv2126858794:85 , srv221551599:85 , srv245860647:85 , srv417388467:85 , srv425433026:85 , srv451741432:85 , srv501007149:85 , srv530756191:85 , srv599914082:85 , srv641720926:85 , srv648251510:85 , srv752052393:85 , srv763035038:85 , srv872835888:85 , srv940958428:85 , srv947620637:85 , srv969235016:85 , srv978010862:85 , srv1150811298:86 , srv1268956207:86 , srv1601245979:86 , srv2057196944:86 , srv843230722:641 }
2023-08-12 15:53:10,826 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=315 (was 298) - SystemLoadAverage LEAK? -, ProcessCount=173 (was 173), AvailableMemoryMB=21547 (was 22406)
2023-08-12 15:53:10,842 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=315, ProcessCount=173, AvailableMemoryMB=21546
2023-08-12 15:53:10,843 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-08-12 15:53:10,844 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-08-12 15:53:10,856 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1001064692:84 , srv1095635044:84 , srv1153833124:84 , srv1473175095:84 , srv1494148272:84 , srv1589370776:84 , srv1602320073:84 , srv165036380:84 , srv1713682253:84 , srv1715816481:84 , srv1725353909:84 , srv1780987346:84 , srv1786507202:84 , srv1839391469:84 , srv1844443490:84 , srv1966301159:84 , srv283608499:84 , srv335153718:84 , srv38965957:84 , srv487721577:84 , srv497900305:84 , srv593761845:84 , srv602288274:84 , srv611598520:84 , srv779491659:84 , srv806185790:84 , srv869019902:84 , srv94988907:84 , srv966688732:84 , srv822041530:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-08-12 15:53:10,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-12 15:53:10,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1473175095=3, srv165036380=7, srv1844443490=14, srv283608499=16, srv806185790=25, srv1095635044=1, srv1780987346=11, srv1839391469=13, srv611598520=23, srv779491659=24, srv602288274=22, srv822041530=26, srv1725353909=10, srv966688732=29, srv1589370776=5, srv497900305=20, srv869019902=27, srv335153718=17, srv593761845=21, srv1153833124=2, srv487721577=19, srv94988907=28, srv1001064692=0, srv1713682253=8, srv1715816481=9, srv38965957=18, srv1786507202=12, srv1602320073=6, srv1494148272=4, srv1966301159=15} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-12 15:53:10,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-12 15:53:10,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-12 15:53:10,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-12 15:53:10,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-12 15:53:10,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-12 15:53:10,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-12 15:53:10,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-12 15:53:10,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-12 15:53:10,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-12 15:53:10,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-12 15:53:10,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-12 15:53:10,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-12 15:53:10,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-12 15:53:10,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-12 15:53:10,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-12 15:53:10,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-12 15:53:10,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-12 15:53:10,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-12 15:53:10,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-12 15:53:10,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-12 15:53:10,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-12 15:53:10,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-12 15:53:10,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-12 15:53:10,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-12 15:53:10,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-12 15:53:10,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-12 15:53:10,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-12 15:53:10,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-12 15:53:10,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-12 15:53:10,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-12 15:53:10,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-08-12 15:53:10,889 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0771884637057116, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-12 15:53:10,889 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912436261592983, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0771884637057116, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-08-12 15:55:10,872 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5736210 different iterations.  Found a solution that moves 1848 regions; Going from a computed imbalance of 0.9912436261592983 to a new imbalance of 4.313324190526491E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6844444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-12 15:55:10,880 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1001064692:90 , srv1095635044:90 , srv1153833124:90 , srv1473175095:90 , srv1494148272:90 , srv1589370776:90 , srv1602320073:90 , srv165036380:90 , srv1713682253:90 , srv1715816481:90 , srv1725353909:90 , srv1780987346:90 , srv1786507202:90 , srv1839391469:90 , srv1844443490:90 , srv1966301159:90 , srv283608499:90 , srv335153718:90 , srv38965957:90 , srv487721577:90 , srv497900305:90 , srv593761845:90 , srv602288274:90 , srv611598520:90 , srv779491659:90 , srv806185790:90 , srv822041530:90 , srv869019902:90 , srv94988907:90 , srv966688732:90 }
2023-08-12 15:55:10,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-12 15:55:10,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1473175095=3, srv165036380=7, srv1844443490=14, srv283608499=16, srv806185790=25, srv1095635044=1, srv1780987346=11, srv1839391469=13, srv611598520=23, srv779491659=24, srv602288274=22, srv822041530=26, srv1725353909=10, srv966688732=29, srv1589370776=5, srv497900305=20, srv869019902=27, srv335153718=17, srv593761845=21, srv1153833124=2, srv487721577=19, srv94988907=28, srv1001064692=0, srv1713682253=8, srv1715816481=9, srv38965957=18, srv1786507202=12, srv1602320073=6, srv1494148272=4, srv1966301159=15} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-12 15:55:10,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-12 15:55:10,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-12 15:55:10,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-12 15:55:10,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-12 15:55:10,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-12 15:55:10,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-12 15:55:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-12 15:55:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-12 15:55:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-12 15:55:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-12 15:55:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-12 15:55:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-12 15:55:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-12 15:55:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-12 15:55:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-12 15:55:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-12 15:55:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-12 15:55:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-12 15:55:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-12 15:55:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-12 15:55:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-12 15:55:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-12 15:55:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-12 15:55:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-12 15:55:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-12 15:55:10,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-12 15:55:10,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-12 15:55:10,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-12 15:55:10,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-12 15:55:10,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-12 15:55:10,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-08-12 15:55:10,900 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-12 15:55:10,900 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1001064692:90 , srv1095635044:90 , srv1153833124:90 , srv1473175095:90 , srv1494148272:90 , srv1589370776:90 , srv1602320073:90 , srv165036380:90 , srv1713682253:90 , srv1715816481:90 , srv1725353909:90 , srv1780987346:90 , srv1786507202:90 , srv1839391469:90 , srv1844443490:90 , srv1966301159:90 , srv283608499:90 , srv335153718:90 , srv38965957:90 , srv487721577:90 , srv497900305:90 , srv593761845:90 , srv602288274:90 , srv611598520:90 , srv779491659:90 , srv806185790:90 , srv822041530:90 , srv869019902:90 , srv94988907:90 , srv966688732:90 }
2023-08-12 15:55:10,920 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=227 (was 315), ProcessCount=169 (was 173), AvailableMemoryMB=21547 (was 21546) - AvailableMemoryMB LEAK? -