2023-11-13 08:55:11,253 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-11-13 08:55:11,545 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-11-13 08:55:11,651 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-11-13 08:55:11,721 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-13 08:55:11,731 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-13 08:55:11,746 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=121, ProcessCount=169, AvailableMemoryMB=3571
2023-11-13 08:55:11,749 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-13 08:55:11,749 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-13 08:55:11,868 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv104160571:84 , srv1069351860:84 , srv1082417507:84 , srv1087243005:84 , srv1101919741:84 , srv1102590333:84 , srv1135642186:84 , srv1136896670:84 , srv1148728588:84 , srv1153847990:84 , srv1192550867:84 , srv122076593:84 , srv1222373313:84 , srv1233014118:84 , srv1233946041:84 , srv128101194:84 , srv1288627709:84 , srv1318950717:84 , srv1325007117:84 , srv1328327538:84 , srv1366378540:84 , srv1405724148:84 , srv1420600466:84 , srv1462461291:84 , srv1469397589:84 , srv1475089606:84 , srv1505426902:84 , srv1537339291:84 , srv1552847940:84 , srv1569491415:84 , srv1580575639:84 , srv1592596358:84 , srv1599465578:84 , srv1599674213:84 , srv166791497:84 , srv1672027151:84 , srv1673856823:84 , srv1685069120:84 , srv1693055652:84 , srv1703833936:84 , srv171021637:84 , srv1728761992:84 , srv1741109537:84 , srv1758283938:84 , srv176273025:84 , srv1806741123:84 , srv1808423165:84 , srv1825585239:84 , srv1905667136:84 , srv1918638166:84 , srv198217139:84 , srv2005932792:84 , srv2021856643:84 , srv2048068438:84 , srv2050057174:84 , srv2091836025:84 , srv2099022303:84 , srv219505189:84 , srv226421956:84 , srv239238052:84 , srv28013559:84 , srv280408145:84 , srv289675077:84 , srv347487995:84 , srv352234480:84 , srv353336489:84 , srv406793401:84 , srv416357454:84 , srv419630628:84 , srv422051170:84 , srv460607717:84 , srv461210485:84 , srv474383757:84 , srv500056315:84 , srv513768960:84 , srv5553139:84 , srv561154617:84 , srv565602939:84 , srv599115434:84 , srv669773518:84 , srv671016393:84 , srv694385826:84 , srv694682889:84 , srv718472509:84 , srv720825849:84 , srv721701402:84 , srv786476156:84 , srv803872292:84 , srv815095370:84 , srv817825699:84 , srv819029373:84 , srv819780327:84 , srv858688916:84 , srv865968290:84 , srv900068256:84 , srv936577654:84 , srv983862975:84 , srv994989412:84 , srv998619195:84 , srv2008426936:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-11-13 08:55:11,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-13 08:55:11,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1318950717=17, srv721701402=86, srv1462461291=23, srv694385826=82, srv280408145=62, srv994989412=98, srv1192550867=10, srv718472509=84, srv1808423165=46, srv1672027151=35, srv513768960=75, srv1592596358=31, srv936577654=96, srv2021856643=53, srv669773518=80, srv104160571=0, srv176273025=44, srv1580575639=30, srv1101919741=4, srv2099022303=57, srv817825699=90, srv1082417507=2, srv865968290=94, srv1505426902=26, srv416357454=68, srv983862975=97, srv1087243005=3, srv1685069120=37, srv1569491415=29, srv561154617=77, srv1102590333=5, srv461210485=72, srv1069351860=1, srv198217139=50, srv1703833936=39, srv353336489=66, srv1136896670=7, srv1405724148=21, srv819029373=91, srv900068256=95, srv226421956=59, srv1135642186=6, srv347487995=64, srv998619195=99, srv239238052=60, srv1153847990=9, srv122076593=11, srv2005932792=51, srv171021637=40, srv1741109537=42, srv166791497=34, srv720825849=85, srv1325007117=18, srv419630628=69, srv671016393=81, srv803872292=88, srv1328327538=19, srv1673856823=36, srv1288627709=16, srv1222373313=12, srv422051170=70, srv1599674213=33, srv815095370=89, srv474383757=73, srv219505189=58, srv1806741123=45, srv694682889=83, srv1552847940=28, srv1728761992=41, srv1233014118=13, srv1148728588=8, srv1599465578=32, srv1693055652=38, srv500056315=74, srv2091836025=56, srv28013559=61, srv352234480=65, srv2048068438=54, srv1233946041=14, srv2050057174=55, srv1420600466=22, srv1537339291=27, srv1758283938=43, srv1918638166=49, srv1475089606=25, srv2008426936=52, srv786476156=87, srv5553139=76, srv406793401=67, srv460607717=71, srv599115434=79, srv819780327=92, srv1905667136=48, srv1825585239=47, srv858688916=93, srv565602939=78, srv128101194=15, srv1469397589=24, srv1366378540=20, srv289675077=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-13 08:55:11,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-13 08:55:11,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-13 08:55:11,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-13 08:55:11,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-13 08:55:11,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-13 08:55:11,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-13 08:55:11,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-13 08:55:11,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-13 08:55:11,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-13 08:55:11,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-13 08:55:11,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-13 08:55:11,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-13 08:55:11,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-13 08:55:11,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-13 08:55:11,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-13 08:55:11,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-13 08:55:11,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-13 08:55:11,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-13 08:55:11,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-13 08:55:11,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-13 08:55:11,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-13 08:55:11,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-13 08:55:11,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-13 08:55:11,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-13 08:55:11,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-13 08:55:11,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-13 08:55:11,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-13 08:55:11,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-13 08:55:11,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-13 08:55:11,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-13 08:55:11,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-13 08:55:11,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-13 08:55:11,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-13 08:55:11,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-13 08:55:11,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-13 08:55:11,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-13 08:55:11,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-13 08:55:11,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-13 08:55:11,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-13 08:55:11,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-13 08:55:11,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-13 08:55:11,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-13 08:55:11,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-13 08:55:11,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-13 08:55:11,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-13 08:55:11,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-13 08:55:11,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-13 08:55:11,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-13 08:55:11,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-13 08:55:11,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-13 08:55:11,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-13 08:55:11,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-13 08:55:11,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-13 08:55:11,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-13 08:55:11,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-13 08:55:11,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-13 08:55:11,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-13 08:55:11,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-13 08:55:11,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-13 08:55:11,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-13 08:55:11,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-13 08:55:11,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-13 08:55:11,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-13 08:55:11,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-13 08:55:11,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-13 08:55:11,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-13 08:55:11,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-13 08:55:11,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-13 08:55:11,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-13 08:55:11,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-13 08:55:11,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-13 08:55:11,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-13 08:55:11,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-13 08:55:11,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-13 08:55:11,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-13 08:55:11,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-13 08:55:11,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-13 08:55:11,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-13 08:55:11,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-13 08:55:11,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-13 08:55:11,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-13 08:55:11,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-13 08:55:11,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-13 08:55:11,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-13 08:55:11,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-13 08:55:11,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-13 08:55:11,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-13 08:55:11,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-13 08:55:11,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-13 08:55:11,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-13 08:55:11,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-13 08:55:11,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-13 08:55:11,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-13 08:55:11,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-13 08:55:11,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-13 08:55:11,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-13 08:55:11,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-13 08:55:11,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-13 08:55:11,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-13 08:55:11,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-13 08:55:11,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-13 08:55:12,267 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-13 08:55:12,268 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-13 08:55:12,268 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-13 08:55:12,953 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 938 ms to try 5000 different iterations.  Found a solution that moves 4718 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17577325491102277. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038270699819136625, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05710957825243148, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5242222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038270699819136625, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17075, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23966666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-13 08:55:12,968 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1741109537:82 , srv1918638166:82 , srv2050057174:82 , srv460607717:82 , srv1102590333:83 , srv1136896670:83 , srv1328327538:83 , srv1599674213:83 , srv1728761992:83 , srv858688916:83 , srv104160571:84 , srv1069351860:84 , srv1101919741:84 , srv1148728588:84 , srv1475089606:84 , srv1592596358:84 , srv166791497:84 , srv2005932792:84 , srv2091836025:84 , srv280408145:84 , srv422051170:84 , srv5553139:84 , srv819029373:84 , srv865968290:84 , srv1192550867:85 , srv1420600466:85 , srv1462461291:85 , srv1673856823:85 , srv1825585239:85 , srv2048068438:85 , srv289675077:85 , srv561154617:85 , srv694385826:85 , srv721701402:85 , srv803872292:85 , srv1135642186:86 , srv1222373313:86 , srv1469397589:86 , srv1552847940:86 , srv1569491415:86 , srv1599465578:86 , srv1758283938:86 , srv176273025:86 , srv2021856643:86 , srv226421956:86 , srv28013559:86 , srv353336489:86 , srv416357454:86 , srv419630628:86 , srv500056315:86 , srv786476156:86 , srv817825699:86 , srv900068256:86 , srv936577654:86 , srv994989412:86 , srv1087243005:87 , srv1153847990:87 , srv1233946041:87 , srv1580575639:87 , srv1808423165:87 , srv219505189:87 , srv352234480:87 , srv461210485:87 , srv815095370:87 , srv983862975:87 , srv998619195:87 , srv122076593:88 , srv1233014118:88 , srv1325007117:88 , srv1366378540:88 , srv1405724148:88 , srv1505426902:88 , srv1537339291:88 , srv1672027151:88 , srv1703833936:88 , srv1806741123:88 , srv1905667136:88 , srv239238052:88 , srv128101194:89 , srv1318950717:89 , srv2099022303:89 , srv474383757:89 , srv599115434:89 , srv694682889:89 , srv1693055652:90 , srv198217139:90 , srv406793401:90 , srv565602939:90 , srv669773518:90 , srv718472509:90 , srv720825849:90 , srv819780327:90 , srv1082417507:91 , srv1685069120:91 , srv671016393:91 , srv1288627709:92 , srv513768960:93 , srv171021637:94 , srv347487995:94 , srv2008426936:430 }
2023-11-13 08:55:12,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-13 08:55:12,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1318950717=17, srv721701402=86, srv1462461291=23, srv694385826=82, srv280408145=62, srv994989412=98, srv1192550867=10, srv718472509=84, srv1808423165=46, srv1672027151=35, srv513768960=75, srv1592596358=31, srv936577654=96, srv2021856643=53, srv669773518=80, srv104160571=0, srv176273025=44, srv1580575639=30, srv1101919741=4, srv2099022303=57, srv817825699=90, srv1082417507=2, srv865968290=94, srv1505426902=26, srv416357454=68, srv983862975=97, srv1087243005=3, srv1685069120=37, srv1569491415=29, srv561154617=77, srv1102590333=5, srv461210485=72, srv1069351860=1, srv198217139=50, srv1703833936=39, srv353336489=66, srv1136896670=7, srv1405724148=21, srv819029373=91, srv900068256=95, srv226421956=59, srv1135642186=6, srv347487995=64, srv998619195=99, srv239238052=60, srv1153847990=9, srv122076593=11, srv2005932792=51, srv171021637=40, srv1741109537=42, srv166791497=34, srv720825849=85, srv1325007117=18, srv419630628=69, srv671016393=81, srv803872292=88, srv1328327538=19, srv1673856823=36, srv1288627709=16, srv1222373313=12, srv422051170=70, srv1599674213=33, srv815095370=89, srv474383757=73, srv219505189=58, srv1806741123=45, srv694682889=83, srv1552847940=28, srv1728761992=41, srv1233014118=13, srv1148728588=8, srv1599465578=32, srv1693055652=38, srv500056315=74, srv2091836025=56, srv28013559=61, srv352234480=65, srv2048068438=54, srv1233946041=14, srv2050057174=55, srv1420600466=22, srv1537339291=27, srv1758283938=43, srv1918638166=49, srv1475089606=25, srv2008426936=52, srv786476156=87, srv5553139=76, srv406793401=67, srv460607717=71, srv599115434=79, srv819780327=92, srv1905667136=48, srv1825585239=47, srv858688916=93, srv565602939=78, srv128101194=15, srv1469397589=24, srv1366378540=20, srv289675077=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-13 08:55:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-13 08:55:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-13 08:55:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-13 08:55:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-13 08:55:12,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-13 08:55:12,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-13 08:55:12,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-13 08:55:12,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-13 08:55:12,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-13 08:55:12,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-13 08:55:12,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-13 08:55:13,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-13 08:55:13,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-13 08:55:13,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-13 08:55:13,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-13 08:55:13,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-13 08:55:13,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-13 08:55:13,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-13 08:55:13,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-13 08:55:13,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-13 08:55:13,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-13 08:55:13,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-13 08:55:13,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-13 08:55:13,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-13 08:55:13,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-13 08:55:13,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-13 08:55:13,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-13 08:55:13,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-13 08:55:13,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-13 08:55:13,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-13 08:55:13,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-13 08:55:13,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-13 08:55:13,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-13 08:55:13,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-13 08:55:13,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-13 08:55:13,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-13 08:55:13,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-13 08:55:13,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-13 08:55:13,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-13 08:55:13,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-13 08:55:13,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-13 08:55:13,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-13 08:55:13,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-13 08:55:13,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-13 08:55:13,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-13 08:55:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-13 08:55:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-13 08:55:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-13 08:55:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-13 08:55:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-13 08:55:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-13 08:55:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-13 08:55:13,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-13 08:55:13,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-13 08:55:13,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-13 08:55:13,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-13 08:55:13,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-13 08:55:13,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-13 08:55:13,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-13 08:55:13,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-13 08:55:13,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-13 08:55:13,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-13 08:55:13,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-13 08:55:13,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-13 08:55:13,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-13 08:55:13,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-13 08:55:13,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-13 08:55:13,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-13 08:55:13,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-13 08:55:13,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-13 08:55:13,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-13 08:55:13,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-13 08:55:13,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-13 08:55:13,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-13 08:55:13,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-13 08:55:13,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-13 08:55:13,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-13 08:55:13,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-13 08:55:13,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-13 08:55:13,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-13 08:55:13,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-13 08:55:13,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-13 08:55:13,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-13 08:55:13,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-13 08:55:13,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-13 08:55:13,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-13 08:55:13,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-13 08:55:13,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-13 08:55:13,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-13 08:55:13,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-13 08:55:13,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-13 08:55:13,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-13 08:55:13,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-13 08:55:13,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-13 08:55:13,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-13 08:55:13,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-13 08:55:13,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-13 08:55:13,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-13 08:55:13,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-13 08:55:13,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-13 08:55:13,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-13 08:55:13,197 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038270699819136625, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05710957825243148, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038270699819136625, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17075, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23966666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-13 08:55:13,197 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-13 08:55:13,197 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17574021876892715, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038270699819136625, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05710957825243148, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038270699819136625, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17075, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23966666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-13 08:55:13,452 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 407 ms to try 5000 different iterations.  Found a solution that moves 2577 regions; Going from a computed imbalance of 0.17574021876892715 to a new imbalance of 0.015278395687378458. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0215110675162575); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03879787329147873, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28633333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0215110675162575); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012166666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04475, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-13 08:55:13,460 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv347487995:77 , srv513768960:77 , srv1082417507:79 , srv1685069120:79 , srv171021637:79 , srv1693055652:81 , srv565602939:81 , srv718472509:81 , srv1325007117:82 , srv1905667136:82 , srv406793401:82 , srv671016393:82 , srv983862975:82 , srv122076593:83 , srv128101194:83 , srv1505426902:83 , srv1672027151:83 , srv1703833936:83 , srv1806741123:83 , srv2099022303:83 , srv599115434:83 , srv669773518:83 , srv720825849:83 , srv1087243005:84 , srv1153847990:84 , srv1222373313:84 , srv1318950717:84 , srv1405724148:84 , srv1537339291:84 , srv1599465578:84 , srv219505189:84 , srv461210485:84 , srv474383757:84 , srv500056315:84 , srv694682889:84 , srv819780327:84 , srv900068256:84 , srv936577654:84 , srv1233014118:85 , srv1233946041:85 , srv1366378540:85 , srv1462461291:85 , srv1469397589:85 , srv1569491415:85 , srv1808423165:85 , srv198217139:85 , srv226421956:85 , srv352234480:85 , srv419630628:85 , srv561154617:85 , srv786476156:85 , srv998619195:85 , srv104160571:86 , srv1420600466:86 , srv1580575639:86 , srv1758283938:86 , srv176273025:86 , srv239238052:86 , srv694385826:86 , srv721701402:86 , srv1069351860:87 , srv1135642186:87 , srv1288627709:87 , srv1552847940:87 , srv1825585239:87 , srv2005932792:87 , srv280408145:87 , srv289675077:87 , srv416357454:87 , srv803872292:87 , srv817825699:87 , srv994989412:87 , srv1136896670:88 , srv1148728588:88 , srv1592596358:88 , srv2048068438:88 , srv2091836025:88 , srv353336489:88 , srv422051170:88 , srv865968290:88 , srv1101919741:89 , srv1475089606:89 , srv1673856823:89 , srv1728761992:89 , srv1741109537:89 , srv2021856643:89 , srv2050057174:89 , srv28013559:89 , srv5553139:89 , srv819029373:89 , srv858688916:89 , srv1102590333:90 , srv1192550867:90 , srv1328327538:90 , srv166791497:90 , srv815095370:90 , srv1599674213:91 , srv1918638166:92 , srv460607717:93 , srv2008426936:535 }
2023-11-13 08:55:13,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-13 08:55:13,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1318950717=17, srv721701402=86, srv1462461291=23, srv694385826=82, srv280408145=62, srv994989412=98, srv1192550867=10, srv718472509=84, srv1808423165=46, srv1672027151=35, srv513768960=75, srv1592596358=31, srv936577654=96, srv2021856643=53, srv669773518=80, srv104160571=0, srv176273025=44, srv1580575639=30, srv1101919741=4, srv2099022303=57, srv817825699=90, srv1082417507=2, srv865968290=94, srv1505426902=26, srv416357454=68, srv983862975=97, srv1087243005=3, srv1685069120=37, srv1569491415=29, srv561154617=77, srv1102590333=5, srv461210485=72, srv1069351860=1, srv198217139=50, srv1703833936=39, srv353336489=66, srv1136896670=7, srv1405724148=21, srv819029373=91, srv900068256=95, srv226421956=59, srv1135642186=6, srv347487995=64, srv998619195=99, srv239238052=60, srv1153847990=9, srv122076593=11, srv2005932792=51, srv171021637=40, srv1741109537=42, srv166791497=34, srv720825849=85, srv1325007117=18, srv419630628=69, srv671016393=81, srv803872292=88, srv1328327538=19, srv1673856823=36, srv1288627709=16, srv1222373313=12, srv422051170=70, srv1599674213=33, srv815095370=89, srv474383757=73, srv219505189=58, srv1806741123=45, srv694682889=83, srv1552847940=28, srv1728761992=41, srv1233014118=13, srv1148728588=8, srv1599465578=32, srv1693055652=38, srv500056315=74, srv2091836025=56, srv28013559=61, srv352234480=65, srv2048068438=54, srv1233946041=14, srv2050057174=55, srv1420600466=22, srv1537339291=27, srv1758283938=43, srv1918638166=49, srv1475089606=25, srv2008426936=52, srv786476156=87, srv5553139=76, srv406793401=67, srv460607717=71, srv599115434=79, srv819780327=92, srv1905667136=48, srv1825585239=47, srv858688916=93, srv565602939=78, srv128101194=15, srv1469397589=24, srv1366378540=20, srv289675077=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-13 08:55:13,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-13 08:55:13,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-13 08:55:13,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-13 08:55:13,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-13 08:55:13,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-13 08:55:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-13 08:55:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-13 08:55:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-13 08:55:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-13 08:55:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-13 08:55:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-13 08:55:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-13 08:55:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-13 08:55:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-13 08:55:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-13 08:55:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-13 08:55:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-13 08:55:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-13 08:55:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-13 08:55:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-13 08:55:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-13 08:55:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-13 08:55:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-13 08:55:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-13 08:55:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-13 08:55:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-13 08:55:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-13 08:55:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-13 08:55:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-13 08:55:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-13 08:55:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-13 08:55:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-13 08:55:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-13 08:55:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-13 08:55:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-13 08:55:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-13 08:55:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-13 08:55:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-13 08:55:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-13 08:55:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-13 08:55:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-13 08:55:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-13 08:55:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-13 08:55:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-13 08:55:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-13 08:55:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-13 08:55:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-13 08:55:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-13 08:55:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-13 08:55:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-13 08:55:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-13 08:55:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-13 08:55:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-13 08:55:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-13 08:55:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-13 08:55:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-13 08:55:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-13 08:55:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-13 08:55:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-13 08:55:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-13 08:55:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-13 08:55:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-13 08:55:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-13 08:55:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-13 08:55:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-13 08:55:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-13 08:55:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-13 08:55:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-13 08:55:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-13 08:55:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-13 08:55:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-13 08:55:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-13 08:55:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-13 08:55:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-13 08:55:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-13 08:55:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-13 08:55:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-13 08:55:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-13 08:55:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-13 08:55:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-13 08:55:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-13 08:55:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-13 08:55:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-13 08:55:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-13 08:55:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-13 08:55:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-13 08:55:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-13 08:55:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-13 08:55:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-13 08:55:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-13 08:55:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-13 08:55:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-13 08:55:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-13 08:55:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-13 08:55:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-13 08:55:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-13 08:55:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-13 08:55:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-13 08:55:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-13 08:55:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-13 08:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-13 08:55:13,596 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0215110675162575); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03879787329147873, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0215110675162575); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012166666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04475, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-13 08:55:13,596 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-13 08:55:13,596 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015260351147704779, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0215110675162575); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03879787329147873, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0215110675162575); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012166666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04475, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-13 08:55:13,812 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 317 ms to try 5000 different iterations.  Found a solution that moves 980 regions; Going from a computed imbalance of 0.015260351147704779 to a new imbalance of 6.560046303316354E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0070023956850382004); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01005037815259212); MoveCostFunction : (multiplier=7.0, imbalance=0.10888888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0070023956850382004); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-13 08:55:13,813 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1288627709:78 , srv815095370:81 , srv1192550867:82 , srv2021856643:82 , srv460607717:82 , srv1101919741:83 , srv1673856823:83 , srv171021637:83 , srv1918638166:83 , srv198217139:83 , srv28013559:83 , srv694682889:83 , srv1102590333:84 , srv1135642186:84 , srv1233014118:84 , srv1328327538:84 , srv1366378540:84 , srv1552847940:84 , srv1599674213:84 , srv166791497:84 , srv239238052:84 , srv353336489:84 , srv416357454:84 , srv474383757:84 , srv671016393:84 , srv720825849:84 , srv819029373:84 , srv819780327:84 , srv994989412:84 , srv1148728588:85 , srv1233946041:85 , srv1318950717:85 , srv1405724148:85 , srv1420600466:85 , srv1475089606:85 , srv1537339291:85 , srv1569491415:85 , srv1580575639:85 , srv1728761992:85 , srv1825585239:85 , srv2048068438:85 , srv2099022303:85 , srv352234480:85 , srv406793401:85 , srv419630628:85 , srv5553139:85 , srv565602939:85 , srv669773518:85 , srv817825699:85 , srv858688916:85 , srv1069351860:86 , srv1087243005:86 , srv1153847990:86 , srv122076593:86 , srv128101194:86 , srv1325007117:86 , srv1469397589:86 , srv1505426902:86 , srv1592596358:86 , srv1672027151:86 , srv1693055652:86 , srv1703833936:86 , srv1758283938:86 , srv176273025:86 , srv2005932792:86 , srv2091836025:86 , srv226421956:86 , srv289675077:86 , srv347487995:86 , srv422051170:86 , srv461210485:86 , srv694385826:86 , srv786476156:86 , srv803872292:86 , srv998619195:86 , srv1082417507:87 , srv1136896670:87 , srv1222373313:87 , srv1599465578:87 , srv1685069120:87 , srv1741109537:87 , srv1806741123:87 , srv1808423165:87 , srv1905667136:87 , srv219505189:87 , srv280408145:87 , srv561154617:87 , srv599115434:87 , srv721701402:87 , srv865968290:87 , srv936577654:87 , srv104160571:88 , srv1462461291:88 , srv2050057174:88 , srv500056315:88 , srv513768960:88 , srv718472509:88 , srv900068256:88 , srv983862975:88 , srv2008426936:555 }
2023-11-13 08:55:13,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-13 08:55:13,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1318950717=17, srv721701402=86, srv1462461291=23, srv694385826=82, srv280408145=62, srv994989412=98, srv1192550867=10, srv718472509=84, srv1808423165=46, srv1672027151=35, srv513768960=75, srv1592596358=31, srv936577654=96, srv2021856643=53, srv669773518=80, srv104160571=0, srv176273025=44, srv1580575639=30, srv1101919741=4, srv2099022303=57, srv817825699=90, srv1082417507=2, srv865968290=94, srv1505426902=26, srv416357454=68, srv983862975=97, srv1087243005=3, srv1685069120=37, srv1569491415=29, srv561154617=77, srv1102590333=5, srv461210485=72, srv1069351860=1, srv198217139=50, srv1703833936=39, srv353336489=66, srv1136896670=7, srv1405724148=21, srv819029373=91, srv900068256=95, srv226421956=59, srv1135642186=6, srv347487995=64, srv998619195=99, srv239238052=60, srv1153847990=9, srv122076593=11, srv2005932792=51, srv171021637=40, srv1741109537=42, srv166791497=34, srv720825849=85, srv1325007117=18, srv419630628=69, srv671016393=81, srv803872292=88, srv1328327538=19, srv1673856823=36, srv1288627709=16, srv1222373313=12, srv422051170=70, srv1599674213=33, srv815095370=89, srv474383757=73, srv219505189=58, srv1806741123=45, srv694682889=83, srv1552847940=28, srv1728761992=41, srv1233014118=13, srv1148728588=8, srv1599465578=32, srv1693055652=38, srv500056315=74, srv2091836025=56, srv28013559=61, srv352234480=65, srv2048068438=54, srv1233946041=14, srv2050057174=55, srv1420600466=22, srv1537339291=27, srv1758283938=43, srv1918638166=49, srv1475089606=25, srv2008426936=52, srv786476156=87, srv5553139=76, srv406793401=67, srv460607717=71, srv599115434=79, srv819780327=92, srv1905667136=48, srv1825585239=47, srv858688916=93, srv565602939=78, srv128101194=15, srv1469397589=24, srv1366378540=20, srv289675077=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-13 08:55:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-13 08:55:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-13 08:55:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-13 08:55:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-13 08:55:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-13 08:55:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-13 08:55:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-13 08:55:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-13 08:55:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-13 08:55:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-13 08:55:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-13 08:55:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-13 08:55:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-13 08:55:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-13 08:55:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-13 08:55:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-13 08:55:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-13 08:55:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-13 08:55:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-13 08:55:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-13 08:55:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-13 08:55:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-13 08:55:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-13 08:55:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-13 08:55:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-13 08:55:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-13 08:55:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-13 08:55:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-13 08:55:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-13 08:55:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-13 08:55:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-13 08:55:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-13 08:55:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-13 08:55:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-13 08:55:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-13 08:55:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-13 08:55:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-13 08:55:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-13 08:55:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-13 08:55:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-13 08:55:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-13 08:55:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-13 08:55:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-13 08:55:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-13 08:55:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-13 08:55:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-13 08:55:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-13 08:55:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-13 08:55:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-13 08:55:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-13 08:55:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-13 08:55:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-13 08:55:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-13 08:55:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-13 08:55:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-13 08:55:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-13 08:55:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-13 08:55:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-13 08:55:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-13 08:55:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-13 08:55:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-13 08:55:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-13 08:55:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-13 08:55:13,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-13 08:55:13,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-13 08:55:13,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-13 08:55:13,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-13 08:55:13,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-13 08:55:13,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-13 08:55:13,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-13 08:55:13,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-13 08:55:13,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-13 08:55:13,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-13 08:55:13,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-13 08:55:13,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-13 08:55:13,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-13 08:55:13,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-13 08:55:13,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-13 08:55:13,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-13 08:55:13,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-13 08:55:13,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-13 08:55:13,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-13 08:55:13,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-13 08:55:13,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-13 08:55:13,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-13 08:55:13,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-13 08:55:13,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-13 08:55:13,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-13 08:55:13,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-13 08:55:13,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-13 08:55:13,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-13 08:55:13,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-13 08:55:13,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-13 08:55:13,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-13 08:55:13,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-13 08:55:13,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-13 08:55:13,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-13 08:55:13,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-13 08:55:13,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-13 08:55:13,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-13 08:55:13,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-13 08:55:13,942 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0070023956850382004); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01005037815259212); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0070023956850382004); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-13 08:55:13,942 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-13 08:55:13,942 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.491425236648887E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0070023956850382004); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01005037815259212); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0070023956850382004); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-13 08:55:14,265 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 423 ms to try 5000 different iterations.  Found a solution that moves 257 regions; Going from a computed imbalance of 6.491425236648887E-4 to a new imbalance of 1.3488547801285635E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.028555555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-13 08:55:14,266 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv718472509:82 , srv1808423165:83 , srv599115434:83 , srv819780327:83 , srv865968290:83 , srv104160571:84 , srv1136896670:84 , srv128101194:84 , srv1318950717:84 , srv1462461291:84 , srv1475089606:84 , srv1580575639:84 , srv1592596358:84 , srv1599465578:84 , srv1599674213:84 , srv166791497:84 , srv1673856823:84 , srv1741109537:84 , srv1758283938:84 , srv176273025:84 , srv2048068438:84 , srv2050057174:84 , srv2091836025:84 , srv219505189:84 , srv239238052:84 , srv280408145:84 , srv289675077:84 , srv353336489:84 , srv422051170:84 , srv474383757:84 , srv500056315:84 , srv513768960:84 , srv5553139:84 , srv669773518:84 , srv694385826:84 , srv721701402:84 , srv803872292:84 , srv815095370:84 , srv817825699:84 , srv900068256:84 , srv998619195:84 , srv1069351860:85 , srv1082417507:85 , srv1087243005:85 , srv1101919741:85 , srv1102590333:85 , srv1135642186:85 , srv1148728588:85 , srv1153847990:85 , srv1192550867:85 , srv122076593:85 , srv1222373313:85 , srv1233014118:85 , srv1233946041:85 , srv1288627709:85 , srv1328327538:85 , srv1366378540:85 , srv1405724148:85 , srv1469397589:85 , srv1505426902:85 , srv1552847940:85 , srv1569491415:85 , srv1672027151:85 , srv1685069120:85 , srv1693055652:85 , srv1703833936:85 , srv1806741123:85 , srv1825585239:85 , srv1905667136:85 , srv1918638166:85 , srv198217139:85 , srv2005932792:85 , srv2021856643:85 , srv2099022303:85 , srv226421956:85 , srv28013559:85 , srv347487995:85 , srv406793401:85 , srv416357454:85 , srv460607717:85 , srv461210485:85 , srv561154617:85 , srv671016393:85 , srv720825849:85 , srv786476156:85 , srv819029373:85 , srv858688916:85 , srv936577654:85 , srv983862975:85 , srv994989412:85 , srv1325007117:86 , srv1420600466:86 , srv1537339291:86 , srv171021637:86 , srv1728761992:86 , srv352234480:86 , srv419630628:86 , srv565602939:86 , srv694682889:86 , srv2008426936:623 }
2023-11-13 08:55:14,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-13 08:55:14,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1318950717=17, srv721701402=86, srv1462461291=23, srv694385826=82, srv280408145=62, srv994989412=98, srv1192550867=10, srv718472509=84, srv1808423165=46, srv1672027151=35, srv513768960=75, srv1592596358=31, srv936577654=96, srv2021856643=53, srv669773518=80, srv104160571=0, srv176273025=44, srv1580575639=30, srv1101919741=4, srv2099022303=57, srv817825699=90, srv1082417507=2, srv865968290=94, srv1505426902=26, srv416357454=68, srv983862975=97, srv1087243005=3, srv1685069120=37, srv1569491415=29, srv561154617=77, srv1102590333=5, srv461210485=72, srv1069351860=1, srv198217139=50, srv1703833936=39, srv353336489=66, srv1136896670=7, srv1405724148=21, srv819029373=91, srv900068256=95, srv226421956=59, srv1135642186=6, srv347487995=64, srv998619195=99, srv239238052=60, srv1153847990=9, srv122076593=11, srv2005932792=51, srv171021637=40, srv1741109537=42, srv166791497=34, srv720825849=85, srv1325007117=18, srv419630628=69, srv671016393=81, srv803872292=88, srv1328327538=19, srv1673856823=36, srv1288627709=16, srv1222373313=12, srv422051170=70, srv1599674213=33, srv815095370=89, srv474383757=73, srv219505189=58, srv1806741123=45, srv694682889=83, srv1552847940=28, srv1728761992=41, srv1233014118=13, srv1148728588=8, srv1599465578=32, srv1693055652=38, srv500056315=74, srv2091836025=56, srv28013559=61, srv352234480=65, srv2048068438=54, srv1233946041=14, srv2050057174=55, srv1420600466=22, srv1537339291=27, srv1758283938=43, srv1918638166=49, srv1475089606=25, srv2008426936=52, srv786476156=87, srv5553139=76, srv406793401=67, srv460607717=71, srv599115434=79, srv819780327=92, srv1905667136=48, srv1825585239=47, srv858688916=93, srv565602939=78, srv128101194=15, srv1469397589=24, srv1366378540=20, srv289675077=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-13 08:55:14,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-13 08:55:14,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-13 08:55:14,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-13 08:55:14,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-13 08:55:14,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-13 08:55:14,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-13 08:55:14,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-13 08:55:14,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-13 08:55:14,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-13 08:55:14,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-13 08:55:14,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-13 08:55:14,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-13 08:55:14,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-13 08:55:14,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-13 08:55:14,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-13 08:55:14,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-13 08:55:14,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-13 08:55:14,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-13 08:55:14,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-13 08:55:14,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-13 08:55:14,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-13 08:55:14,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-13 08:55:14,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-13 08:55:14,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-13 08:55:14,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-13 08:55:14,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-13 08:55:14,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-13 08:55:14,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-13 08:55:14,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-13 08:55:14,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-13 08:55:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-13 08:55:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-13 08:55:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-13 08:55:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-13 08:55:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-13 08:55:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-13 08:55:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-13 08:55:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-13 08:55:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-13 08:55:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-13 08:55:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-13 08:55:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-13 08:55:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-13 08:55:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-13 08:55:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-13 08:55:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-13 08:55:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-13 08:55:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-13 08:55:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-13 08:55:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-13 08:55:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-13 08:55:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-13 08:55:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-13 08:55:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-13 08:55:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-13 08:55:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-13 08:55:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-13 08:55:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-13 08:55:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-13 08:55:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-13 08:55:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-13 08:55:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-13 08:55:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-13 08:55:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-13 08:55:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-13 08:55:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-13 08:55:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-13 08:55:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-13 08:55:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-13 08:55:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-13 08:55:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-13 08:55:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-13 08:55:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-13 08:55:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-13 08:55:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-13 08:55:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-13 08:55:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-13 08:55:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-13 08:55:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-13 08:55:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-13 08:55:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-13 08:55:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-13 08:55:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-13 08:55:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-13 08:55:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-13 08:55:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-13 08:55:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-13 08:55:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-13 08:55:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-13 08:55:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-13 08:55:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-13 08:55:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-13 08:55:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-13 08:55:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-13 08:55:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-13 08:55:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-13 08:55:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-13 08:55:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-13 08:55:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-13 08:55:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-13 08:55:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-13 08:55:14,398 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.1688995338679613E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-13 08:55:14,399 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv718472509:82 , srv1808423165:83 , srv599115434:83 , srv819780327:83 , srv865968290:83 , srv104160571:84 , srv1136896670:84 , srv128101194:84 , srv1318950717:84 , srv1462461291:84 , srv1475089606:84 , srv1580575639:84 , srv1592596358:84 , srv1599465578:84 , srv1599674213:84 , srv166791497:84 , srv1673856823:84 , srv1741109537:84 , srv1758283938:84 , srv176273025:84 , srv2048068438:84 , srv2050057174:84 , srv2091836025:84 , srv219505189:84 , srv239238052:84 , srv280408145:84 , srv289675077:84 , srv353336489:84 , srv422051170:84 , srv474383757:84 , srv500056315:84 , srv513768960:84 , srv5553139:84 , srv669773518:84 , srv694385826:84 , srv721701402:84 , srv803872292:84 , srv815095370:84 , srv817825699:84 , srv900068256:84 , srv998619195:84 , srv1069351860:85 , srv1082417507:85 , srv1087243005:85 , srv1101919741:85 , srv1102590333:85 , srv1135642186:85 , srv1148728588:85 , srv1153847990:85 , srv1192550867:85 , srv122076593:85 , srv1222373313:85 , srv1233014118:85 , srv1233946041:85 , srv1288627709:85 , srv1328327538:85 , srv1366378540:85 , srv1405724148:85 , srv1469397589:85 , srv1505426902:85 , srv1552847940:85 , srv1569491415:85 , srv1672027151:85 , srv1685069120:85 , srv1693055652:85 , srv1703833936:85 , srv1806741123:85 , srv1825585239:85 , srv1905667136:85 , srv1918638166:85 , srv198217139:85 , srv2005932792:85 , srv2021856643:85 , srv2099022303:85 , srv226421956:85 , srv28013559:85 , srv347487995:85 , srv406793401:85 , srv416357454:85 , srv460607717:85 , srv461210485:85 , srv561154617:85 , srv671016393:85 , srv720825849:85 , srv786476156:85 , srv819029373:85 , srv858688916:85 , srv936577654:85 , srv983862975:85 , srv994989412:85 , srv1325007117:86 , srv1420600466:86 , srv1537339291:86 , srv171021637:86 , srv1728761992:86 , srv352234480:86 , srv419630628:86 , srv565602939:86 , srv694682889:86 , srv2008426936:623 }
2023-11-13 08:55:14,428 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=151 (was 121) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=3080 (was 3571)
2023-11-13 08:55:14,436 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=151, ProcessCount=169, AvailableMemoryMB=3080
2023-11-13 08:55:14,437 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-13 08:55:14,437 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-11-13 08:55:14,449 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1092386805:84 , srv1236807898:84 , srv1371265444:84 , srv139724345:84 , srv1433472109:84 , srv1654448083:84 , srv1676675337:84 , srv1713767277:84 , srv1853429727:84 , srv185470962:84 , srv1858581667:84 , srv1885935825:84 , srv1893815518:84 , srv2045095842:84 , srv2094220696:84 , srv237597360:84 , srv280535841:84 , srv334343617:84 , srv375220360:84 , srv394302300:84 , srv552164162:84 , srv600400885:84 , srv62135770:84 , srv665975768:84 , srv714502973:84 , srv733084857:84 , srv815168733:84 , srv866618635:84 , srv869404408:84 , srv151932809:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-11-13 08:55:14,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-13 08:55:14,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv185470962=10, srv866618635=28, srv1858581667=11, srv1092386805=0, srv1713767277=8, srv334343617=18, srv869404408=29, srv1654448083=6, srv1885935825=12, srv665975768=24, srv394302300=20, srv1853429727=9, srv375220360=19, srv815168733=27, srv237597360=16, srv62135770=23, srv1676675337=7, srv600400885=22, srv2094220696=15, srv733084857=26, srv151932809=5, srv1433472109=4, srv139724345=3, srv280535841=17, srv552164162=21, srv1236807898=1, srv2045095842=14, srv1893815518=13, srv714502973=25, srv1371265444=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-13 08:55:14,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-13 08:55:14,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-13 08:55:14,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-13 08:55:14,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-13 08:55:14,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-13 08:55:14,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-13 08:55:14,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-13 08:55:14,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-13 08:55:14,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-13 08:55:14,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-13 08:55:14,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-13 08:55:14,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-13 08:55:14,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-13 08:55:14,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-13 08:55:14,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-13 08:55:14,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-13 08:55:14,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-13 08:55:14,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-13 08:55:14,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-13 08:55:14,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-13 08:55:14,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-13 08:55:14,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-13 08:55:14,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-13 08:55:14,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-13 08:55:14,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-13 08:55:14,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-13 08:55:14,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-13 08:55:14,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-13 08:55:14,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-13 08:55:14,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-13 08:55:14,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-13 08:55:14,487 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0869668313233152, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-13 08:55:14,487 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912467072910052, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0869668313233152, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-11-13 08:57:14,466 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6306413 different iterations.  Found a solution that moves 1846 regions; Going from a computed imbalance of 0.9912467072910052 to a new imbalance of 4.308656090753194E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6837037037037037, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-13 08:57:14,472 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1092386805:90 , srv1236807898:90 , srv1371265444:90 , srv139724345:90 , srv1433472109:90 , srv151932809:90 , srv1654448083:90 , srv1676675337:90 , srv1713767277:90 , srv1853429727:90 , srv185470962:90 , srv1858581667:90 , srv1885935825:90 , srv1893815518:90 , srv2045095842:90 , srv2094220696:90 , srv237597360:90 , srv280535841:90 , srv334343617:90 , srv375220360:90 , srv394302300:90 , srv552164162:90 , srv600400885:90 , srv62135770:90 , srv665975768:90 , srv714502973:90 , srv733084857:90 , srv815168733:90 , srv866618635:90 , srv869404408:90 }
2023-11-13 08:57:14,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-13 08:57:14,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv185470962=10, srv866618635=28, srv1858581667=11, srv1092386805=0, srv1713767277=8, srv334343617=18, srv869404408=29, srv1654448083=6, srv1885935825=12, srv665975768=24, srv394302300=20, srv1853429727=9, srv375220360=19, srv815168733=27, srv237597360=16, srv62135770=23, srv1676675337=7, srv600400885=22, srv2094220696=15, srv733084857=26, srv151932809=5, srv1433472109=4, srv139724345=3, srv280535841=17, srv552164162=21, srv1236807898=1, srv2045095842=14, srv1893815518=13, srv714502973=25, srv1371265444=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-13 08:57:14,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-13 08:57:14,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-13 08:57:14,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-13 08:57:14,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-13 08:57:14,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-13 08:57:14,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-13 08:57:14,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-13 08:57:14,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-13 08:57:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-13 08:57:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-13 08:57:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-13 08:57:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-13 08:57:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-13 08:57:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-13 08:57:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-13 08:57:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-13 08:57:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-13 08:57:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-13 08:57:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-13 08:57:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-13 08:57:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-13 08:57:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-13 08:57:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-13 08:57:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-13 08:57:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-13 08:57:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-13 08:57:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-13 08:57:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-13 08:57:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-13 08:57:14,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-13 08:57:14,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-13 08:57:14,489 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-13 08:57:14,489 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1092386805:90 , srv1236807898:90 , srv1371265444:90 , srv139724345:90 , srv1433472109:90 , srv151932809:90 , srv1654448083:90 , srv1676675337:90 , srv1713767277:90 , srv1853429727:90 , srv185470962:90 , srv1858581667:90 , srv1885935825:90 , srv1893815518:90 , srv2045095842:90 , srv2094220696:90 , srv237597360:90 , srv280535841:90 , srv334343617:90 , srv375220360:90 , srv394302300:90 , srv552164162:90 , srv600400885:90 , srv62135770:90 , srv665975768:90 , srv714502973:90 , srv733084857:90 , srv815168733:90 , srv866618635:90 , srv869404408:90 }
2023-11-13 08:57:14,501 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=156 (was 151) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 169), AvailableMemoryMB=3058 (was 3080)