2023-05-10 03:53:38,695 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-05-10 03:53:39,029 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-05-10 03:53:39,142 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-05-10 03:53:39,207 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-10 03:53:39,214 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-05-10 03:53:39,227 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=377, ProcessCount=171, AvailableMemoryMB=11036
2023-05-10 03:53:39,229 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-10 03:53:39,229 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-05-10 03:53:39,366 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1019180119:84 , srv1022434489:84 , srv1047172419:84 , srv1082236040:84 , srv1084043961:84 , srv1093441677:84 , srv1101618161:84 , srv111035028:84 , srv1120538450:84 , srv1149601767:84 , srv1181167898:84 , srv1259739912:84 , srv1273080607:84 , srv1294610183:84 , srv1313410833:84 , srv1315164689:84 , srv1319436259:84 , srv1320118400:84 , srv133175909:84 , srv1343413333:84 , srv134682157:84 , srv1356274689:84 , srv1357199025:84 , srv1396529305:84 , srv1418659040:84 , srv142139550:84 , srv1428594470:84 , srv1463286021:84 , srv1535355935:84 , srv1535837876:84 , srv154837443:84 , srv1579074811:84 , srv1616179480:84 , srv1638958482:84 , srv1654732250:84 , srv1660272222:84 , srv1675488616:84 , srv1699784919:84 , srv1714957000:84 , srv1737606680:84 , srv1743742048:84 , srv1747792537:84 , srv1781965495:84 , srv1787066361:84 , srv1803964967:84 , srv1819875134:84 , srv18474103:84 , srv1897507850:84 , srv1899275178:84 , srv192509774:84 , srv193278736:84 , srv1937993421:84 , srv1938410670:84 , srv1972950140:84 , srv2019583655:84 , srv2062672354:84 , srv2079013538:84 , srv2080074526:84 , srv2091829182:84 , srv2118511252:84 , srv2131912310:84 , srv2133372914:84 , srv2135338812:84 , srv340886895:84 , srv352278526:84 , srv357787227:84 , srv374004340:84 , srv403509454:84 , srv408848222:84 , srv437535411:84 , srv43967776:84 , srv471614609:84 , srv499710671:84 , srv508149291:84 , srv515538169:84 , srv535113807:84 , srv543964527:84 , srv548571276:84 , srv555038431:84 , srv586394118:84 , srv588405267:84 , srv6366723:84 , srv655810377:84 , srv665281540:84 , srv66941151:84 , srv719098409:84 , srv758995920:84 , srv85372653:84 , srv864079373:84 , srv874956761:84 , srv877893443:84 , srv880688458:84 , srv899289723:84 , srv901334138:84 , srv930797978:84 , srv937796707:84 , srv961965243:84 , srv968588902:84 , srv974691480:84 , srv872720629:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-05-10 03:53:39,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-10 03:53:39,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1699784919=37, srv1747792537=41, srv1938410670=52, srv437535411=69, srv1803964967=44, srv665281540=83, srv134682157=20, srv193278736=50, srv1972950140=53, srv499710671=72, srv543964527=76, srv1660272222=35, srv142139550=25, srv1463286021=27, srv758995920=86, srv340886895=63, srv1675488616=36, srv899289723=93, srv877893443=91, srv1937993421=51, srv1357199025=22, srv930797978=95, srv133175909=18, srv2131912310=60, srv1120538450=8, srv880688458=92, srv1428594470=26, srv111035028=7, srv66941151=84, srv1149601767=9, srv2133372914=61, srv548571276=77, srv2062672354=55, srv408848222=68, srv403509454=67, srv555038431=78, srv1396529305=23, srv1343413333=19, srv154837443=30, srv1897507850=47, srv719098409=85, srv1019180119=0, srv872720629=89, srv1737606680=39, srv1259739912=11, srv18474103=46, srv508149291=73, srv6366723=81, srv1638958482=33, srv1654732250=34, srv874956761=90, srv968588902=98, srv1616179480=32, srv471614609=71, srv1819875134=45, srv85372653=87, srv1181167898=10, srv2079013538=56, srv2080074526=57, srv588405267=80, srv43967776=70, srv1101618161=6, srv1899275178=48, srv1535355935=28, srv2019583655=54, srv1787066361=43, srv1294610183=13, srv2135338812=62, srv1320118400=17, srv352278526=64, srv901334138=94, srv864079373=88, srv515538169=74, srv1047172419=2, srv974691480=99, srv374004340=66, srv1315164689=15, srv1535837876=29, srv1418659040=24, srv1579074811=31, srv1356274689=21, srv2091829182=58, srv961965243=97, srv1743742048=40, srv1714957000=38, srv357787227=65, srv1093441677=5, srv1022434489=1, srv535113807=75, srv655810377=82, srv937796707=96, srv586394118=79, srv1084043961=4, srv1781965495=42, srv1273080607=12, srv192509774=49, srv1313410833=14, srv1319436259=16, srv2118511252=59, srv1082236040=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-10 03:53:39,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-10 03:53:39,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-10 03:53:39,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-10 03:53:39,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-10 03:53:39,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-10 03:53:39,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-10 03:53:39,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-10 03:53:39,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-10 03:53:39,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-10 03:53:39,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-10 03:53:39,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-10 03:53:39,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-10 03:53:39,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-10 03:53:39,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-10 03:53:39,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-10 03:53:39,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-10 03:53:39,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-10 03:53:39,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-10 03:53:39,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-10 03:53:39,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-10 03:53:39,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-10 03:53:39,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-10 03:53:39,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-10 03:53:39,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-10 03:53:39,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-10 03:53:39,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-10 03:53:39,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-10 03:53:39,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-10 03:53:39,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-10 03:53:39,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-10 03:53:39,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-10 03:53:39,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-10 03:53:39,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-10 03:53:39,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-10 03:53:39,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-10 03:53:39,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-10 03:53:39,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-10 03:53:39,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-10 03:53:39,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-10 03:53:39,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-10 03:53:39,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-10 03:53:39,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-10 03:53:39,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-10 03:53:39,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-10 03:53:39,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-10 03:53:39,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-10 03:53:39,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-10 03:53:39,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-10 03:53:39,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-10 03:53:39,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-10 03:53:39,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-10 03:53:39,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-10 03:53:39,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-10 03:53:39,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-10 03:53:39,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-10 03:53:39,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-10 03:53:39,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-10 03:53:39,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-10 03:53:39,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-10 03:53:39,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-10 03:53:39,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-10 03:53:39,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-10 03:53:39,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-10 03:53:39,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-10 03:53:39,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-10 03:53:39,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-10 03:53:39,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-10 03:53:39,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-10 03:53:39,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-10 03:53:39,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-10 03:53:39,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-10 03:53:39,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-10 03:53:39,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-10 03:53:39,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-10 03:53:39,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-10 03:53:39,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-10 03:53:39,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-10 03:53:39,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-10 03:53:39,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-10 03:53:39,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-10 03:53:39,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-10 03:53:39,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-10 03:53:39,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-10 03:53:39,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-10 03:53:39,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-10 03:53:39,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-10 03:53:39,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-10 03:53:39,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-10 03:53:39,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-10 03:53:39,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-10 03:53:39,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-10 03:53:39,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-10 03:53:39,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-10 03:53:39,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-10 03:53:39,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-10 03:53:39,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-10 03:53:39,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-10 03:53:39,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-10 03:53:39,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-10 03:53:39,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-10 03:53:39,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-10 03:53:39,740 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-10 03:53:39,740 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-10 03:53:39,741 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-10 03:53:40,529 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1014 ms to try 5000 different iterations.  Found a solution that moves 4667 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1806830646838993. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04257194639405188, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05665379136755759, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5185555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04257194639405188, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17575, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.244, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-10 03:53:40,562 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1047172419:81 , srv1082236040:81 , srv1320118400:81 , srv1675488616:81 , srv1654732250:82 , srv1938410670:82 , srv515538169:82 , srv1343413333:83 , srv1356274689:83 , srv1357199025:83 , srv1819875134:83 , srv1897507850:83 , srv1937993421:83 , srv2118511252:83 , srv2135338812:83 , srv43967776:83 , srv66941151:83 , srv968588902:83 , srv1101618161:84 , srv1418659040:84 , srv1535355935:84 , srv1972950140:84 , srv2091829182:84 , srv374004340:84 , srv437535411:84 , srv499710671:84 , srv543964527:84 , srv899289723:84 , srv937796707:84 , srv111035028:85 , srv1315164689:85 , srv1319436259:85 , srv1428594470:85 , srv1699784919:85 , srv1714957000:85 , srv2131912310:85 , srv408848222:85 , srv471614609:85 , srv655810377:85 , srv665281540:85 , srv758995920:85 , srv880688458:85 , srv901334138:85 , srv930797978:85 , srv1120538450:86 , srv1396529305:86 , srv1463286021:86 , srv154837443:86 , srv1579074811:86 , srv1616179480:86 , srv1660272222:86 , srv1737606680:86 , srv1743742048:86 , srv1747792537:86 , srv1803964967:86 , srv2133372914:86 , srv555038431:86 , srv6366723:86 , srv864079373:86 , srv874956761:86 , srv877893443:86 , srv1093441677:87 , srv1181167898:87 , srv1781965495:87 , srv18474103:87 , srv1899275178:87 , srv2062672354:87 , srv2079013538:87 , srv340886895:87 , srv352278526:87 , srv403509454:87 , srv548571276:87 , srv586394118:87 , srv85372653:87 , srv974691480:87 , srv1019180119:88 , srv133175909:88 , srv357787227:88 , srv508149291:88 , srv535113807:88 , srv588405267:88 , srv719098409:88 , srv1022434489:89 , srv1084043961:89 , srv1273080607:89 , srv134682157:89 , srv1535837876:89 , srv1787066361:89 , srv192509774:89 , srv2019583655:89 , srv1313410833:90 , srv1259739912:91 , srv193278736:91 , srv1294610183:92 , srv1638958482:92 , srv961965243:94 , srv2080074526:96 , srv1149601767:98 , srv142139550:98 , srv872720629:468 }
2023-05-10 03:53:40,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-10 03:53:40,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1699784919=37, srv1747792537=41, srv1938410670=52, srv437535411=69, srv1803964967=44, srv665281540=83, srv134682157=20, srv193278736=50, srv1972950140=53, srv499710671=72, srv543964527=76, srv1660272222=35, srv142139550=25, srv1463286021=27, srv758995920=86, srv340886895=63, srv1675488616=36, srv899289723=93, srv877893443=91, srv1937993421=51, srv1357199025=22, srv930797978=95, srv133175909=18, srv2131912310=60, srv1120538450=8, srv880688458=92, srv1428594470=26, srv111035028=7, srv66941151=84, srv1149601767=9, srv2133372914=61, srv548571276=77, srv2062672354=55, srv408848222=68, srv403509454=67, srv555038431=78, srv1396529305=23, srv1343413333=19, srv154837443=30, srv1897507850=47, srv719098409=85, srv1019180119=0, srv872720629=89, srv1737606680=39, srv1259739912=11, srv18474103=46, srv508149291=73, srv6366723=81, srv1638958482=33, srv1654732250=34, srv874956761=90, srv968588902=98, srv1616179480=32, srv471614609=71, srv1819875134=45, srv85372653=87, srv1181167898=10, srv2079013538=56, srv2080074526=57, srv588405267=80, srv43967776=70, srv1101618161=6, srv1899275178=48, srv1535355935=28, srv2019583655=54, srv1787066361=43, srv1294610183=13, srv2135338812=62, srv1320118400=17, srv352278526=64, srv901334138=94, srv864079373=88, srv515538169=74, srv1047172419=2, srv974691480=99, srv374004340=66, srv1315164689=15, srv1535837876=29, srv1418659040=24, srv1579074811=31, srv1356274689=21, srv2091829182=58, srv961965243=97, srv1743742048=40, srv1714957000=38, srv357787227=65, srv1093441677=5, srv1022434489=1, srv535113807=75, srv655810377=82, srv937796707=96, srv586394118=79, srv1084043961=4, srv1781965495=42, srv1273080607=12, srv192509774=49, srv1313410833=14, srv1319436259=16, srv2118511252=59, srv1082236040=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-10 03:53:40,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-10 03:53:40,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-10 03:53:40,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-10 03:53:40,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-10 03:53:40,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-10 03:53:40,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-10 03:53:40,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-10 03:53:40,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-10 03:53:40,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-10 03:53:40,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-10 03:53:40,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-10 03:53:40,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-10 03:53:40,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-10 03:53:40,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-10 03:53:40,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-10 03:53:40,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-10 03:53:40,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-10 03:53:40,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-10 03:53:40,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-10 03:53:40,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-10 03:53:40,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-10 03:53:40,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-10 03:53:40,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-10 03:53:40,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-10 03:53:40,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-10 03:53:40,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-10 03:53:40,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-10 03:53:40,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-10 03:53:40,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-10 03:53:40,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-10 03:53:40,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-10 03:53:40,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-10 03:53:40,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-10 03:53:40,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-10 03:53:40,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-10 03:53:40,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-10 03:53:40,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-10 03:53:40,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-10 03:53:40,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-10 03:53:40,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-10 03:53:40,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-10 03:53:40,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-10 03:53:40,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-10 03:53:40,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-10 03:53:40,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-10 03:53:40,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-10 03:53:40,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-10 03:53:40,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-10 03:53:40,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-10 03:53:40,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-10 03:53:40,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-10 03:53:40,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-10 03:53:40,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-10 03:53:40,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-10 03:53:40,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-10 03:53:40,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-10 03:53:40,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-10 03:53:40,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-10 03:53:40,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-10 03:53:40,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-10 03:53:40,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-10 03:53:40,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-10 03:53:40,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-10 03:53:40,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-10 03:53:40,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-10 03:53:40,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-10 03:53:40,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-10 03:53:40,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-10 03:53:40,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-10 03:53:40,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-10 03:53:40,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-10 03:53:40,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-10 03:53:40,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-10 03:53:40,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-10 03:53:40,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-10 03:53:40,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-10 03:53:40,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-10 03:53:40,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-10 03:53:40,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-10 03:53:40,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-10 03:53:40,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-10 03:53:40,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-10 03:53:40,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-10 03:53:40,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-10 03:53:40,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-10 03:53:40,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-10 03:53:40,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-10 03:53:40,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-10 03:53:40,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-10 03:53:40,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-10 03:53:40,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-10 03:53:40,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-10 03:53:40,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-10 03:53:40,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-10 03:53:40,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-10 03:53:40,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-10 03:53:40,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-10 03:53:40,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-10 03:53:40,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-10 03:53:40,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-10 03:53:40,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-10 03:53:40,767 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04257194639405188, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05665379136755759, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04257194639405188, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17575, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.244, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-10 03:53:40,767 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-10 03:53:40,767 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18065038565143635, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04257194639405188, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05665379136755759, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04257194639405188, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17575, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.244, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-10 03:53:41,093 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 462 ms to try 5000 different iterations.  Found a solution that moves 2632 regions; Going from a computed imbalance of 0.18065038565143635 to a new imbalance of 0.01768836729009539. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02308320857289348); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032414958235299086, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29244444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02308320857289348); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.014916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-10 03:53:41,103 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1149601767:74 , srv142139550:74 , srv2080074526:77 , srv961965243:79 , srv1259739912:80 , srv1294610183:80 , srv1313410833:80 , srv1638958482:80 , srv1899275178:81 , srv192509774:81 , srv719098409:81 , srv1019180119:82 , srv134682157:82 , srv1535837876:82 , srv193278736:82 , srv357787227:82 , srv1273080607:83 , srv133175909:83 , srv2079013538:83 , srv508149291:83 , srv535113807:83 , srv586394118:83 , srv588405267:83 , srv1084043961:84 , srv1463286021:84 , srv6366723:84 , srv880688458:84 , srv930797978:84 , srv974691480:84 , srv1022434489:85 , srv1093441677:85 , srv1181167898:85 , srv1660272222:85 , srv1737606680:85 , srv18474103:85 , srv2019583655:85 , srv2062672354:85 , srv403509454:85 , srv1396529305:86 , srv1579074811:86 , srv1747792537:86 , srv1781965495:86 , srv1972950140:86 , srv2131912310:86 , srv352278526:86 , srv408848222:86 , srv437535411:86 , srv543964527:86 , srv548571276:86 , srv555038431:86 , srv665281540:86 , srv874956761:86 , srv877893443:86 , srv111035028:87 , srv1315164689:87 , srv1343413333:87 , srv154837443:87 , srv1699784919:87 , srv1787066361:87 , srv1803964967:87 , srv2091829182:87 , srv2118511252:87 , srv471614609:87 , srv499710671:87 , srv66941151:87 , srv758995920:87 , srv85372653:87 , srv864079373:87 , srv901334138:87 , srv937796707:87 , srv1101618161:88 , srv1120538450:88 , srv1356274689:88 , srv1714957000:88 , srv1897507850:88 , srv1938410670:88 , srv2133372914:88 , srv968588902:88 , srv1082236040:89 , srv1428594470:89 , srv1616179480:89 , srv1654732250:89 , srv1819875134:89 , srv1937993421:89 , srv340886895:89 , srv374004340:89 , srv655810377:89 , srv1319436259:90 , srv1535355935:90 , srv1675488616:90 , srv2135338812:90 , srv43967776:90 , srv515538169:90 , srv899289723:90 , srv1320118400:91 , srv1743742048:91 , srv1047172419:92 , srv1357199025:92 , srv1418659040:92 , srv872720629:511 }
2023-05-10 03:53:41,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-10 03:53:41,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1699784919=37, srv1747792537=41, srv1938410670=52, srv437535411=69, srv1803964967=44, srv665281540=83, srv134682157=20, srv193278736=50, srv1972950140=53, srv499710671=72, srv543964527=76, srv1660272222=35, srv142139550=25, srv1463286021=27, srv758995920=86, srv340886895=63, srv1675488616=36, srv899289723=93, srv877893443=91, srv1937993421=51, srv1357199025=22, srv930797978=95, srv133175909=18, srv2131912310=60, srv1120538450=8, srv880688458=92, srv1428594470=26, srv111035028=7, srv66941151=84, srv1149601767=9, srv2133372914=61, srv548571276=77, srv2062672354=55, srv408848222=68, srv403509454=67, srv555038431=78, srv1396529305=23, srv1343413333=19, srv154837443=30, srv1897507850=47, srv719098409=85, srv1019180119=0, srv872720629=89, srv1737606680=39, srv1259739912=11, srv18474103=46, srv508149291=73, srv6366723=81, srv1638958482=33, srv1654732250=34, srv874956761=90, srv968588902=98, srv1616179480=32, srv471614609=71, srv1819875134=45, srv85372653=87, srv1181167898=10, srv2079013538=56, srv2080074526=57, srv588405267=80, srv43967776=70, srv1101618161=6, srv1899275178=48, srv1535355935=28, srv2019583655=54, srv1787066361=43, srv1294610183=13, srv2135338812=62, srv1320118400=17, srv352278526=64, srv901334138=94, srv864079373=88, srv515538169=74, srv1047172419=2, srv974691480=99, srv374004340=66, srv1315164689=15, srv1535837876=29, srv1418659040=24, srv1579074811=31, srv1356274689=21, srv2091829182=58, srv961965243=97, srv1743742048=40, srv1714957000=38, srv357787227=65, srv1093441677=5, srv1022434489=1, srv535113807=75, srv655810377=82, srv937796707=96, srv586394118=79, srv1084043961=4, srv1781965495=42, srv1273080607=12, srv192509774=49, srv1313410833=14, srv1319436259=16, srv2118511252=59, srv1082236040=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-10 03:53:41,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-10 03:53:41,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-10 03:53:41,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-10 03:53:41,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-10 03:53:41,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-10 03:53:41,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-10 03:53:41,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-10 03:53:41,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-10 03:53:41,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-10 03:53:41,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-10 03:53:41,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-10 03:53:41,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-10 03:53:41,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-10 03:53:41,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-10 03:53:41,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-10 03:53:41,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-10 03:53:41,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-10 03:53:41,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-10 03:53:41,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-10 03:53:41,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-10 03:53:41,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-10 03:53:41,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-10 03:53:41,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-10 03:53:41,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-10 03:53:41,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-10 03:53:41,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-10 03:53:41,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-10 03:53:41,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-10 03:53:41,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-10 03:53:41,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-10 03:53:41,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-10 03:53:41,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-10 03:53:41,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-10 03:53:41,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-10 03:53:41,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-10 03:53:41,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-10 03:53:41,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-10 03:53:41,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-10 03:53:41,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-10 03:53:41,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-10 03:53:41,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-10 03:53:41,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-10 03:53:41,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-10 03:53:41,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-10 03:53:41,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-10 03:53:41,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-10 03:53:41,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-10 03:53:41,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-10 03:53:41,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-10 03:53:41,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-10 03:53:41,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-10 03:53:41,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-10 03:53:41,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-10 03:53:41,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-10 03:53:41,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-10 03:53:41,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-10 03:53:41,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-10 03:53:41,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-10 03:53:41,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-10 03:53:41,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-10 03:53:41,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-10 03:53:41,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-10 03:53:41,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-10 03:53:41,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-10 03:53:41,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-10 03:53:41,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-10 03:53:41,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-10 03:53:41,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-10 03:53:41,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-10 03:53:41,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-10 03:53:41,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-10 03:53:41,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-10 03:53:41,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-10 03:53:41,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-10 03:53:41,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-10 03:53:41,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-10 03:53:41,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-10 03:53:41,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-10 03:53:41,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-10 03:53:41,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-10 03:53:41,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-10 03:53:41,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-10 03:53:41,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-10 03:53:41,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-10 03:53:41,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-10 03:53:41,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-10 03:53:41,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-10 03:53:41,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-10 03:53:41,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-10 03:53:41,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-10 03:53:41,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-10 03:53:41,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-10 03:53:41,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-10 03:53:41,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-10 03:53:41,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-10 03:53:41,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-10 03:53:41,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-10 03:53:41,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-10 03:53:41,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-10 03:53:41,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-10 03:53:41,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-10 03:53:41,247 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02308320857289348); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032414958235299086, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02308320857289348); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.014916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-10 03:53:41,248 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-10 03:53:41,248 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01766993763219041, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02308320857289348); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032414958235299086, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02308320857289348); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.014916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-10 03:53:41,560 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 407 ms to try 5000 different iterations.  Found a solution that moves 1006 regions; Going from a computed imbalance of 0.01766993763219041 to a new imbalance of 7.954146640163682E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006440224633376166); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006581953356624206); MoveCostFunction : (multiplier=7.0, imbalance=0.11177777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006440224633376166); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-10 03:53:41,563 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1418659040:80 , srv1743742048:80 , srv340886895:81 , srv1120538450:82 , srv1319436259:82 , srv1787066361:82 , srv1357199025:83 , srv1428594470:83 , srv1535355935:83 , srv2019583655:83 , srv2133372914:83 , srv85372653:83 , srv1022434489:84 , srv1047172419:84 , srv1093441677:84 , srv1616179480:84 , srv1714957000:84 , srv1803964967:84 , srv43967776:84 , srv548571276:84 , srv655810377:84 , srv864079373:84 , srv877893443:84 , srv899289723:84 , srv1084043961:85 , srv111035028:85 , srv1149601767:85 , srv1294610183:85 , srv1315164689:85 , srv142139550:85 , srv154837443:85 , srv1579074811:85 , srv1747792537:85 , srv1781965495:85 , srv1819875134:85 , srv193278736:85 , srv2080074526:85 , srv2135338812:85 , srv352278526:85 , srv374004340:85 , srv403509454:85 , srv515538169:85 , srv961965243:85 , srv1101618161:86 , srv1181167898:86 , srv1259739912:86 , srv1273080607:86 , srv1320118400:86 , srv133175909:86 , srv134682157:86 , srv1396529305:86 , srv1638958482:86 , srv1660272222:86 , srv1675488616:86 , srv1699784919:86 , srv1737606680:86 , srv18474103:86 , srv1937993421:86 , srv2062672354:86 , srv2079013538:86 , srv408848222:86 , srv499710671:86 , srv508149291:86 , srv555038431:86 , srv588405267:86 , srv665281540:86 , srv66941151:86 , srv758995920:86 , srv874956761:86 , srv901334138:86 , srv937796707:86 , srv968588902:86 , srv1082236040:87 , srv1343413333:87 , srv1356274689:87 , srv1463286021:87 , srv1535837876:87 , srv192509774:87 , srv1938410670:87 , srv1972950140:87 , srv2091829182:87 , srv2131912310:87 , srv437535411:87 , srv471614609:87 , srv535113807:87 , srv543964527:87 , srv586394118:87 , srv6366723:87 , srv974691480:87 , srv1019180119:88 , srv1313410833:88 , srv1654732250:88 , srv1897507850:88 , srv2118511252:88 , srv357787227:88 , srv930797978:88 , srv1899275178:89 , srv719098409:89 , srv880688458:89 , srv872720629:536 }
2023-05-10 03:53:41,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-10 03:53:41,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1699784919=37, srv1747792537=41, srv1938410670=52, srv437535411=69, srv1803964967=44, srv665281540=83, srv134682157=20, srv193278736=50, srv1972950140=53, srv499710671=72, srv543964527=76, srv1660272222=35, srv142139550=25, srv1463286021=27, srv758995920=86, srv340886895=63, srv1675488616=36, srv899289723=93, srv877893443=91, srv1937993421=51, srv1357199025=22, srv930797978=95, srv133175909=18, srv2131912310=60, srv1120538450=8, srv880688458=92, srv1428594470=26, srv111035028=7, srv66941151=84, srv1149601767=9, srv2133372914=61, srv548571276=77, srv2062672354=55, srv408848222=68, srv403509454=67, srv555038431=78, srv1396529305=23, srv1343413333=19, srv154837443=30, srv1897507850=47, srv719098409=85, srv1019180119=0, srv872720629=89, srv1737606680=39, srv1259739912=11, srv18474103=46, srv508149291=73, srv6366723=81, srv1638958482=33, srv1654732250=34, srv874956761=90, srv968588902=98, srv1616179480=32, srv471614609=71, srv1819875134=45, srv85372653=87, srv1181167898=10, srv2079013538=56, srv2080074526=57, srv588405267=80, srv43967776=70, srv1101618161=6, srv1899275178=48, srv1535355935=28, srv2019583655=54, srv1787066361=43, srv1294610183=13, srv2135338812=62, srv1320118400=17, srv352278526=64, srv901334138=94, srv864079373=88, srv515538169=74, srv1047172419=2, srv974691480=99, srv374004340=66, srv1315164689=15, srv1535837876=29, srv1418659040=24, srv1579074811=31, srv1356274689=21, srv2091829182=58, srv961965243=97, srv1743742048=40, srv1714957000=38, srv357787227=65, srv1093441677=5, srv1022434489=1, srv535113807=75, srv655810377=82, srv937796707=96, srv586394118=79, srv1084043961=4, srv1781965495=42, srv1273080607=12, srv192509774=49, srv1313410833=14, srv1319436259=16, srv2118511252=59, srv1082236040=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-10 03:53:41,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-10 03:53:41,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-10 03:53:41,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-10 03:53:41,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-10 03:53:41,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-10 03:53:41,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-10 03:53:41,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-10 03:53:41,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-10 03:53:41,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-10 03:53:41,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-10 03:53:41,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-10 03:53:41,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-10 03:53:41,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-10 03:53:41,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-10 03:53:41,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-10 03:53:41,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-10 03:53:41,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-10 03:53:41,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-10 03:53:41,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-10 03:53:41,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-10 03:53:41,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-10 03:53:41,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-10 03:53:41,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-10 03:53:41,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-10 03:53:41,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-10 03:53:41,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-10 03:53:41,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-10 03:53:41,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-10 03:53:41,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-10 03:53:41,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-10 03:53:41,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-10 03:53:41,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-10 03:53:41,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-10 03:53:41,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-10 03:53:41,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-10 03:53:41,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-10 03:53:41,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-10 03:53:41,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-10 03:53:41,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-10 03:53:41,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-10 03:53:41,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-10 03:53:41,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-10 03:53:41,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-10 03:53:41,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-10 03:53:41,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-10 03:53:41,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-10 03:53:41,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-10 03:53:41,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-10 03:53:41,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-10 03:53:41,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-10 03:53:41,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-10 03:53:41,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-10 03:53:41,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-10 03:53:41,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-10 03:53:41,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-10 03:53:41,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-10 03:53:41,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-10 03:53:41,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-10 03:53:41,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-10 03:53:41,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-10 03:53:41,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-10 03:53:41,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-10 03:53:41,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-10 03:53:41,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-10 03:53:41,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-10 03:53:41,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-10 03:53:41,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-10 03:53:41,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-10 03:53:41,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-10 03:53:41,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-10 03:53:41,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-10 03:53:41,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-10 03:53:41,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-10 03:53:41,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-10 03:53:41,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-10 03:53:41,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-10 03:53:41,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-10 03:53:41,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-10 03:53:41,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-10 03:53:41,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-10 03:53:41,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-10 03:53:41,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-10 03:53:41,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-10 03:53:41,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-10 03:53:41,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-10 03:53:41,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-10 03:53:41,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-10 03:53:41,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-10 03:53:41,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-10 03:53:41,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-10 03:53:41,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-10 03:53:41,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-10 03:53:41,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-10 03:53:41,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-10 03:53:41,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-10 03:53:41,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-10 03:53:41,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-10 03:53:41,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-10 03:53:41,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-10 03:53:41,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-10 03:53:41,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-10 03:53:41,737 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006440224633376166); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006581953356624206); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006440224633376166); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-10 03:53:41,737 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-10 03:53:41,737 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.88370501458463E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006440224633376166); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006581953356624206); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006440224633376166); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-10 03:53:42,030 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 403 ms to try 5000 different iterations.  Found a solution that moves 256 regions; Going from a computed imbalance of 7.88370501458463E-4 to a new imbalance of 1.983656572913158E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.028444444444444446, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-10 03:53:42,032 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1616179480:83 , srv1654732250:83 , srv1897507850:83 , srv2118511252:83 , srv352278526:83 , srv471614609:83 , srv719098409:83 , srv961965243:83 , srv1019180119:84 , srv1022434489:84 , srv1101618161:84 , srv1149601767:84 , srv1181167898:84 , srv1273080607:84 , srv1294610183:84 , srv1313410833:84 , srv1320118400:84 , srv1356274689:84 , srv1396529305:84 , srv1535837876:84 , srv154837443:84 , srv1638958482:84 , srv1699784919:84 , srv1737606680:84 , srv1781965495:84 , srv1787066361:84 , srv18474103:84 , srv193278736:84 , srv1937993421:84 , srv2062672354:84 , srv2080074526:84 , srv2091829182:84 , srv2131912310:84 , srv2133372914:84 , srv2135338812:84 , srv357787227:84 , srv535113807:84 , srv555038431:84 , srv655810377:84 , srv758995920:84 , srv880688458:84 , srv899289723:84 , srv901334138:84 , srv974691480:84 , srv1082236040:85 , srv1084043961:85 , srv1093441677:85 , srv111035028:85 , srv1259739912:85 , srv1315164689:85 , srv1319436259:85 , srv133175909:85 , srv1343413333:85 , srv134682157:85 , srv1357199025:85 , srv142139550:85 , srv1463286021:85 , srv1535355935:85 , srv1579074811:85 , srv1660272222:85 , srv1714957000:85 , srv1743742048:85 , srv1747792537:85 , srv1803964967:85 , srv1819875134:85 , srv192509774:85 , srv1972950140:85 , srv2019583655:85 , srv340886895:85 , srv374004340:85 , srv403509454:85 , srv408848222:85 , srv437535411:85 , srv43967776:85 , srv499710671:85 , srv508149291:85 , srv515538169:85 , srv543964527:85 , srv548571276:85 , srv586394118:85 , srv588405267:85 , srv6366723:85 , srv665281540:85 , srv85372653:85 , srv864079373:85 , srv874956761:85 , srv930797978:85 , srv937796707:85 , srv968588902:85 , srv1047172419:86 , srv1120538450:86 , srv1418659040:86 , srv1428594470:86 , srv1675488616:86 , srv1899275178:86 , srv1938410670:86 , srv2079013538:86 , srv66941151:86 , srv877893443:86 , srv872720629:627 }
2023-05-10 03:53:42,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-10 03:53:42,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1699784919=37, srv1747792537=41, srv1938410670=52, srv437535411=69, srv1803964967=44, srv665281540=83, srv134682157=20, srv193278736=50, srv1972950140=53, srv499710671=72, srv543964527=76, srv1660272222=35, srv142139550=25, srv1463286021=27, srv758995920=86, srv340886895=63, srv1675488616=36, srv899289723=93, srv877893443=91, srv1937993421=51, srv1357199025=22, srv930797978=95, srv133175909=18, srv2131912310=60, srv1120538450=8, srv880688458=92, srv1428594470=26, srv111035028=7, srv66941151=84, srv1149601767=9, srv2133372914=61, srv548571276=77, srv2062672354=55, srv408848222=68, srv403509454=67, srv555038431=78, srv1396529305=23, srv1343413333=19, srv154837443=30, srv1897507850=47, srv719098409=85, srv1019180119=0, srv872720629=89, srv1737606680=39, srv1259739912=11, srv18474103=46, srv508149291=73, srv6366723=81, srv1638958482=33, srv1654732250=34, srv874956761=90, srv968588902=98, srv1616179480=32, srv471614609=71, srv1819875134=45, srv85372653=87, srv1181167898=10, srv2079013538=56, srv2080074526=57, srv588405267=80, srv43967776=70, srv1101618161=6, srv1899275178=48, srv1535355935=28, srv2019583655=54, srv1787066361=43, srv1294610183=13, srv2135338812=62, srv1320118400=17, srv352278526=64, srv901334138=94, srv864079373=88, srv515538169=74, srv1047172419=2, srv974691480=99, srv374004340=66, srv1315164689=15, srv1535837876=29, srv1418659040=24, srv1579074811=31, srv1356274689=21, srv2091829182=58, srv961965243=97, srv1743742048=40, srv1714957000=38, srv357787227=65, srv1093441677=5, srv1022434489=1, srv535113807=75, srv655810377=82, srv937796707=96, srv586394118=79, srv1084043961=4, srv1781965495=42, srv1273080607=12, srv192509774=49, srv1313410833=14, srv1319436259=16, srv2118511252=59, srv1082236040=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-10 03:53:42,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-10 03:53:42,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-10 03:53:42,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-10 03:53:42,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-10 03:53:42,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-10 03:53:42,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-10 03:53:42,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-10 03:53:42,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-10 03:53:42,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-10 03:53:42,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-10 03:53:42,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-10 03:53:42,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-10 03:53:42,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-10 03:53:42,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-10 03:53:42,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-10 03:53:42,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-10 03:53:42,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-10 03:53:42,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-10 03:53:42,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-10 03:53:42,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-10 03:53:42,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-10 03:53:42,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-10 03:53:42,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-10 03:53:42,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-10 03:53:42,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-10 03:53:42,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-10 03:53:42,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-10 03:53:42,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-10 03:53:42,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-10 03:53:42,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-10 03:53:42,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-10 03:53:42,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-10 03:53:42,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-10 03:53:42,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-10 03:53:42,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-10 03:53:42,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-10 03:53:42,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-10 03:53:42,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-10 03:53:42,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-10 03:53:42,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-10 03:53:42,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-10 03:53:42,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-10 03:53:42,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-10 03:53:42,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-10 03:53:42,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-10 03:53:42,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-10 03:53:42,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-10 03:53:42,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-10 03:53:42,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-10 03:53:42,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-10 03:53:42,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-10 03:53:42,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-10 03:53:42,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-10 03:53:42,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-10 03:53:42,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-10 03:53:42,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-10 03:53:42,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-10 03:53:42,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-10 03:53:42,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-10 03:53:42,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-10 03:53:42,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-10 03:53:42,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-10 03:53:42,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-10 03:53:42,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-10 03:53:42,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-10 03:53:42,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-10 03:53:42,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-10 03:53:42,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-10 03:53:42,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-10 03:53:42,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-10 03:53:42,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-10 03:53:42,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-10 03:53:42,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-10 03:53:42,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-10 03:53:42,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-10 03:53:42,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-10 03:53:42,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-10 03:53:42,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-10 03:53:42,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-10 03:53:42,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-10 03:53:42,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-10 03:53:42,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-10 03:53:42,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-10 03:53:42,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-10 03:53:42,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-10 03:53:42,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-10 03:53:42,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-10 03:53:42,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-10 03:53:42,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-10 03:53:42,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-10 03:53:42,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-10 03:53:42,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-10 03:53:42,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-10 03:53:42,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-10 03:53:42,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-10 03:53:42,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-10 03:53:42,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-10 03:53:42,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-10 03:53:42,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-10 03:53:42,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-10 03:53:42,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-10 03:53:42,184 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.8044015416185505E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-10 03:53:42,184 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1616179480:83 , srv1654732250:83 , srv1897507850:83 , srv2118511252:83 , srv352278526:83 , srv471614609:83 , srv719098409:83 , srv961965243:83 , srv1019180119:84 , srv1022434489:84 , srv1101618161:84 , srv1149601767:84 , srv1181167898:84 , srv1273080607:84 , srv1294610183:84 , srv1313410833:84 , srv1320118400:84 , srv1356274689:84 , srv1396529305:84 , srv1535837876:84 , srv154837443:84 , srv1638958482:84 , srv1699784919:84 , srv1737606680:84 , srv1781965495:84 , srv1787066361:84 , srv18474103:84 , srv193278736:84 , srv1937993421:84 , srv2062672354:84 , srv2080074526:84 , srv2091829182:84 , srv2131912310:84 , srv2133372914:84 , srv2135338812:84 , srv357787227:84 , srv535113807:84 , srv555038431:84 , srv655810377:84 , srv758995920:84 , srv880688458:84 , srv899289723:84 , srv901334138:84 , srv974691480:84 , srv1082236040:85 , srv1084043961:85 , srv1093441677:85 , srv111035028:85 , srv1259739912:85 , srv1315164689:85 , srv1319436259:85 , srv133175909:85 , srv1343413333:85 , srv134682157:85 , srv1357199025:85 , srv142139550:85 , srv1463286021:85 , srv1535355935:85 , srv1579074811:85 , srv1660272222:85 , srv1714957000:85 , srv1743742048:85 , srv1747792537:85 , srv1803964967:85 , srv1819875134:85 , srv192509774:85 , srv1972950140:85 , srv2019583655:85 , srv340886895:85 , srv374004340:85 , srv403509454:85 , srv408848222:85 , srv437535411:85 , srv43967776:85 , srv499710671:85 , srv508149291:85 , srv515538169:85 , srv543964527:85 , srv548571276:85 , srv586394118:85 , srv588405267:85 , srv6366723:85 , srv665281540:85 , srv85372653:85 , srv864079373:85 , srv874956761:85 , srv930797978:85 , srv937796707:85 , srv968588902:85 , srv1047172419:86 , srv1120538450:86 , srv1418659040:86 , srv1428594470:86 , srv1675488616:86 , srv1899275178:86 , srv1938410670:86 , srv2079013538:86 , srv66941151:86 , srv877893443:86 , srv872720629:627 }
2023-05-10 03:53:42,233 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=377 (was 377), ProcessCount=171 (was 171), AvailableMemoryMB=10185 (was 11036)
2023-05-10 03:53:42,245 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=262, MaxFileDescriptor=60000, SystemLoadAverage=377, ProcessCount=171, AvailableMemoryMB=10185
2023-05-10 03:53:42,245 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-10 03:53:42,246 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-05-10 03:53:42,255 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1026882755:84 , srv1135235531:84 , srv1321297038:84 , srv1346161963:84 , srv1421624392:84 , srv1510399204:84 , srv1544033191:84 , srv1589864005:84 , srv1655894880:84 , srv1666954855:84 , srv1754733630:84 , srv1759363323:84 , srv1886626152:84 , srv1971680511:84 , srv2042222080:84 , srv283711121:84 , srv317913936:84 , srv384520420:84 , srv408466975:84 , srv451976878:84 , srv479151632:84 , srv698728424:84 , srv710553890:84 , srv728706074:84 , srv814406808:84 , srv81993754:84 , srv841409341:84 , srv871934122:84 , srv927094003:84 , srv212325893:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-05-10 03:53:42,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-10 03:53:42,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv408466975=19, srv451976878=20, srv2042222080=14, srv814406808=25, srv871934122=28, srv1655894880=8, srv1886626152=12, srv1421624392=4, srv479151632=21, srv1754733630=10, srv1971680511=13, srv212325893=15, srv1510399204=5, srv384520420=18, srv1589864005=7, srv1759363323=11, srv81993754=26, srv317913936=17, srv1346161963=3, srv1666954855=9, srv283711121=16, srv710553890=23, srv1135235531=1, srv841409341=27, srv1026882755=0, srv1321297038=2, srv698728424=22, srv728706074=24, srv927094003=29, srv1544033191=6} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-10 03:53:42,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-10 03:53:42,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-10 03:53:42,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-10 03:53:42,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-10 03:53:42,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-10 03:53:42,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-10 03:53:42,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-10 03:53:42,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-10 03:53:42,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-10 03:53:42,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-10 03:53:42,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-10 03:53:42,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-10 03:53:42,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-10 03:53:42,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-10 03:53:42,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-10 03:53:42,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-10 03:53:42,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-10 03:53:42,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-10 03:53:42,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-10 03:53:42,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-10 03:53:42,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-10 03:53:42,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-10 03:53:42,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-10 03:53:42,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-10 03:53:42,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-10 03:53:42,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-10 03:53:42,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-10 03:53:42,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-10 03:53:42,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-10 03:53:42,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-10 03:53:42,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-05-10 03:53:42,298 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.078454227539969, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-10 03:53:42,298 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912440249973492, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.078454227539969, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-05-10 03:55:42,277 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6749004 different iterations.  Found a solution that moves 1839 regions; Going from a computed imbalance of 0.9912440249973492 to a new imbalance of 4.292317741546655E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6811111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-10 03:55:42,284 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1026882755:90 , srv1135235531:90 , srv1321297038:90 , srv1346161963:90 , srv1421624392:90 , srv1510399204:90 , srv1544033191:90 , srv1589864005:90 , srv1655894880:90 , srv1666954855:90 , srv1754733630:90 , srv1759363323:90 , srv1886626152:90 , srv1971680511:90 , srv2042222080:90 , srv212325893:90 , srv283711121:90 , srv317913936:90 , srv384520420:90 , srv408466975:90 , srv451976878:90 , srv479151632:90 , srv698728424:90 , srv710553890:90 , srv728706074:90 , srv814406808:90 , srv81993754:90 , srv841409341:90 , srv871934122:90 , srv927094003:90 }
2023-05-10 03:55:42,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-10 03:55:42,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv408466975=19, srv451976878=20, srv2042222080=14, srv814406808=25, srv871934122=28, srv1655894880=8, srv1886626152=12, srv1421624392=4, srv479151632=21, srv1754733630=10, srv1971680511=13, srv212325893=15, srv1510399204=5, srv384520420=18, srv1589864005=7, srv1759363323=11, srv81993754=26, srv317913936=17, srv1346161963=3, srv1666954855=9, srv283711121=16, srv710553890=23, srv1135235531=1, srv841409341=27, srv1026882755=0, srv1321297038=2, srv698728424=22, srv728706074=24, srv927094003=29, srv1544033191=6} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-10 03:55:42,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-10 03:55:42,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-10 03:55:42,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-10 03:55:42,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-10 03:55:42,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-10 03:55:42,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-10 03:55:42,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-10 03:55:42,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-10 03:55:42,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-10 03:55:42,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-10 03:55:42,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-10 03:55:42,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-10 03:55:42,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-10 03:55:42,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-10 03:55:42,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-10 03:55:42,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-10 03:55:42,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-10 03:55:42,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-10 03:55:42,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-10 03:55:42,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-10 03:55:42,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-10 03:55:42,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-10 03:55:42,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-10 03:55:42,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-10 03:55:42,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-10 03:55:42,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-10 03:55:42,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-10 03:55:42,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-10 03:55:42,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-10 03:55:42,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-10 03:55:42,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-05-10 03:55:42,302 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-10 03:55:42,302 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1026882755:90 , srv1135235531:90 , srv1321297038:90 , srv1346161963:90 , srv1421624392:90 , srv1510399204:90 , srv1544033191:90 , srv1589864005:90 , srv1655894880:90 , srv1666954855:90 , srv1754733630:90 , srv1759363323:90 , srv1886626152:90 , srv1971680511:90 , srv2042222080:90 , srv212325893:90 , srv283711121:90 , srv317913936:90 , srv384520420:90 , srv408466975:90 , srv451976878:90 , srv479151632:90 , srv698728424:90 , srv710553890:90 , srv728706074:90 , srv814406808:90 , srv81993754:90 , srv841409341:90 , srv871934122:90 , srv927094003:90 }
2023-05-10 03:55:42,320 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 262), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=160 (was 377), ProcessCount=169 (was 171), AvailableMemoryMB=10086 (was 10185)