2023-05-28 03:53:23,941 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-05-28 03:53:24,210 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-05-28 03:53:24,307 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-05-28 03:53:24,361 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-28 03:53:24,368 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-05-28 03:53:24,380 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=281, ProcessCount=174, AvailableMemoryMB=5326
2023-05-28 03:53:24,382 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-28 03:53:24,382 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-05-28 03:53:24,505 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1040929536:84 , srv1050982692:84 , srv1067730853:84 , srv1073745487:84 , srv1115153424:84 , srv1122083929:84 , srv1186775009:84 , srv1211297679:84 , srv1239277996:84 , srv1244496503:84 , srv1245997129:84 , srv128385485:84 , srv1288130461:84 , srv1340659409:84 , srv134929259:84 , srv1352464462:84 , srv1355395531:84 , srv1368207252:84 , srv1385403639:84 , srv1414094364:84 , srv1416467161:84 , srv1438960149:84 , srv1486485506:84 , srv1487369977:84 , srv1517923349:84 , srv155544575:84 , srv1566220754:84 , srv1570778221:84 , srv1602884632:84 , srv1636553654:84 , srv166933970:84 , srv168392873:84 , srv1692124799:84 , srv1773742287:84 , srv1777664123:84 , srv1786429765:84 , srv1803880389:84 , srv1819836875:84 , srv1868318759:84 , srv1875065460:84 , srv1882133123:84 , srv1945994684:84 , srv1946615099:84 , srv1948498760:84 , srv1965069570:84 , srv1977235489:84 , srv1979449324:84 , srv1985960272:84 , srv2004056184:84 , srv2007742081:84 , srv2010459690:84 , srv2056270559:84 , srv207194226:84 , srv2075009683:84 , srv2098302050:84 , srv2109630910:84 , srv2116827313:84 , srv223370537:84 , srv228226275:84 , srv23509751:84 , srv259735501:84 , srv363402242:84 , srv374576610:84 , srv375318510:84 , srv422148337:84 , srv450763988:84 , srv47545890:84 , srv519435713:84 , srv520139509:84 , srv535194233:84 , srv535759443:84 , srv542327104:84 , srv554422365:84 , srv554433879:84 , srv605841741:84 , srv610820674:84 , srv612051048:84 , srv613150980:84 , srv643777511:84 , srv664947068:84 , srv675008288:84 , srv676266805:84 , srv698413480:84 , srv712197518:84 , srv713216110:84 , srv752439751:84 , srv760977287:84 , srv772888704:84 , srv773558928:84 , srv830552939:84 , srv841150949:84 , srv872019921:84 , srv895767919:84 , srv921839055:84 , srv941051115:84 , srv944461121:84 , srv982248893:84 , srv988747463:84 , srv997737457:84 , srv962615980:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-05-28 03:53:24,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-28 03:53:24,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1438960149=21, srv612051048=76, srv1965069570=44, srv676266805=81, srv1486485506=22, srv895767919=92, srv1979449324=46, srv542327104=71, srv872019921=91, srv1414094364=19, srv554422365=72, srv1819836875=37, srv1948498760=43, srv1067730853=2, srv613150980=77, srv519435713=67, srv207194226=52, srv554433879=73, srv1570778221=27, srv712197518=83, srv2116827313=56, srv1875065460=39, srv1517923349=24, srv1786429765=35, srv643777511=78, srv1073745487=3, srv1882133123=40, srv664947068=79, srv773558928=88, srv1692124799=32, srv1777664123=34, srv375318510=63, srv675008288=80, srv752439751=85, srv2098302050=54, srv2007742081=49, srv1122083929=5, srv1340659409=13, srv1977235489=45, srv1803880389=36, srv166933970=30, srv1245997129=10, srv1385403639=18, srv374576610=62, srv1368207252=17, srv422148337=64, srv520139509=68, srv1985960272=47, srv1946615099=42, srv1352464462=15, srv128385485=11, srv134929259=14, srv944461121=95, srv2004056184=48, srv535194233=69, srv1566220754=26, srv962615980=96, srv1244496503=9, srv1416467161=20, srv1636553654=29, srv363402242=61, srv1040929536=0, srv450763988=65, srv535759443=70, srv2056270559=51, srv1945994684=41, srv841150949=90, srv772888704=87, srv830552939=89, srv259735501=60, srv2109630910=55, srv155544575=25, srv610820674=75, srv1355395531=16, srv997737457=99, srv47545890=66, srv982248893=97, srv1211297679=7, srv23509751=59, srv1186775009=6, srv760977287=86, srv1050982692=1, srv168392873=31, srv713216110=84, srv921839055=93, srv1288130461=12, srv1487369977=23, srv605841741=74, srv228226275=58, srv2010459690=50, srv941051115=94, srv698413480=82, srv1239277996=8, srv1773742287=33, srv2075009683=53, srv1115153424=4, srv988747463=98, srv1602884632=28, srv223370537=57, srv1868318759=38} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-28 03:53:24,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-28 03:53:24,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-28 03:53:24,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-28 03:53:24,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-28 03:53:24,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-28 03:53:24,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-28 03:53:24,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-28 03:53:24,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-28 03:53:24,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-28 03:53:24,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-28 03:53:24,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-28 03:53:24,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-28 03:53:24,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-28 03:53:24,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-28 03:53:24,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-28 03:53:24,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-28 03:53:24,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-28 03:53:24,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-28 03:53:24,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-28 03:53:24,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-28 03:53:24,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-28 03:53:24,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-28 03:53:24,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-28 03:53:24,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-28 03:53:24,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-28 03:53:24,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-28 03:53:24,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-28 03:53:24,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-28 03:53:24,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-28 03:53:24,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-28 03:53:24,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-28 03:53:24,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-28 03:53:24,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-28 03:53:24,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-28 03:53:24,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-28 03:53:24,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-28 03:53:24,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-28 03:53:24,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-28 03:53:24,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-28 03:53:24,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-28 03:53:24,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-28 03:53:24,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-28 03:53:24,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-28 03:53:24,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-28 03:53:24,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-28 03:53:24,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-28 03:53:24,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-28 03:53:24,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-28 03:53:24,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-28 03:53:24,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-28 03:53:24,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-28 03:53:24,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-28 03:53:24,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-28 03:53:24,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-28 03:53:24,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-28 03:53:24,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-28 03:53:24,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-28 03:53:24,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-28 03:53:24,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-28 03:53:24,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-28 03:53:24,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-28 03:53:24,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-28 03:53:24,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-28 03:53:24,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-28 03:53:24,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-28 03:53:24,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-28 03:53:24,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-28 03:53:24,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-28 03:53:24,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-28 03:53:24,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-28 03:53:24,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-28 03:53:24,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-28 03:53:24,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-28 03:53:24,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-28 03:53:24,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-28 03:53:24,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-28 03:53:24,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-28 03:53:24,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-28 03:53:24,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-28 03:53:24,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-28 03:53:24,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-28 03:53:24,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-28 03:53:24,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-28 03:53:24,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-28 03:53:24,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-28 03:53:24,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-28 03:53:24,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-28 03:53:24,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-28 03:53:24,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-28 03:53:24,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-28 03:53:24,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-28 03:53:24,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-28 03:53:24,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-28 03:53:24,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-28 03:53:24,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-28 03:53:24,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-28 03:53:24,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-28 03:53:24,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-28 03:53:24,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-28 03:53:24,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-28 03:53:24,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-28 03:53:24,758 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-28 03:53:24,758 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-28 03:53:24,758 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-28 03:53:25,041 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 430 ms to try 5000 different iterations.  Found a solution that moves 4691 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18339758834472644. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040511753169985724, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05186315413136211, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5212222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040511753169985724, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.179, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.242, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-28 03:53:25,054 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv168392873:81 , srv1186775009:82 , srv1368207252:82 , srv1786429765:82 , srv1948498760:82 , srv2098302050:82 , srv47545890:82 , srv941051115:82 , srv1245997129:83 , srv128385485:83 , srv2075009683:83 , srv613150980:83 , srv664947068:83 , srv830552939:83 , srv997737457:83 , srv1040929536:84 , srv1355395531:84 , srv1946615099:84 , srv1979449324:84 , srv450763988:84 , srv612051048:84 , srv982248893:84 , srv1211297679:85 , srv1340659409:85 , srv1438960149:85 , srv1570778221:85 , srv1636553654:85 , srv1773742287:85 , srv1977235489:85 , srv374576610:85 , srv535194233:85 , srv554422365:85 , srv643777511:85 , srv1050982692:86 , srv1115153424:86 , srv1244496503:86 , srv1416467161:86 , srv1602884632:86 , srv1777664123:86 , srv1882133123:86 , srv1985960272:86 , srv2007742081:86 , srv228226275:86 , srv23509751:86 , srv519435713:86 , srv554433879:86 , srv675008288:86 , srv712197518:86 , srv760977287:86 , srv872019921:86 , srv895767919:86 , srv1067730853:87 , srv1239277996:87 , srv1352464462:87 , srv1414094364:87 , srv1487369977:87 , srv1566220754:87 , srv1692124799:87 , srv1803880389:87 , srv1868318759:87 , srv1875065460:87 , srv2004056184:87 , srv2010459690:87 , srv2056270559:87 , srv207194226:87 , srv375318510:87 , srv535759443:87 , srv605841741:87 , srv610820674:87 , srv676266805:87 , srv752439751:87 , srv773558928:87 , srv944461121:87 , srv988747463:87 , srv1122083929:88 , srv1288130461:88 , srv134929259:88 , srv155544575:88 , srv166933970:88 , srv1965069570:88 , srv2109630910:88 , srv422148337:88 , srv542327104:88 , srv841150949:88 , srv921839055:88 , srv1073745487:89 , srv1517923349:89 , srv363402242:89 , srv698413480:89 , srv1819836875:90 , srv223370537:90 , srv1945994684:91 , srv259735501:91 , srv520139509:91 , srv713216110:91 , srv772888704:91 , srv2116827313:93 , srv1385403639:94 , srv1486485506:96 , srv962615980:450 }
2023-05-28 03:53:25,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-28 03:53:25,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1438960149=21, srv612051048=76, srv1965069570=44, srv676266805=81, srv1486485506=22, srv895767919=92, srv1979449324=46, srv542327104=71, srv872019921=91, srv1414094364=19, srv554422365=72, srv1819836875=37, srv1948498760=43, srv1067730853=2, srv613150980=77, srv519435713=67, srv207194226=52, srv554433879=73, srv1570778221=27, srv712197518=83, srv2116827313=56, srv1875065460=39, srv1517923349=24, srv1786429765=35, srv643777511=78, srv1073745487=3, srv1882133123=40, srv664947068=79, srv773558928=88, srv1692124799=32, srv1777664123=34, srv375318510=63, srv675008288=80, srv752439751=85, srv2098302050=54, srv2007742081=49, srv1122083929=5, srv1340659409=13, srv1977235489=45, srv1803880389=36, srv166933970=30, srv1245997129=10, srv1385403639=18, srv374576610=62, srv1368207252=17, srv422148337=64, srv520139509=68, srv1985960272=47, srv1946615099=42, srv1352464462=15, srv128385485=11, srv134929259=14, srv944461121=95, srv2004056184=48, srv535194233=69, srv1566220754=26, srv962615980=96, srv1244496503=9, srv1416467161=20, srv1636553654=29, srv363402242=61, srv1040929536=0, srv450763988=65, srv535759443=70, srv2056270559=51, srv1945994684=41, srv841150949=90, srv772888704=87, srv830552939=89, srv259735501=60, srv2109630910=55, srv155544575=25, srv610820674=75, srv1355395531=16, srv997737457=99, srv47545890=66, srv982248893=97, srv1211297679=7, srv23509751=59, srv1186775009=6, srv760977287=86, srv1050982692=1, srv168392873=31, srv713216110=84, srv921839055=93, srv1288130461=12, srv1487369977=23, srv605841741=74, srv228226275=58, srv2010459690=50, srv941051115=94, srv698413480=82, srv1239277996=8, srv1773742287=33, srv2075009683=53, srv1115153424=4, srv988747463=98, srv1602884632=28, srv223370537=57, srv1868318759=38} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-28 03:53:25,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-28 03:53:25,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-28 03:53:25,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-28 03:53:25,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-28 03:53:25,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-28 03:53:25,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-28 03:53:25,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-28 03:53:25,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-28 03:53:25,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-28 03:53:25,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-28 03:53:25,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-28 03:53:25,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-28 03:53:25,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-28 03:53:25,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-28 03:53:25,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-28 03:53:25,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-28 03:53:25,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-28 03:53:25,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-28 03:53:25,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-28 03:53:25,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-28 03:53:25,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-28 03:53:25,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-28 03:53:25,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-28 03:53:25,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-28 03:53:25,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-28 03:53:25,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-28 03:53:25,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-28 03:53:25,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-28 03:53:25,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-28 03:53:25,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-28 03:53:25,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-28 03:53:25,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-28 03:53:25,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-28 03:53:25,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-28 03:53:25,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-28 03:53:25,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-28 03:53:25,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-28 03:53:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-28 03:53:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-28 03:53:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-28 03:53:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-28 03:53:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-28 03:53:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-28 03:53:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-28 03:53:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-28 03:53:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-28 03:53:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-28 03:53:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-28 03:53:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-28 03:53:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-28 03:53:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-28 03:53:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-28 03:53:25,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-28 03:53:25,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-28 03:53:25,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-28 03:53:25,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-28 03:53:25,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-28 03:53:25,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-28 03:53:25,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-28 03:53:25,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-28 03:53:25,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-28 03:53:25,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-28 03:53:25,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-28 03:53:25,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-28 03:53:25,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-28 03:53:25,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-28 03:53:25,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-28 03:53:25,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-28 03:53:25,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-28 03:53:25,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-28 03:53:25,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-28 03:53:25,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-28 03:53:25,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-28 03:53:25,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-28 03:53:25,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-28 03:53:25,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-28 03:53:25,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-28 03:53:25,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-28 03:53:25,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-28 03:53:25,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-28 03:53:25,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-28 03:53:25,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-28 03:53:25,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-28 03:53:25,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-28 03:53:25,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-28 03:53:25,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-28 03:53:25,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-28 03:53:25,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-28 03:53:25,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-28 03:53:25,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-28 03:53:25,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-28 03:53:25,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-28 03:53:25,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-28 03:53:25,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-28 03:53:25,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-28 03:53:25,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-28 03:53:25,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-28 03:53:25,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-28 03:53:25,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-28 03:53:25,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-28 03:53:25,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-28 03:53:25,236 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040511753169985724, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05186315413136211, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040511753169985724, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.179, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.242, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-28 03:53:25,236 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-28 03:53:25,236 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18336474126067162, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040511753169985724, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05186315413136211, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040511753169985724, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.179, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.242, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-28 03:53:25,621 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 506 ms to try 5000 different iterations.  Found a solution that moves 2676 regions; Going from a computed imbalance of 0.18336474126067162 to a new imbalance of 0.015779524995710642. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022062275110248244); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0293819615291366, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29733333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022062275110248244); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04491666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-28 03:53:25,638 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1486485506:76 , srv2116827313:79 , srv1385403639:80 , srv520139509:81 , srv772888704:81 , srv1067730853:82 , srv1073745487:82 , srv1945994684:82 , srv2056270559:82 , srv223370537:82 , srv363402242:82 , srv375318510:82 , srv422148337:82 , srv713216110:82 , srv1239277996:83 , srv207194226:83 , srv1050982692:84 , srv1122083929:84 , srv1288130461:84 , srv134929259:84 , srv1416467161:84 , srv1517923349:84 , srv1566220754:84 , srv166933970:84 , srv1803880389:84 , srv1819836875:84 , srv1868318759:84 , srv1875065460:84 , srv2010459690:84 , srv228226275:84 , srv259735501:84 , srv698413480:84 , srv712197518:84 , srv752439751:84 , srv944461121:84 , srv1414094364:85 , srv1438960149:85 , srv1636553654:85 , srv1965069570:85 , srv2004056184:85 , srv2109630910:85 , srv542327104:85 , srv554422365:85 , srv554433879:85 , srv760977287:85 , srv841150949:85 , srv872019921:85 , srv988747463:85 , srv1115153424:86 , srv1352464462:86 , srv155544575:86 , srv1602884632:86 , srv1773742287:86 , srv2007742081:86 , srv535194233:86 , srv535759443:86 , srv676266805:86 , srv895767919:86 , srv1244496503:87 , srv1340659409:87 , srv1570778221:87 , srv1777664123:87 , srv1882133123:87 , srv1977235489:87 , srv1979449324:87 , srv1985960272:87 , srv2075009683:87 , srv519435713:87 , srv605841741:87 , srv643777511:87 , srv675008288:87 , srv982248893:87 , srv1040929536:88 , srv1211297679:88 , srv1245997129:88 , srv1355395531:88 , srv1368207252:88 , srv1692124799:88 , srv1946615099:88 , srv2098302050:88 , srv374576610:88 , srv450763988:88 , srv613150980:88 , srv773558928:88 , srv921839055:88 , srv128385485:89 , srv1487369977:89 , srv23509751:89 , srv47545890:89 , srv610820674:89 , srv612051048:89 , srv664947068:89 , srv830552939:89 , srv1186775009:90 , srv1786429765:90 , srv1948498760:90 , srv941051115:90 , srv997737457:90 , srv168392873:94 , srv962615980:520 }
2023-05-28 03:53:25,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-28 03:53:25,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1438960149=21, srv612051048=76, srv1965069570=44, srv676266805=81, srv1486485506=22, srv895767919=92, srv1979449324=46, srv542327104=71, srv872019921=91, srv1414094364=19, srv554422365=72, srv1819836875=37, srv1948498760=43, srv1067730853=2, srv613150980=77, srv519435713=67, srv207194226=52, srv554433879=73, srv1570778221=27, srv712197518=83, srv2116827313=56, srv1875065460=39, srv1517923349=24, srv1786429765=35, srv643777511=78, srv1073745487=3, srv1882133123=40, srv664947068=79, srv773558928=88, srv1692124799=32, srv1777664123=34, srv375318510=63, srv675008288=80, srv752439751=85, srv2098302050=54, srv2007742081=49, srv1122083929=5, srv1340659409=13, srv1977235489=45, srv1803880389=36, srv166933970=30, srv1245997129=10, srv1385403639=18, srv374576610=62, srv1368207252=17, srv422148337=64, srv520139509=68, srv1985960272=47, srv1946615099=42, srv1352464462=15, srv128385485=11, srv134929259=14, srv944461121=95, srv2004056184=48, srv535194233=69, srv1566220754=26, srv962615980=96, srv1244496503=9, srv1416467161=20, srv1636553654=29, srv363402242=61, srv1040929536=0, srv450763988=65, srv535759443=70, srv2056270559=51, srv1945994684=41, srv841150949=90, srv772888704=87, srv830552939=89, srv259735501=60, srv2109630910=55, srv155544575=25, srv610820674=75, srv1355395531=16, srv997737457=99, srv47545890=66, srv982248893=97, srv1211297679=7, srv23509751=59, srv1186775009=6, srv760977287=86, srv1050982692=1, srv168392873=31, srv713216110=84, srv921839055=93, srv1288130461=12, srv1487369977=23, srv605841741=74, srv228226275=58, srv2010459690=50, srv941051115=94, srv698413480=82, srv1239277996=8, srv1773742287=33, srv2075009683=53, srv1115153424=4, srv988747463=98, srv1602884632=28, srv223370537=57, srv1868318759=38} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-28 03:53:25,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-28 03:53:25,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-28 03:53:25,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-28 03:53:25,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-28 03:53:25,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-28 03:53:25,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-28 03:53:25,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-28 03:53:25,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-28 03:53:25,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-28 03:53:25,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-28 03:53:25,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-28 03:53:25,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-28 03:53:25,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-28 03:53:25,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-28 03:53:25,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-28 03:53:25,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-28 03:53:25,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-28 03:53:25,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-28 03:53:25,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-28 03:53:25,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-28 03:53:25,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-28 03:53:25,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-28 03:53:25,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-28 03:53:25,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-28 03:53:25,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-28 03:53:25,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-28 03:53:25,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-28 03:53:25,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-28 03:53:25,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-28 03:53:25,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-28 03:53:25,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-28 03:53:25,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-28 03:53:25,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-28 03:53:25,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-28 03:53:25,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-28 03:53:25,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-28 03:53:25,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-28 03:53:25,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-28 03:53:25,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-28 03:53:25,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-28 03:53:25,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-28 03:53:25,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-28 03:53:25,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-28 03:53:25,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-28 03:53:25,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-28 03:53:25,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-28 03:53:25,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-28 03:53:25,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-28 03:53:25,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-28 03:53:25,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-28 03:53:25,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-28 03:53:25,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-28 03:53:25,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-28 03:53:25,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-28 03:53:25,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-28 03:53:25,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-28 03:53:25,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-28 03:53:25,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-28 03:53:25,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-28 03:53:25,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-28 03:53:25,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-28 03:53:25,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-28 03:53:25,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-28 03:53:25,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-28 03:53:25,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-28 03:53:25,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-28 03:53:25,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-28 03:53:25,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-28 03:53:25,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-28 03:53:25,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-28 03:53:25,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-28 03:53:25,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-28 03:53:25,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-28 03:53:25,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-28 03:53:25,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-28 03:53:25,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-28 03:53:25,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-28 03:53:25,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-28 03:53:25,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-28 03:53:25,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-28 03:53:25,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-28 03:53:25,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-28 03:53:25,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-28 03:53:25,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-28 03:53:25,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-28 03:53:25,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-28 03:53:25,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-28 03:53:25,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-28 03:53:25,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-28 03:53:25,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-28 03:53:25,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-28 03:53:25,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-28 03:53:25,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-28 03:53:25,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-28 03:53:25,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-28 03:53:25,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-28 03:53:25,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-28 03:53:25,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-28 03:53:25,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-28 03:53:25,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-28 03:53:25,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-28 03:53:25,840 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022062275110248244); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0293819615291366, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022062275110248244); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04491666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-28 03:53:25,840 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-28 03:53:25,841 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01576078724322063, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022062275110248244); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0293819615291366, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022062275110248244); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04491666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-28 03:53:26,268 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 583 ms to try 5000 different iterations.  Found a solution that moves 922 regions; Going from a computed imbalance of 0.01576078724322063 to a new imbalance of 9.696114875787149E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007767349109159703); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007324451309205374); MoveCostFunction : (multiplier=7.0, imbalance=0.10244444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007767349109159703); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00575); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-28 03:53:26,270 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv610820674:81 , srv1487369977:82 , srv168392873:82 , srv1692124799:82 , srv259735501:82 , srv921839055:82 , srv1385403639:83 , srv155544575:83 , srv23509751:83 , srv605841741:83 , srv773558928:83 , srv1211297679:84 , srv1244496503:84 , srv1352464462:84 , srv1517923349:84 , srv1819836875:84 , srv1882133123:84 , srv1985960272:84 , srv2109630910:84 , srv2116827313:84 , srv519435713:84 , srv535759443:84 , srv612051048:84 , srv675008288:84 , srv676266805:84 , srv698413480:84 , srv713216110:84 , srv1040929536:85 , srv1288130461:85 , srv134929259:85 , srv1355395531:85 , srv166933970:85 , srv1777664123:85 , srv1786429765:85 , srv1945994684:85 , srv1946615099:85 , srv1948498760:85 , srv1965069570:85 , srv1977235489:85 , srv2004056184:85 , srv2007742081:85 , srv223370537:85 , srv374576610:85 , srv542327104:85 , srv643777511:85 , srv772888704:85 , srv841150949:85 , srv941051115:85 , srv988747463:85 , srv997737457:85 , srv1115153424:86 , srv1122083929:86 , srv1186775009:86 , srv1239277996:86 , srv1245997129:86 , srv128385485:86 , srv1340659409:86 , srv1414094364:86 , srv1438960149:86 , srv1486485506:86 , srv1570778221:86 , srv1602884632:86 , srv1773742287:86 , srv1803880389:86 , srv1875065460:86 , srv1979449324:86 , srv207194226:86 , srv450763988:86 , srv47545890:86 , srv520139509:86 , srv535194233:86 , srv613150980:86 , srv664947068:86 , srv760977287:86 , srv830552939:86 , srv872019921:86 , srv895767919:86 , srv982248893:86 , srv1050982692:87 , srv1073745487:87 , srv1368207252:87 , srv1416467161:87 , srv1566220754:87 , srv1636553654:87 , srv1868318759:87 , srv2010459690:87 , srv2075009683:87 , srv2098302050:87 , srv228226275:87 , srv363402242:87 , srv422148337:87 , srv554422365:87 , srv554433879:87 , srv712197518:87 , srv752439751:87 , srv944461121:87 , srv1067730853:88 , srv2056270559:88 , srv375318510:88 , srv962615980:557 }
2023-05-28 03:53:26,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-28 03:53:26,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1438960149=21, srv612051048=76, srv1965069570=44, srv676266805=81, srv1486485506=22, srv895767919=92, srv1979449324=46, srv542327104=71, srv872019921=91, srv1414094364=19, srv554422365=72, srv1819836875=37, srv1948498760=43, srv1067730853=2, srv613150980=77, srv519435713=67, srv207194226=52, srv554433879=73, srv1570778221=27, srv712197518=83, srv2116827313=56, srv1875065460=39, srv1517923349=24, srv1786429765=35, srv643777511=78, srv1073745487=3, srv1882133123=40, srv664947068=79, srv773558928=88, srv1692124799=32, srv1777664123=34, srv375318510=63, srv675008288=80, srv752439751=85, srv2098302050=54, srv2007742081=49, srv1122083929=5, srv1340659409=13, srv1977235489=45, srv1803880389=36, srv166933970=30, srv1245997129=10, srv1385403639=18, srv374576610=62, srv1368207252=17, srv422148337=64, srv520139509=68, srv1985960272=47, srv1946615099=42, srv1352464462=15, srv128385485=11, srv134929259=14, srv944461121=95, srv2004056184=48, srv535194233=69, srv1566220754=26, srv962615980=96, srv1244496503=9, srv1416467161=20, srv1636553654=29, srv363402242=61, srv1040929536=0, srv450763988=65, srv535759443=70, srv2056270559=51, srv1945994684=41, srv841150949=90, srv772888704=87, srv830552939=89, srv259735501=60, srv2109630910=55, srv155544575=25, srv610820674=75, srv1355395531=16, srv997737457=99, srv47545890=66, srv982248893=97, srv1211297679=7, srv23509751=59, srv1186775009=6, srv760977287=86, srv1050982692=1, srv168392873=31, srv713216110=84, srv921839055=93, srv1288130461=12, srv1487369977=23, srv605841741=74, srv228226275=58, srv2010459690=50, srv941051115=94, srv698413480=82, srv1239277996=8, srv1773742287=33, srv2075009683=53, srv1115153424=4, srv988747463=98, srv1602884632=28, srv223370537=57, srv1868318759=38} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-28 03:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-28 03:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-28 03:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-28 03:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-28 03:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-28 03:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-28 03:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-28 03:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-28 03:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-28 03:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-28 03:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-28 03:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-28 03:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-28 03:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-28 03:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-28 03:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-28 03:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-28 03:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-28 03:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-28 03:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-28 03:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-28 03:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-28 03:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-28 03:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-28 03:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-28 03:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-28 03:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-28 03:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-28 03:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-28 03:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-28 03:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-28 03:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-28 03:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-28 03:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-28 03:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-28 03:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-28 03:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-28 03:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-28 03:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-28 03:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-28 03:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-28 03:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-28 03:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-28 03:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-28 03:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-28 03:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-28 03:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-28 03:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-28 03:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-28 03:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-28 03:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-28 03:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-28 03:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-28 03:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-28 03:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-28 03:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-28 03:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-28 03:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-28 03:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-28 03:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-28 03:53:26,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-28 03:53:26,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-28 03:53:26,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-28 03:53:26,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-28 03:53:26,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-28 03:53:26,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-28 03:53:26,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-28 03:53:26,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-28 03:53:26,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-28 03:53:26,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-28 03:53:26,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-28 03:53:26,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-28 03:53:26,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-28 03:53:26,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-28 03:53:26,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-28 03:53:26,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-28 03:53:26,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-28 03:53:26,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-28 03:53:26,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-28 03:53:26,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-28 03:53:26,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-28 03:53:26,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-28 03:53:26,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-28 03:53:26,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-28 03:53:26,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-28 03:53:26,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-28 03:53:26,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-28 03:53:26,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-28 03:53:26,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-28 03:53:26,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-28 03:53:26,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-28 03:53:26,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-28 03:53:26,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-28 03:53:26,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-28 03:53:26,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-28 03:53:26,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-28 03:53:26,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-28 03:53:26,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-28 03:53:26,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-28 03:53:26,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-28 03:53:26,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-28 03:53:26,420 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007767349109159703); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007324451309205374); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007767349109159703); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00575); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-28 03:53:26,420 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-28 03:53:26,420 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.631555055922451E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007767349109159703); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007324451309205374); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007767349109159703); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00575); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-28 03:53:26,701 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 388 ms to try 5000 different iterations.  Found a solution that moves 275 regions; Going from a computed imbalance of 9.631555055922451E-4 to a new imbalance of 5.364096612525532E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00259499648053841); MoveCostFunction : (multiplier=7.0, imbalance=0.030555555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-28 03:53:26,702 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1040929536:84 , srv1073745487:84 , srv1115153424:84 , srv1122083929:84 , srv1186775009:84 , srv128385485:84 , srv1288130461:84 , srv1340659409:84 , srv1414094364:84 , srv1486485506:84 , srv1487369977:84 , srv1517923349:84 , srv1566220754:84 , srv1570778221:84 , srv1602884632:84 , srv1692124799:84 , srv1777664123:84 , srv1803880389:84 , srv1868318759:84 , srv1965069570:84 , srv2004056184:84 , srv2010459690:84 , srv2109630910:84 , srv23509751:84 , srv363402242:84 , srv450763988:84 , srv520139509:84 , srv554422365:84 , srv554433879:84 , srv613150980:84 , srv664947068:84 , srv752439751:84 , srv773558928:84 , srv830552939:84 , srv841150949:84 , srv895767919:84 , srv921839055:84 , srv988747463:84 , srv997737457:84 , srv1050982692:85 , srv1067730853:85 , srv1211297679:85 , srv1244496503:85 , srv1245997129:85 , srv134929259:85 , srv1352464462:85 , srv1355395531:85 , srv1368207252:85 , srv1385403639:85 , srv1416467161:85 , srv155544575:85 , srv1636553654:85 , srv166933970:85 , srv1773742287:85 , srv1786429765:85 , srv1819836875:85 , srv1882133123:85 , srv1945994684:85 , srv1946615099:85 , srv1948498760:85 , srv1977235489:85 , srv1979449324:85 , srv1985960272:85 , srv2007742081:85 , srv2056270559:85 , srv207194226:85 , srv2075009683:85 , srv2098302050:85 , srv223370537:85 , srv228226275:85 , srv259735501:85 , srv374576610:85 , srv375318510:85 , srv422148337:85 , srv47545890:85 , srv519435713:85 , srv535194233:85 , srv542327104:85 , srv605841741:85 , srv610820674:85 , srv612051048:85 , srv675008288:85 , srv676266805:85 , srv698413480:85 , srv712197518:85 , srv713216110:85 , srv760977287:85 , srv772888704:85 , srv941051115:85 , srv944461121:85 , srv1239277996:86 , srv1438960149:86 , srv168392873:86 , srv1875065460:86 , srv2116827313:86 , srv535759443:86 , srv643777511:86 , srv872019921:86 , srv982248893:86 , srv962615980:615 }
2023-05-28 03:53:26,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-28 03:53:26,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1438960149=21, srv612051048=76, srv1965069570=44, srv676266805=81, srv1486485506=22, srv895767919=92, srv1979449324=46, srv542327104=71, srv872019921=91, srv1414094364=19, srv554422365=72, srv1819836875=37, srv1948498760=43, srv1067730853=2, srv613150980=77, srv519435713=67, srv207194226=52, srv554433879=73, srv1570778221=27, srv712197518=83, srv2116827313=56, srv1875065460=39, srv1517923349=24, srv1786429765=35, srv643777511=78, srv1073745487=3, srv1882133123=40, srv664947068=79, srv773558928=88, srv1692124799=32, srv1777664123=34, srv375318510=63, srv675008288=80, srv752439751=85, srv2098302050=54, srv2007742081=49, srv1122083929=5, srv1340659409=13, srv1977235489=45, srv1803880389=36, srv166933970=30, srv1245997129=10, srv1385403639=18, srv374576610=62, srv1368207252=17, srv422148337=64, srv520139509=68, srv1985960272=47, srv1946615099=42, srv1352464462=15, srv128385485=11, srv134929259=14, srv944461121=95, srv2004056184=48, srv535194233=69, srv1566220754=26, srv962615980=96, srv1244496503=9, srv1416467161=20, srv1636553654=29, srv363402242=61, srv1040929536=0, srv450763988=65, srv535759443=70, srv2056270559=51, srv1945994684=41, srv841150949=90, srv772888704=87, srv830552939=89, srv259735501=60, srv2109630910=55, srv155544575=25, srv610820674=75, srv1355395531=16, srv997737457=99, srv47545890=66, srv982248893=97, srv1211297679=7, srv23509751=59, srv1186775009=6, srv760977287=86, srv1050982692=1, srv168392873=31, srv713216110=84, srv921839055=93, srv1288130461=12, srv1487369977=23, srv605841741=74, srv228226275=58, srv2010459690=50, srv941051115=94, srv698413480=82, srv1239277996=8, srv1773742287=33, srv2075009683=53, srv1115153424=4, srv988747463=98, srv1602884632=28, srv223370537=57, srv1868318759=38} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-28 03:53:26,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-28 03:53:26,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-28 03:53:26,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-28 03:53:26,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-28 03:53:26,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-28 03:53:26,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-28 03:53:26,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-28 03:53:26,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-28 03:53:26,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-28 03:53:26,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-28 03:53:26,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-28 03:53:26,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-28 03:53:26,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-28 03:53:26,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-28 03:53:26,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-28 03:53:26,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-28 03:53:26,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-28 03:53:26,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-28 03:53:26,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-28 03:53:26,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-28 03:53:26,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-28 03:53:26,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-28 03:53:26,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-28 03:53:26,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-28 03:53:26,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-28 03:53:26,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-28 03:53:26,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-28 03:53:26,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-28 03:53:26,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-28 03:53:26,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-28 03:53:26,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-28 03:53:26,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-28 03:53:26,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-28 03:53:26,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-28 03:53:26,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-28 03:53:26,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-28 03:53:26,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-28 03:53:26,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-28 03:53:26,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-28 03:53:26,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-28 03:53:26,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-28 03:53:26,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-28 03:53:26,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-28 03:53:26,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-28 03:53:26,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-28 03:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-28 03:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-28 03:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-28 03:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-28 03:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-28 03:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-28 03:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-28 03:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-28 03:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-28 03:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-28 03:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-28 03:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-28 03:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-28 03:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-28 03:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-28 03:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-28 03:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-28 03:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-28 03:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-28 03:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-28 03:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-28 03:53:26,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-28 03:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-28 03:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-28 03:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-28 03:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-28 03:53:26,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-28 03:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-28 03:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-28 03:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-28 03:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-28 03:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-28 03:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-28 03:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-28 03:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-28 03:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-28 03:53:26,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-28 03:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-28 03:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-28 03:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-28 03:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-28 03:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-28 03:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-28 03:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-28 03:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-28 03:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-28 03:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-28 03:53:26,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-28 03:53:26,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-28 03:53:26,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-28 03:53:26,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-28 03:53:26,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-28 03:53:26,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-28 03:53:26,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-28 03:53:26,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-28 03:53:26,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-28 03:53:26,852 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00259499648053841); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-28 03:53:26,853 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-28 03:53:26,853 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.1715374968770276E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00259499648053841); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-28 03:53:27,106 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 360 ms to try 5000 different iterations.  Found a solution that moves 65 regions; Going from a computed imbalance of 5.1715374968770276E-5 to a new imbalance of 6.487218467624833E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0013400504203456161); MoveCostFunction : (multiplier=7.0, imbalance=0.007222222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-28 03:53:27,106 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv168392873:83 , srv1819836875:83 , srv1875065460:83 , srv1945994684:83 , srv374576610:83 , srv535759443:83 , srv542327104:83 , srv643777511:83 , srv872019921:83 , srv944461121:83 , srv982248893:83 , srv1050982692:84 , srv1067730853:84 , srv1073745487:84 , srv1115153424:84 , srv1122083929:84 , srv1186775009:84 , srv1211297679:84 , srv1239277996:84 , srv1244496503:84 , srv1245997129:84 , srv128385485:84 , srv1340659409:84 , srv134929259:84 , srv1352464462:84 , srv1355395531:84 , srv1368207252:84 , srv1385403639:84 , srv1414094364:84 , srv1416467161:84 , srv1438960149:84 , srv1486485506:84 , srv1487369977:84 , srv155544575:84 , srv1566220754:84 , srv1570778221:84 , srv1602884632:84 , srv1636553654:84 , srv166933970:84 , srv1773742287:84 , srv1777664123:84 , srv1786429765:84 , srv1868318759:84 , srv1882133123:84 , srv1946615099:84 , srv1948498760:84 , srv1965069570:84 , srv1977235489:84 , srv1979449324:84 , srv1985960272:84 , srv2007742081:84 , srv2010459690:84 , srv2056270559:84 , srv2075009683:84 , srv2098302050:84 , srv2116827313:84 , srv223370537:84 , srv228226275:84 , srv23509751:84 , srv259735501:84 , srv363402242:84 , srv375318510:84 , srv422148337:84 , srv450763988:84 , srv47545890:84 , srv519435713:84 , srv520139509:84 , srv535194233:84 , srv554433879:84 , srv605841741:84 , srv610820674:84 , srv612051048:84 , srv664947068:84 , srv675008288:84 , srv676266805:84 , srv698413480:84 , srv712197518:84 , srv713216110:84 , srv752439751:84 , srv760977287:84 , srv772888704:84 , srv773558928:84 , srv830552939:84 , srv841150949:84 , srv895767919:84 , srv921839055:84 , srv941051115:84 , srv997737457:84 , srv1040929536:85 , srv1288130461:85 , srv1517923349:85 , srv1692124799:85 , srv1803880389:85 , srv2004056184:85 , srv207194226:85 , srv2109630910:85 , srv554422365:85 , srv613150980:85 , srv988747463:85 , srv962615980:684 }
2023-05-28 03:53:27,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-28 03:53:27,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1438960149=21, srv612051048=76, srv1965069570=44, srv676266805=81, srv1486485506=22, srv895767919=92, srv1979449324=46, srv542327104=71, srv872019921=91, srv1414094364=19, srv554422365=72, srv1819836875=37, srv1948498760=43, srv1067730853=2, srv613150980=77, srv519435713=67, srv207194226=52, srv554433879=73, srv1570778221=27, srv712197518=83, srv2116827313=56, srv1875065460=39, srv1517923349=24, srv1786429765=35, srv643777511=78, srv1073745487=3, srv1882133123=40, srv664947068=79, srv773558928=88, srv1692124799=32, srv1777664123=34, srv375318510=63, srv675008288=80, srv752439751=85, srv2098302050=54, srv2007742081=49, srv1122083929=5, srv1340659409=13, srv1977235489=45, srv1803880389=36, srv166933970=30, srv1245997129=10, srv1385403639=18, srv374576610=62, srv1368207252=17, srv422148337=64, srv520139509=68, srv1985960272=47, srv1946615099=42, srv1352464462=15, srv128385485=11, srv134929259=14, srv944461121=95, srv2004056184=48, srv535194233=69, srv1566220754=26, srv962615980=96, srv1244496503=9, srv1416467161=20, srv1636553654=29, srv363402242=61, srv1040929536=0, srv450763988=65, srv535759443=70, srv2056270559=51, srv1945994684=41, srv841150949=90, srv772888704=87, srv830552939=89, srv259735501=60, srv2109630910=55, srv155544575=25, srv610820674=75, srv1355395531=16, srv997737457=99, srv47545890=66, srv982248893=97, srv1211297679=7, srv23509751=59, srv1186775009=6, srv760977287=86, srv1050982692=1, srv168392873=31, srv713216110=84, srv921839055=93, srv1288130461=12, srv1487369977=23, srv605841741=74, srv228226275=58, srv2010459690=50, srv941051115=94, srv698413480=82, srv1239277996=8, srv1773742287=33, srv2075009683=53, srv1115153424=4, srv988747463=98, srv1602884632=28, srv223370537=57, srv1868318759=38} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-28 03:53:27,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-28 03:53:27,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-28 03:53:27,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-28 03:53:27,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-28 03:53:27,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-28 03:53:27,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-28 03:53:27,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-28 03:53:27,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-28 03:53:27,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-28 03:53:27,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-28 03:53:27,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-28 03:53:27,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-28 03:53:27,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-28 03:53:27,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-28 03:53:27,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-28 03:53:27,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-28 03:53:27,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-28 03:53:27,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-28 03:53:27,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-28 03:53:27,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-28 03:53:27,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-28 03:53:27,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-28 03:53:27,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-28 03:53:27,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-28 03:53:27,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-28 03:53:27,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-28 03:53:27,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-28 03:53:27,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-28 03:53:27,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-28 03:53:27,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-28 03:53:27,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-28 03:53:27,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-28 03:53:27,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-28 03:53:27,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-28 03:53:27,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-28 03:53:27,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-28 03:53:27,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-28 03:53:27,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-28 03:53:27,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-28 03:53:27,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-28 03:53:27,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-28 03:53:27,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-28 03:53:27,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-28 03:53:27,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-28 03:53:27,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-28 03:53:27,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-28 03:53:27,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-28 03:53:27,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-28 03:53:27,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-28 03:53:27,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-28 03:53:27,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-28 03:53:27,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-28 03:53:27,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-28 03:53:27,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-28 03:53:27,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-28 03:53:27,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-28 03:53:27,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-28 03:53:27,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-28 03:53:27,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-28 03:53:27,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-28 03:53:27,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-28 03:53:27,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-28 03:53:27,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-28 03:53:27,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-28 03:53:27,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-28 03:53:27,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-28 03:53:27,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-28 03:53:27,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-28 03:53:27,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-28 03:53:27,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-28 03:53:27,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-28 03:53:27,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-28 03:53:27,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-28 03:53:27,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-28 03:53:27,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-28 03:53:27,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-28 03:53:27,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-28 03:53:27,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-28 03:53:27,249 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.032078739728369E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0013400504203456161); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-28 03:53:27,249 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv168392873:83 , srv1819836875:83 , srv1875065460:83 , srv1945994684:83 , srv374576610:83 , srv535759443:83 , srv542327104:83 , srv643777511:83 , srv872019921:83 , srv944461121:83 , srv982248893:83 , srv1050982692:84 , srv1067730853:84 , srv1073745487:84 , srv1115153424:84 , srv1122083929:84 , srv1186775009:84 , srv1211297679:84 , srv1239277996:84 , srv1244496503:84 , srv1245997129:84 , srv128385485:84 , srv1340659409:84 , srv134929259:84 , srv1352464462:84 , srv1355395531:84 , srv1368207252:84 , srv1385403639:84 , srv1414094364:84 , srv1416467161:84 , srv1438960149:84 , srv1486485506:84 , srv1487369977:84 , srv155544575:84 , srv1566220754:84 , srv1570778221:84 , srv1602884632:84 , srv1636553654:84 , srv166933970:84 , srv1773742287:84 , srv1777664123:84 , srv1786429765:84 , srv1868318759:84 , srv1882133123:84 , srv1946615099:84 , srv1948498760:84 , srv1965069570:84 , srv1977235489:84 , srv1979449324:84 , srv1985960272:84 , srv2007742081:84 , srv2010459690:84 , srv2056270559:84 , srv2075009683:84 , srv2098302050:84 , srv2116827313:84 , srv223370537:84 , srv228226275:84 , srv23509751:84 , srv259735501:84 , srv363402242:84 , srv375318510:84 , srv422148337:84 , srv450763988:84 , srv47545890:84 , srv519435713:84 , srv520139509:84 , srv535194233:84 , srv554433879:84 , srv605841741:84 , srv610820674:84 , srv612051048:84 , srv664947068:84 , srv675008288:84 , srv676266805:84 , srv698413480:84 , srv712197518:84 , srv713216110:84 , srv752439751:84 , srv760977287:84 , srv772888704:84 , srv773558928:84 , srv830552939:84 , srv841150949:84 , srv895767919:84 , srv921839055:84 , srv941051115:84 , srv997737457:84 , srv1040929536:85 , srv1288130461:85 , srv1517923349:85 , srv1692124799:85 , srv1803880389:85 , srv2004056184:85 , srv207194226:85 , srv2109630910:85 , srv554422365:85 , srv613150980:85 , srv988747463:85 , srv962615980:684 }
2023-05-28 03:53:27,276 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=338 (was 281) - SystemLoadAverage LEAK? -, ProcessCount=174 (was 174), AvailableMemoryMB=4805 (was 5326)
2023-05-28 03:53:27,284 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=338, ProcessCount=174, AvailableMemoryMB=4804
2023-05-28 03:53:27,285 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-28 03:53:27,285 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-05-28 03:53:27,297 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv106330688:84 , srv1139230175:84 , srv1179578569:84 , srv1181845286:84 , srv1205969226:84 , srv1289218608:84 , srv1356199384:84 , srv1595483676:84 , srv1664285322:84 , srv1691718928:84 , srv170537461:84 , srv1905841253:84 , srv2132564619:84 , srv21719858:84 , srv24117185:84 , srv268416055:84 , srv282542916:84 , srv328221885:84 , srv463885721:84 , srv489150024:84 , srv559065629:84 , srv576007260:84 , srv65510117:84 , srv686036840:84 , srv767223232:84 , srv818639607:84 , srv834090359:84 , srv946041043:84 , srv947284034:84 , srv503568018:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-05-28 03:53:27,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-28 03:53:27,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1181845286=3, srv2132564619=12, srv106330688=0, srv834090359=27, srv1905841253=11, srv767223232=25, srv489150024=19, srv686036840=24, srv576007260=22, srv170537461=10, srv24117185=14, srv818639607=26, srv282542916=16, srv1664285322=8, srv1205969226=4, srv946041043=28, srv1179578569=2, srv1356199384=6, srv328221885=17, srv503568018=20, srv65510117=23, srv947284034=29, srv1289218608=5, srv1595483676=7, srv1691718928=9, srv268416055=15, srv559065629=21, srv463885721=18, srv1139230175=1, srv21719858=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-28 03:53:27,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-28 03:53:27,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-28 03:53:27,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-28 03:53:27,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-28 03:53:27,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-28 03:53:27,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-28 03:53:27,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-28 03:53:27,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-28 03:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-28 03:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-28 03:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-28 03:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-28 03:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-28 03:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-28 03:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-28 03:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-28 03:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-28 03:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-28 03:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-28 03:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-28 03:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-28 03:53:27,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-28 03:53:27,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-28 03:53:27,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-28 03:53:27,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-28 03:53:27,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-28 03:53:27,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-28 03:53:27,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-28 03:53:27,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-28 03:53:27,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-28 03:53:27,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-05-28 03:53:27,338 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0578459746602955, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-28 03:53:27,339 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912375314041592, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0578459746602955, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-05-28 03:55:27,318 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6117813 different iterations.  Found a solution that moves 1851 regions; Going from a computed imbalance of 0.9912375314041592 to a new imbalance of 4.320326340186438E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6855555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-28 03:55:27,325 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv106330688:90 , srv1139230175:90 , srv1179578569:90 , srv1181845286:90 , srv1205969226:90 , srv1289218608:90 , srv1356199384:90 , srv1595483676:90 , srv1664285322:90 , srv1691718928:90 , srv170537461:90 , srv1905841253:90 , srv2132564619:90 , srv21719858:90 , srv24117185:90 , srv268416055:90 , srv282542916:90 , srv328221885:90 , srv463885721:90 , srv489150024:90 , srv503568018:90 , srv559065629:90 , srv576007260:90 , srv65510117:90 , srv686036840:90 , srv767223232:90 , srv818639607:90 , srv834090359:90 , srv946041043:90 , srv947284034:90 }
2023-05-28 03:55:27,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-28 03:55:27,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1181845286=3, srv2132564619=12, srv106330688=0, srv834090359=27, srv1905841253=11, srv767223232=25, srv489150024=19, srv686036840=24, srv576007260=22, srv170537461=10, srv24117185=14, srv818639607=26, srv282542916=16, srv1664285322=8, srv1205969226=4, srv946041043=28, srv1179578569=2, srv1356199384=6, srv328221885=17, srv503568018=20, srv65510117=23, srv947284034=29, srv1289218608=5, srv1595483676=7, srv1691718928=9, srv268416055=15, srv559065629=21, srv463885721=18, srv1139230175=1, srv21719858=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-28 03:55:27,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-28 03:55:27,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-28 03:55:27,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-28 03:55:27,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-28 03:55:27,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-28 03:55:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-28 03:55:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-28 03:55:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-28 03:55:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-28 03:55:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-28 03:55:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-28 03:55:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-28 03:55:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-28 03:55:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-28 03:55:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-28 03:55:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-28 03:55:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-28 03:55:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-28 03:55:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-28 03:55:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-28 03:55:27,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-28 03:55:27,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-28 03:55:27,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-28 03:55:27,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-28 03:55:27,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-28 03:55:27,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-28 03:55:27,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-28 03:55:27,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-28 03:55:27,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-28 03:55:27,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-28 03:55:27,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-05-28 03:55:27,345 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-28 03:55:27,345 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv106330688:90 , srv1139230175:90 , srv1179578569:90 , srv1181845286:90 , srv1205969226:90 , srv1289218608:90 , srv1356199384:90 , srv1595483676:90 , srv1664285322:90 , srv1691718928:90 , srv170537461:90 , srv1905841253:90 , srv2132564619:90 , srv21719858:90 , srv24117185:90 , srv268416055:90 , srv282542916:90 , srv328221885:90 , srv463885721:90 , srv489150024:90 , srv503568018:90 , srv559065629:90 , srv576007260:90 , srv65510117:90 , srv686036840:90 , srv767223232:90 , srv818639607:90 , srv834090359:90 , srv946041043:90 , srv947284034:90 }
2023-05-28 03:55:27,358 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=239 (was 338), ProcessCount=170 (was 174), AvailableMemoryMB=4758 (was 4804)