2023-03-27 04:59:05,957 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-03-27 04:59:06,239 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-03-27 04:59:06,359 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-03-27 04:59:06,409 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-27 04:59:06,417 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-27 04:59:06,428 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=463, ProcessCount=167, AvailableMemoryMB=2109
2023-03-27 04:59:06,430 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-27 04:59:06,430 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-27 04:59:06,536 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1002478872:84 , srv1003569042:84 , srv10133516:84 , srv1032013612:84 , srv1075980083:84 , srv1140920450:84 , srv1159442272:84 , srv1169118102:84 , srv1176722229:84 , srv1181621426:84 , srv1194113634:84 , srv1221742709:84 , srv1248470640:84 , srv1267585682:84 , srv1296160846:84 , srv1296258946:84 , srv1318630393:84 , srv1385468958:84 , srv1405664361:84 , srv1408709053:84 , srv1458102155:84 , srv1481778432:84 , srv1503346841:84 , srv150573391:84 , srv1527057394:84 , srv1559273025:84 , srv1563298001:84 , srv1649981397:84 , srv1654833639:84 , srv1658545062:84 , srv167410550:84 , srv168246915:84 , srv1688222171:84 , srv1702267351:84 , srv1720789576:84 , srv1721983891:84 , srv1738344135:84 , srv1746069312:84 , srv1746902678:84 , srv1757839080:84 , srv1780358328:84 , srv1783778067:84 , srv185873421:84 , srv1877831573:84 , srv1885314523:84 , srv1895480001:84 , srv1898857672:84 , srv1917558593:84 , srv1943809683:84 , srv1946734493:84 , srv1960972481:84 , srv198805136:84 , srv1993170018:84 , srv1993300658:84 , srv1996719717:84 , srv1998564076:84 , srv2000655953:84 , srv200873759:84 , srv2011005051:84 , srv2034721706:84 , srv2040163629:84 , srv2041868949:84 , srv2057170163:84 , srv2107586255:84 , srv22544391:84 , srv232317809:84 , srv265830605:84 , srv301287425:84 , srv338101438:84 , srv346652243:84 , srv360444413:84 , srv380108378:84 , srv392162850:84 , srv394333304:84 , srv399740230:84 , srv40472659:84 , srv417260219:84 , srv417826535:84 , srv426584078:84 , srv460855713:84 , srv495239021:84 , srv518652063:84 , srv53573664:84 , srv573343800:84 , srv578523388:84 , srv583281775:84 , srv650920251:84 , srv72422336:84 , srv734117678:84 , srv81643212:84 , srv848909897:84 , srv850072363:84 , srv853612649:84 , srv917461160:84 , srv926180218:84 , srv934295263:84 , srv95958600:84 , srv961551101:84 , srv982065881:84 , srv557411468:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-03-27 04:59:06,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-27 04:59:06,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1169118102=7, srv185873421=42, srv360444413=70, srv1877831573=43, srv578523388=85, srv1503346841=22, srv1998564076=55, srv53573664=82, srv1702267351=33, srv1248470640=12, srv1720789576=34, srv380108378=71, srv557411468=83, srv1181621426=9, srv399740230=74, srv848909897=91, srv1993170018=52, srv1658545062=29, srv1408709053=19, srv10133516=2, srv917461160=94, srv417826535=77, srv200873759=57, srv1318630393=16, srv1746069312=37, srv394333304=73, srv934295263=96, srv1996719717=54, srv2011005051=58, srv1688222171=32, srv1885314523=44, srv1296258946=15, srv2057170163=62, srv1481778432=21, srv1746902678=38, srv1221742709=11, srv167410550=30, srv1946734493=49, srv426584078=78, srv301287425=67, srv1032013612=3, srv168246915=31, srv22544391=64, srv150573391=23, srv1140920450=5, srv850072363=92, srv1654833639=28, srv2041868949=61, srv346652243=69, srv1159442272=6, srv1075980083=4, srv1194113634=10, srv1783778067=41, srv1993300658=53, srv1563298001=26, srv982065881=99, srv392162850=72, srv198805136=51, srv1721983891=35, srv338101438=68, srv72422336=88, srv2107586255=63, srv853612649=93, srv1780358328=40, srv2040163629=60, srv1527057394=24, srv926180218=95, srv1405664361=18, srv1385468958=17, srv1267585682=13, srv460855713=79, srv95958600=97, srv232317809=65, srv81643212=90, srv1898857672=46, srv1960972481=50, srv1296160846=14, srv1559273025=25, srv1917558593=47, srv1458102155=20, srv1649981397=27, srv1738344135=36, srv265830605=66, srv734117678=89, srv40472659=75, srv961551101=98, srv1895480001=45, srv650920251=87, srv1002478872=0, srv573343800=84, srv417260219=76, srv583281775=86, srv2000655953=56, srv1003569042=1, srv1176722229=8, srv518652063=81, srv1943809683=48, srv1757839080=39, srv2034721706=59, srv495239021=80} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-27 04:59:06,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-27 04:59:06,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-27 04:59:06,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-27 04:59:06,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-27 04:59:06,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-27 04:59:06,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-27 04:59:06,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-27 04:59:06,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-27 04:59:06,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-27 04:59:06,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-27 04:59:06,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-27 04:59:06,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-27 04:59:06,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-27 04:59:06,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-27 04:59:06,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-27 04:59:06,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-27 04:59:06,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-27 04:59:06,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-27 04:59:06,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-27 04:59:06,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-27 04:59:06,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-27 04:59:06,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-27 04:59:06,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-27 04:59:06,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-27 04:59:06,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-27 04:59:06,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-27 04:59:06,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-27 04:59:06,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-27 04:59:06,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-27 04:59:06,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-27 04:59:06,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-27 04:59:06,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-27 04:59:06,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-27 04:59:06,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-27 04:59:06,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-27 04:59:06,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-27 04:59:06,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-27 04:59:06,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-27 04:59:06,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-27 04:59:06,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-27 04:59:06,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-27 04:59:06,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-27 04:59:06,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-27 04:59:06,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-27 04:59:06,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-27 04:59:06,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-27 04:59:06,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-27 04:59:06,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-27 04:59:06,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-27 04:59:06,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-27 04:59:06,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-27 04:59:06,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-27 04:59:06,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-27 04:59:06,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-27 04:59:06,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-27 04:59:06,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-27 04:59:06,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-27 04:59:06,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-27 04:59:06,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-27 04:59:06,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-27 04:59:06,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-27 04:59:06,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-27 04:59:06,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-27 04:59:06,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-27 04:59:06,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-27 04:59:06,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-27 04:59:06,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-27 04:59:06,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-27 04:59:06,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-27 04:59:06,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-27 04:59:06,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-27 04:59:06,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-27 04:59:06,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-27 04:59:06,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-27 04:59:06,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-27 04:59:06,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-27 04:59:06,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-27 04:59:06,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-27 04:59:06,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-27 04:59:06,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-27 04:59:06,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-27 04:59:06,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-27 04:59:06,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-27 04:59:06,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-27 04:59:06,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-27 04:59:06,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-27 04:59:06,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-27 04:59:06,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-27 04:59:06,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-27 04:59:06,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-27 04:59:06,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-27 04:59:06,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-27 04:59:06,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-27 04:59:06,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-27 04:59:06,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-27 04:59:06,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-27 04:59:06,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-27 04:59:06,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-27 04:59:06,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-27 04:59:06,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-27 04:59:06,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-27 04:59:06,848 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 04:59:06,848 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-27 04:59:06,848 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-27 04:59:07,355 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 704 ms to try 5000 different iterations.  Found a solution that moves 4684 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1778010823342679. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039149519453316586, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05837877018301094, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5204444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039149519453316586, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17266666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24291666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 04:59:07,375 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2040163629:81 , srv426584078:82 , srv1169118102:83 , srv1176722229:83 , srv1194113634:83 , srv1248470640:83 , srv1503346841:83 , srv1527057394:83 , srv1720789576:83 , srv853612649:83 , srv1140920450:84 , srv1296258946:84 , srv1559273025:84 , srv1654833639:84 , srv2034721706:84 , srv2041868949:84 , srv346652243:84 , srv72422336:84 , srv1003569042:85 , srv1221742709:85 , srv1318630393:85 , srv1458102155:85 , srv1649981397:85 , srv167410550:85 , srv1721983891:85 , srv1746069312:85 , srv1783778067:85 , srv1898857672:85 , srv198805136:85 , srv232317809:85 , srv360444413:85 , srv392162850:85 , srv495239021:85 , srv518652063:85 , srv578523388:85 , srv982065881:85 , srv1405664361:86 , srv1481778432:86 , srv1563298001:86 , srv1688222171:86 , srv1702267351:86 , srv1877831573:86 , srv1917558593:86 , srv1946734493:86 , srv265830605:86 , srv301287425:86 , srv338101438:86 , srv394333304:86 , srv399740230:86 , srv40472659:86 , srv417260219:86 , srv417826535:86 , srv460855713:86 , srv917461160:86 , srv934295263:86 , srv95958600:86 , srv1181621426:87 , srv1296160846:87 , srv1385468958:87 , srv1738344135:87 , srv1746902678:87 , srv1895480001:87 , srv1993170018:87 , srv1998564076:87 , srv2000655953:87 , srv2107586255:87 , srv850072363:87 , srv926180218:87 , srv1075980083:88 , srv1159442272:88 , srv1267585682:88 , srv1408709053:88 , srv1757839080:88 , srv1780358328:88 , srv1943809683:88 , srv1960972481:88 , srv1996719717:88 , srv200873759:88 , srv2011005051:88 , srv22544391:88 , srv650920251:88 , srv81643212:88 , srv10133516:89 , srv1032013612:89 , srv1658545062:89 , srv53573664:89 , srv573343800:89 , srv583281775:89 , srv150573391:90 , srv1993300658:90 , srv2057170163:90 , srv734117678:90 , srv1002478872:91 , srv168246915:91 , srv185873421:91 , srv380108378:91 , srv848909897:91 , srv1885314523:94 , srv961551101:94 , srv557411468:438 }
2023-03-27 04:59:07,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-27 04:59:07,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1169118102=7, srv185873421=42, srv360444413=70, srv1877831573=43, srv578523388=85, srv1503346841=22, srv1998564076=55, srv53573664=82, srv1702267351=33, srv1248470640=12, srv1720789576=34, srv380108378=71, srv557411468=83, srv1181621426=9, srv399740230=74, srv848909897=91, srv1993170018=52, srv1658545062=29, srv1408709053=19, srv10133516=2, srv917461160=94, srv417826535=77, srv200873759=57, srv1318630393=16, srv1746069312=37, srv394333304=73, srv934295263=96, srv1996719717=54, srv2011005051=58, srv1688222171=32, srv1885314523=44, srv1296258946=15, srv2057170163=62, srv1481778432=21, srv1746902678=38, srv1221742709=11, srv167410550=30, srv1946734493=49, srv426584078=78, srv301287425=67, srv1032013612=3, srv168246915=31, srv22544391=64, srv150573391=23, srv1140920450=5, srv850072363=92, srv1654833639=28, srv2041868949=61, srv346652243=69, srv1159442272=6, srv1075980083=4, srv1194113634=10, srv1783778067=41, srv1993300658=53, srv1563298001=26, srv982065881=99, srv392162850=72, srv198805136=51, srv1721983891=35, srv338101438=68, srv72422336=88, srv2107586255=63, srv853612649=93, srv1780358328=40, srv2040163629=60, srv1527057394=24, srv926180218=95, srv1405664361=18, srv1385468958=17, srv1267585682=13, srv460855713=79, srv95958600=97, srv232317809=65, srv81643212=90, srv1898857672=46, srv1960972481=50, srv1296160846=14, srv1559273025=25, srv1917558593=47, srv1458102155=20, srv1649981397=27, srv1738344135=36, srv265830605=66, srv734117678=89, srv40472659=75, srv961551101=98, srv1895480001=45, srv650920251=87, srv1002478872=0, srv573343800=84, srv417260219=76, srv583281775=86, srv2000655953=56, srv1003569042=1, srv1176722229=8, srv518652063=81, srv1943809683=48, srv1757839080=39, srv2034721706=59, srv495239021=80} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-27 04:59:07,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-27 04:59:07,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-27 04:59:07,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-27 04:59:07,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-27 04:59:07,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-27 04:59:07,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-27 04:59:07,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-27 04:59:07,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-27 04:59:07,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-27 04:59:07,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-27 04:59:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-27 04:59:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-27 04:59:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-27 04:59:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-27 04:59:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-27 04:59:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-27 04:59:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-27 04:59:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-27 04:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-27 04:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-27 04:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-27 04:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-27 04:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-27 04:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-27 04:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-27 04:59:07,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-27 04:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-27 04:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-27 04:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-27 04:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-27 04:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-27 04:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-27 04:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-27 04:59:07,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-27 04:59:07,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-27 04:59:07,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-27 04:59:07,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-27 04:59:07,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-27 04:59:07,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-27 04:59:07,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-27 04:59:07,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-27 04:59:07,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-27 04:59:07,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-27 04:59:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-27 04:59:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-27 04:59:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-27 04:59:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-27 04:59:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-27 04:59:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-27 04:59:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-27 04:59:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-27 04:59:07,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-27 04:59:07,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-27 04:59:07,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-27 04:59:07,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-27 04:59:07,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-27 04:59:07,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-27 04:59:07,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-27 04:59:07,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-27 04:59:07,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-27 04:59:07,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-27 04:59:07,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-27 04:59:07,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-27 04:59:07,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-27 04:59:07,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-27 04:59:07,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-27 04:59:07,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-27 04:59:07,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-27 04:59:07,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-27 04:59:07,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-27 04:59:07,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-27 04:59:07,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-27 04:59:07,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-27 04:59:07,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-27 04:59:07,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-27 04:59:07,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-27 04:59:07,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-27 04:59:07,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-27 04:59:07,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-27 04:59:07,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-27 04:59:07,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-27 04:59:07,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-27 04:59:07,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-27 04:59:07,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-27 04:59:07,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-27 04:59:07,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-27 04:59:07,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-27 04:59:07,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-27 04:59:07,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-27 04:59:07,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-27 04:59:07,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-27 04:59:07,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-27 04:59:07,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-27 04:59:07,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-27 04:59:07,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-27 04:59:07,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-27 04:59:07,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-27 04:59:07,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-27 04:59:07,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-27 04:59:07,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-27 04:59:07,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-27 04:59:07,612 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039149519453316586, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05837877018301094, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039149519453316586, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17266666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24291666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 04:59:07,613 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-27 04:59:07,613 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17776828426526073, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039149519453316586, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05837877018301094, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039149519453316586, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17266666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24291666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-27 04:59:07,844 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 395 ms to try 5000 different iterations.  Found a solution that moves 2696 regions; Going from a computed imbalance of 0.17776828426526073 to a new imbalance of 0.014644574158359848. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01947752517085767); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.028315997865677775, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2995555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.01947752517085767); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04666666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 04:59:07,854 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv961551101:77 , srv185873421:79 , srv380108378:79 , srv1885314523:80 , srv848909897:80 , srv1002478872:81 , srv168246915:81 , srv10133516:82 , srv1032013612:82 , srv1757839080:82 , srv1075980083:83 , srv1159442272:83 , srv1993300658:83 , srv2057170163:83 , srv22544391:83 , srv734117678:83 , srv1267585682:84 , srv1296160846:84 , srv150573391:84 , srv1658545062:84 , srv1688222171:84 , srv1738344135:84 , srv1746902678:84 , srv1877831573:84 , srv1895480001:84 , srv1960972481:84 , srv2000655953:84 , srv200873759:84 , srv417260219:84 , srv53573664:84 , srv573343800:84 , srv650920251:84 , srv1181621426:85 , srv1248470640:85 , srv1408709053:85 , srv1649981397:85 , srv1780358328:85 , srv1946734493:85 , srv1993170018:85 , srv1996719717:85 , srv1998564076:85 , srv2011005051:85 , srv583281775:85 , srv81643212:85 , srv1481778432:86 , srv1559273025:86 , srv1702267351:86 , srv1898857672:86 , srv1943809683:86 , srv2107586255:86 , srv265830605:86 , srv301287425:86 , srv338101438:86 , srv394333304:86 , srv40472659:86 , srv460855713:86 , srv578523388:86 , srv850072363:86 , srv926180218:86 , srv1003569042:87 , srv1140920450:87 , srv1917558593:87 , srv198805136:87 , srv399740230:87 , srv417826535:87 , srv518652063:87 , srv853612649:87 , srv917461160:87 , srv934295263:87 , srv1176722229:88 , srv1296258946:88 , srv1318630393:88 , srv1405664361:88 , srv1458102155:88 , srv1503346841:88 , srv1563298001:88 , srv1721983891:88 , srv1783778067:88 , srv232317809:88 , srv346652243:88 , srv360444413:88 , srv392162850:88 , srv495239021:88 , srv72422336:88 , srv982065881:88 , srv1169118102:89 , srv1221742709:89 , srv1385468958:89 , srv167410550:89 , srv2040163629:89 , srv2041868949:89 , srv95958600:89 , srv1654833639:90 , srv1720789576:90 , srv1746069312:90 , srv1194113634:91 , srv2034721706:91 , srv426584078:91 , srv1527057394:92 , srv557411468:509 }
2023-03-27 04:59:07,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-27 04:59:07,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1169118102=7, srv185873421=42, srv360444413=70, srv1877831573=43, srv578523388=85, srv1503346841=22, srv1998564076=55, srv53573664=82, srv1702267351=33, srv1248470640=12, srv1720789576=34, srv380108378=71, srv557411468=83, srv1181621426=9, srv399740230=74, srv848909897=91, srv1993170018=52, srv1658545062=29, srv1408709053=19, srv10133516=2, srv917461160=94, srv417826535=77, srv200873759=57, srv1318630393=16, srv1746069312=37, srv394333304=73, srv934295263=96, srv1996719717=54, srv2011005051=58, srv1688222171=32, srv1885314523=44, srv1296258946=15, srv2057170163=62, srv1481778432=21, srv1746902678=38, srv1221742709=11, srv167410550=30, srv1946734493=49, srv426584078=78, srv301287425=67, srv1032013612=3, srv168246915=31, srv22544391=64, srv150573391=23, srv1140920450=5, srv850072363=92, srv1654833639=28, srv2041868949=61, srv346652243=69, srv1159442272=6, srv1075980083=4, srv1194113634=10, srv1783778067=41, srv1993300658=53, srv1563298001=26, srv982065881=99, srv392162850=72, srv198805136=51, srv1721983891=35, srv338101438=68, srv72422336=88, srv2107586255=63, srv853612649=93, srv1780358328=40, srv2040163629=60, srv1527057394=24, srv926180218=95, srv1405664361=18, srv1385468958=17, srv1267585682=13, srv460855713=79, srv95958600=97, srv232317809=65, srv81643212=90, srv1898857672=46, srv1960972481=50, srv1296160846=14, srv1559273025=25, srv1917558593=47, srv1458102155=20, srv1649981397=27, srv1738344135=36, srv265830605=66, srv734117678=89, srv40472659=75, srv961551101=98, srv1895480001=45, srv650920251=87, srv1002478872=0, srv573343800=84, srv417260219=76, srv583281775=86, srv2000655953=56, srv1003569042=1, srv1176722229=8, srv518652063=81, srv1943809683=48, srv1757839080=39, srv2034721706=59, srv495239021=80} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-27 04:59:07,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-27 04:59:07,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-27 04:59:07,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-27 04:59:07,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-27 04:59:07,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-27 04:59:07,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-27 04:59:07,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-27 04:59:07,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-27 04:59:07,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-27 04:59:07,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-27 04:59:07,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-27 04:59:07,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-27 04:59:07,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-27 04:59:07,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-27 04:59:07,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-27 04:59:07,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-27 04:59:07,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-27 04:59:07,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-27 04:59:07,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-27 04:59:07,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-27 04:59:07,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-27 04:59:07,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-27 04:59:07,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-27 04:59:07,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-27 04:59:07,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-27 04:59:07,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-27 04:59:07,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-27 04:59:07,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-27 04:59:07,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-27 04:59:07,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-27 04:59:07,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-27 04:59:07,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-27 04:59:07,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-27 04:59:07,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-27 04:59:07,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-27 04:59:07,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-27 04:59:07,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-27 04:59:07,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-27 04:59:07,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-27 04:59:07,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-27 04:59:07,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-27 04:59:07,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-27 04:59:07,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-27 04:59:07,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-27 04:59:07,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-27 04:59:07,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-27 04:59:07,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-27 04:59:07,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-27 04:59:07,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-27 04:59:07,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-27 04:59:07,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-27 04:59:07,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-27 04:59:07,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-27 04:59:07,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-27 04:59:07,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-27 04:59:07,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-27 04:59:07,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-27 04:59:07,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-27 04:59:07,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-27 04:59:07,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-27 04:59:07,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-27 04:59:07,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-27 04:59:07,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-27 04:59:07,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-27 04:59:07,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-27 04:59:07,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-27 04:59:07,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-27 04:59:07,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-27 04:59:07,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-27 04:59:07,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-27 04:59:07,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-27 04:59:07,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-27 04:59:07,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-27 04:59:07,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-27 04:59:07,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-27 04:59:07,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-27 04:59:07,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-27 04:59:07,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-27 04:59:07,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-27 04:59:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-27 04:59:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-27 04:59:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-27 04:59:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-27 04:59:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-27 04:59:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-27 04:59:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-27 04:59:07,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-27 04:59:07,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-27 04:59:07,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-27 04:59:07,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-27 04:59:07,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-27 04:59:07,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-27 04:59:07,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-27 04:59:07,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-27 04:59:07,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-27 04:59:07,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-27 04:59:07,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-27 04:59:07,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-27 04:59:07,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-27 04:59:07,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-27 04:59:07,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-27 04:59:07,999 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01947752517085767); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.028315997865677775, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.01947752517085767); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04666666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 04:59:07,999 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-27 04:59:07,999 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014625696362876636, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01947752517085767); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.028315997865677775, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.01947752517085767); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04666666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-27 04:59:08,237 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 327 ms to try 5000 different iterations.  Found a solution that moves 1055 regions; Going from a computed imbalance of 0.014625696362876636 to a new imbalance of 4.322791157988363E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006120541647078519); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007842444017689564); MoveCostFunction : (multiplier=7.0, imbalance=0.11722222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006120541647078519); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0031666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 04:59:08,239 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1385468958:82 , srv1527057394:82 , srv95958600:82 , srv1194113634:83 , srv1318630393:83 , srv150573391:83 , srv1563298001:83 , srv1885314523:83 , srv1943809683:83 , srv2034721706:83 , srv583281775:83 , srv734117678:83 , srv1221742709:84 , srv1405664361:84 , srv1408709053:84 , srv1654833639:84 , srv167410550:84 , srv1721983891:84 , srv1746069312:84 , srv1783778067:84 , srv2041868949:84 , srv2057170163:84 , srv2107586255:84 , srv232317809:84 , srv399740230:84 , srv460855713:84 , srv81643212:84 , srv917461160:84 , srv926180218:84 , srv1169118102:85 , srv1181621426:85 , srv1458102155:85 , srv1658545062:85 , srv168246915:85 , srv1702267351:85 , srv1720789576:85 , srv1746902678:85 , srv1960972481:85 , srv198805136:85 , srv1993300658:85 , srv1996719717:85 , srv1998564076:85 , srv200873759:85 , srv2011005051:85 , srv265830605:85 , srv346652243:85 , srv360444413:85 , srv392162850:85 , srv394333304:85 , srv40472659:85 , srv417826535:85 , srv426584078:85 , srv495239021:85 , srv518652063:85 , srv53573664:85 , srv578523388:85 , srv650920251:85 , srv72422336:85 , srv850072363:85 , srv934295263:85 , srv982065881:85 , srv1003569042:86 , srv1032013612:86 , srv1075980083:86 , srv1176722229:86 , srv1267585682:86 , srv1296160846:86 , srv1296258946:86 , srv1481778432:86 , srv1503346841:86 , srv1877831573:86 , srv1895480001:86 , srv1898857672:86 , srv1917558593:86 , srv2000655953:86 , srv22544391:86 , srv301287425:86 , srv338101438:86 , srv573343800:86 , srv848909897:86 , srv961551101:86 , srv1002478872:87 , srv10133516:87 , srv1140920450:87 , srv1159442272:87 , srv1559273025:87 , srv1649981397:87 , srv1688222171:87 , srv1738344135:87 , srv1946734493:87 , srv1993170018:87 , srv2040163629:87 , srv380108378:87 , srv1757839080:88 , srv1780358328:88 , srv185873421:88 , srv417260219:88 , srv853612649:88 , srv1248470640:90 , srv557411468:565 }
2023-03-27 04:59:08,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-27 04:59:08,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1169118102=7, srv185873421=42, srv360444413=70, srv1877831573=43, srv578523388=85, srv1503346841=22, srv1998564076=55, srv53573664=82, srv1702267351=33, srv1248470640=12, srv1720789576=34, srv380108378=71, srv557411468=83, srv1181621426=9, srv399740230=74, srv848909897=91, srv1993170018=52, srv1658545062=29, srv1408709053=19, srv10133516=2, srv917461160=94, srv417826535=77, srv200873759=57, srv1318630393=16, srv1746069312=37, srv394333304=73, srv934295263=96, srv1996719717=54, srv2011005051=58, srv1688222171=32, srv1885314523=44, srv1296258946=15, srv2057170163=62, srv1481778432=21, srv1746902678=38, srv1221742709=11, srv167410550=30, srv1946734493=49, srv426584078=78, srv301287425=67, srv1032013612=3, srv168246915=31, srv22544391=64, srv150573391=23, srv1140920450=5, srv850072363=92, srv1654833639=28, srv2041868949=61, srv346652243=69, srv1159442272=6, srv1075980083=4, srv1194113634=10, srv1783778067=41, srv1993300658=53, srv1563298001=26, srv982065881=99, srv392162850=72, srv198805136=51, srv1721983891=35, srv338101438=68, srv72422336=88, srv2107586255=63, srv853612649=93, srv1780358328=40, srv2040163629=60, srv1527057394=24, srv926180218=95, srv1405664361=18, srv1385468958=17, srv1267585682=13, srv460855713=79, srv95958600=97, srv232317809=65, srv81643212=90, srv1898857672=46, srv1960972481=50, srv1296160846=14, srv1559273025=25, srv1917558593=47, srv1458102155=20, srv1649981397=27, srv1738344135=36, srv265830605=66, srv734117678=89, srv40472659=75, srv961551101=98, srv1895480001=45, srv650920251=87, srv1002478872=0, srv573343800=84, srv417260219=76, srv583281775=86, srv2000655953=56, srv1003569042=1, srv1176722229=8, srv518652063=81, srv1943809683=48, srv1757839080=39, srv2034721706=59, srv495239021=80} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-27 04:59:08,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-27 04:59:08,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-27 04:59:08,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-27 04:59:08,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-27 04:59:08,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-27 04:59:08,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-27 04:59:08,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-27 04:59:08,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-27 04:59:08,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-27 04:59:08,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-27 04:59:08,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-27 04:59:08,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-27 04:59:08,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-27 04:59:08,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-27 04:59:08,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-27 04:59:08,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-27 04:59:08,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-27 04:59:08,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-27 04:59:08,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-27 04:59:08,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-27 04:59:08,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-27 04:59:08,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-27 04:59:08,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-27 04:59:08,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-27 04:59:08,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-27 04:59:08,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-27 04:59:08,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-27 04:59:08,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-27 04:59:08,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-27 04:59:08,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-27 04:59:08,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-27 04:59:08,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-27 04:59:08,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-27 04:59:08,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-27 04:59:08,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-27 04:59:08,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-27 04:59:08,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-27 04:59:08,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-27 04:59:08,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-27 04:59:08,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-27 04:59:08,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-27 04:59:08,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-27 04:59:08,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-27 04:59:08,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-27 04:59:08,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-27 04:59:08,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-27 04:59:08,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-27 04:59:08,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-27 04:59:08,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-27 04:59:08,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-27 04:59:08,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-27 04:59:08,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-27 04:59:08,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-27 04:59:08,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-27 04:59:08,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-27 04:59:08,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-27 04:59:08,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-27 04:59:08,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-27 04:59:08,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-27 04:59:08,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-27 04:59:08,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-27 04:59:08,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-27 04:59:08,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-27 04:59:08,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-27 04:59:08,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-27 04:59:08,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-27 04:59:08,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-27 04:59:08,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-27 04:59:08,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-27 04:59:08,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-27 04:59:08,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-27 04:59:08,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-27 04:59:08,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-27 04:59:08,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-27 04:59:08,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-27 04:59:08,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-27 04:59:08,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-27 04:59:08,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-27 04:59:08,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-27 04:59:08,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-27 04:59:08,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-27 04:59:08,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-27 04:59:08,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-27 04:59:08,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-27 04:59:08,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-27 04:59:08,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-27 04:59:08,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-27 04:59:08,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-27 04:59:08,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-27 04:59:08,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-27 04:59:08,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-27 04:59:08,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-27 04:59:08,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-27 04:59:08,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-27 04:59:08,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-27 04:59:08,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-27 04:59:08,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-27 04:59:08,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-27 04:59:08,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-27 04:59:08,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-27 04:59:08,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-27 04:59:08,344 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006120541647078519); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007842444017689564); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006120541647078519); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0031666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 04:59:08,344 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-27 04:59:08,345 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.248918479075937E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006120541647078519); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007842444017689564); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006120541647078519); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0031666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-27 04:59:08,570 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 297 ms to try 5000 different iterations.  Found a solution that moves 245 regions; Going from a computed imbalance of 4.248918479075937E-4 to a new imbalance of 1.934896140798162E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.02722222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 04:59:08,571 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1780358328:81 , srv1002478872:83 , srv167410550:83 , srv1746069312:83 , srv1917558593:83 , srv1993170018:83 , srv392162850:83 , srv573343800:83 , srv1003569042:84 , srv10133516:84 , srv1075980083:84 , srv1159442272:84 , srv1221742709:84 , srv1248470640:84 , srv1267585682:84 , srv1296258946:84 , srv1385468958:84 , srv1405664361:84 , srv1458102155:84 , srv1481778432:84 , srv1503346841:84 , srv1654833639:84 , srv1658545062:84 , srv1702267351:84 , srv1720789576:84 , srv1757839080:84 , srv185873421:84 , srv1946734493:84 , srv1993300658:84 , srv1996719717:84 , srv2034721706:84 , srv301287425:84 , srv360444413:84 , srv417260219:84 , srv417826535:84 , srv426584078:84 , srv495239021:84 , srv53573664:84 , srv650920251:84 , srv853612649:84 , srv934295263:84 , srv982065881:84 , srv1032013612:85 , srv1140920450:85 , srv1169118102:85 , srv1181621426:85 , srv1194113634:85 , srv1296160846:85 , srv1408709053:85 , srv150573391:85 , srv1527057394:85 , srv1559273025:85 , srv1649981397:85 , srv168246915:85 , srv1688222171:85 , srv1738344135:85 , srv1746902678:85 , srv1783778067:85 , srv1877831573:85 , srv1885314523:85 , srv1895480001:85 , srv1898857672:85 , srv1943809683:85 , srv1960972481:85 , srv198805136:85 , srv1998564076:85 , srv2000655953:85 , srv200873759:85 , srv2011005051:85 , srv2040163629:85 , srv2041868949:85 , srv2057170163:85 , srv2107586255:85 , srv22544391:85 , srv232317809:85 , srv265830605:85 , srv338101438:85 , srv346652243:85 , srv380108378:85 , srv394333304:85 , srv399740230:85 , srv40472659:85 , srv518652063:85 , srv583281775:85 , srv72422336:85 , srv81643212:85 , srv848909897:85 , srv850072363:85 , srv917461160:85 , srv926180218:85 , srv95958600:85 , srv961551101:85 , srv1176722229:86 , srv1318630393:86 , srv1563298001:86 , srv1721983891:86 , srv460855713:86 , srv578523388:86 , srv734117678:86 , srv557411468:630 }
2023-03-27 04:59:08,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-27 04:59:08,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1169118102=7, srv185873421=42, srv360444413=70, srv1877831573=43, srv578523388=85, srv1503346841=22, srv1998564076=55, srv53573664=82, srv1702267351=33, srv1248470640=12, srv1720789576=34, srv380108378=71, srv557411468=83, srv1181621426=9, srv399740230=74, srv848909897=91, srv1993170018=52, srv1658545062=29, srv1408709053=19, srv10133516=2, srv917461160=94, srv417826535=77, srv200873759=57, srv1318630393=16, srv1746069312=37, srv394333304=73, srv934295263=96, srv1996719717=54, srv2011005051=58, srv1688222171=32, srv1885314523=44, srv1296258946=15, srv2057170163=62, srv1481778432=21, srv1746902678=38, srv1221742709=11, srv167410550=30, srv1946734493=49, srv426584078=78, srv301287425=67, srv1032013612=3, srv168246915=31, srv22544391=64, srv150573391=23, srv1140920450=5, srv850072363=92, srv1654833639=28, srv2041868949=61, srv346652243=69, srv1159442272=6, srv1075980083=4, srv1194113634=10, srv1783778067=41, srv1993300658=53, srv1563298001=26, srv982065881=99, srv392162850=72, srv198805136=51, srv1721983891=35, srv338101438=68, srv72422336=88, srv2107586255=63, srv853612649=93, srv1780358328=40, srv2040163629=60, srv1527057394=24, srv926180218=95, srv1405664361=18, srv1385468958=17, srv1267585682=13, srv460855713=79, srv95958600=97, srv232317809=65, srv81643212=90, srv1898857672=46, srv1960972481=50, srv1296160846=14, srv1559273025=25, srv1917558593=47, srv1458102155=20, srv1649981397=27, srv1738344135=36, srv265830605=66, srv734117678=89, srv40472659=75, srv961551101=98, srv1895480001=45, srv650920251=87, srv1002478872=0, srv573343800=84, srv417260219=76, srv583281775=86, srv2000655953=56, srv1003569042=1, srv1176722229=8, srv518652063=81, srv1943809683=48, srv1757839080=39, srv2034721706=59, srv495239021=80} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-27 04:59:08,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-27 04:59:08,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-27 04:59:08,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-27 04:59:08,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-27 04:59:08,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-27 04:59:08,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-27 04:59:08,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-27 04:59:08,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-27 04:59:08,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-27 04:59:08,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-27 04:59:08,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-27 04:59:08,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-27 04:59:08,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-27 04:59:08,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-27 04:59:08,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-27 04:59:08,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-27 04:59:08,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-27 04:59:08,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-27 04:59:08,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-27 04:59:08,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-27 04:59:08,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-27 04:59:08,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-27 04:59:08,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-27 04:59:08,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-27 04:59:08,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-27 04:59:08,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-27 04:59:08,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-27 04:59:08,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-27 04:59:08,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-27 04:59:08,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-27 04:59:08,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-27 04:59:08,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-27 04:59:08,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-27 04:59:08,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-27 04:59:08,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-27 04:59:08,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-27 04:59:08,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-27 04:59:08,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-27 04:59:08,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-27 04:59:08,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-27 04:59:08,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-27 04:59:08,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-27 04:59:08,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-27 04:59:08,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-27 04:59:08,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-27 04:59:08,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-27 04:59:08,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-27 04:59:08,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-27 04:59:08,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-27 04:59:08,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-27 04:59:08,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-27 04:59:08,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-27 04:59:08,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-27 04:59:08,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-27 04:59:08,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-27 04:59:08,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-27 04:59:08,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-27 04:59:08,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-27 04:59:08,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-27 04:59:08,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-27 04:59:08,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-27 04:59:08,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-27 04:59:08,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-27 04:59:08,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-27 04:59:08,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-27 04:59:08,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-27 04:59:08,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-27 04:59:08,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-27 04:59:08,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-27 04:59:08,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-27 04:59:08,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-27 04:59:08,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-27 04:59:08,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-27 04:59:08,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-27 04:59:08,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-27 04:59:08,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-27 04:59:08,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-27 04:59:08,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-27 04:59:08,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-27 04:59:08,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-27 04:59:08,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-27 04:59:08,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-27 04:59:08,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-27 04:59:08,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-27 04:59:08,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-27 04:59:08,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-27 04:59:08,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-27 04:59:08,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-27 04:59:08,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-27 04:59:08,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-27 04:59:08,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-27 04:59:08,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-27 04:59:08,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-27 04:59:08,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-27 04:59:08,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-27 04:59:08,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-27 04:59:08,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-27 04:59:08,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-27 04:59:08,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-27 04:59:08,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-27 04:59:08,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-27 04:59:08,686 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.7633434741294947E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 04:59:08,687 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1780358328:81 , srv1002478872:83 , srv167410550:83 , srv1746069312:83 , srv1917558593:83 , srv1993170018:83 , srv392162850:83 , srv573343800:83 , srv1003569042:84 , srv10133516:84 , srv1075980083:84 , srv1159442272:84 , srv1221742709:84 , srv1248470640:84 , srv1267585682:84 , srv1296258946:84 , srv1385468958:84 , srv1405664361:84 , srv1458102155:84 , srv1481778432:84 , srv1503346841:84 , srv1654833639:84 , srv1658545062:84 , srv1702267351:84 , srv1720789576:84 , srv1757839080:84 , srv185873421:84 , srv1946734493:84 , srv1993300658:84 , srv1996719717:84 , srv2034721706:84 , srv301287425:84 , srv360444413:84 , srv417260219:84 , srv417826535:84 , srv426584078:84 , srv495239021:84 , srv53573664:84 , srv650920251:84 , srv853612649:84 , srv934295263:84 , srv982065881:84 , srv1032013612:85 , srv1140920450:85 , srv1169118102:85 , srv1181621426:85 , srv1194113634:85 , srv1296160846:85 , srv1408709053:85 , srv150573391:85 , srv1527057394:85 , srv1559273025:85 , srv1649981397:85 , srv168246915:85 , srv1688222171:85 , srv1738344135:85 , srv1746902678:85 , srv1783778067:85 , srv1877831573:85 , srv1885314523:85 , srv1895480001:85 , srv1898857672:85 , srv1943809683:85 , srv1960972481:85 , srv198805136:85 , srv1998564076:85 , srv2000655953:85 , srv200873759:85 , srv2011005051:85 , srv2040163629:85 , srv2041868949:85 , srv2057170163:85 , srv2107586255:85 , srv22544391:85 , srv232317809:85 , srv265830605:85 , srv338101438:85 , srv346652243:85 , srv380108378:85 , srv394333304:85 , srv399740230:85 , srv40472659:85 , srv518652063:85 , srv583281775:85 , srv72422336:85 , srv81643212:85 , srv848909897:85 , srv850072363:85 , srv917461160:85 , srv926180218:85 , srv95958600:85 , srv961551101:85 , srv1176722229:86 , srv1318630393:86 , srv1563298001:86 , srv1721983891:86 , srv460855713:86 , srv578523388:86 , srv734117678:86 , srv557411468:630 }
2023-03-27 04:59:08,714 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=451 (was 463), ProcessCount=167 (was 167), AvailableMemoryMB=1613 (was 2109)
2023-03-27 04:59:08,720 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=451, ProcessCount=167, AvailableMemoryMB=1612
2023-03-27 04:59:08,721 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-27 04:59:08,721 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-03-27 04:59:08,735 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1097338057:84 , srv1157832106:84 , srv1316765318:84 , srv1336551164:84 , srv1391355168:84 , srv139932835:84 , srv1496312995:84 , srv1607970221:84 , srv1641975088:84 , srv1643977595:84 , srv1648110793:84 , srv1700077985:84 , srv1832656932:84 , srv1841566859:84 , srv1887792527:84 , srv1905167449:84 , srv1981209123:84 , srv2038854307:84 , srv217530511:84 , srv248849142:84 , srv270923889:84 , srv349450709:84 , srv350037620:84 , srv408646113:84 , srv594263926:84 , srv616893719:84 , srv681518771:84 , srv775799096:84 , srv958651420:84 , srv1079467104:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-03-27 04:59:08,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-27 04:59:08,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1641975088=9, srv1648110793=11, srv775799096=28, srv2038854307=18, srv217530511=19, srv1316765318=3, srv1336551164=4, srv248849142=20, srv1700077985=12, srv1643977595=10, srv594263926=25, srv408646113=24, srv1841566859=14, srv1391355168=5, srv1097338057=1, srv139932835=6, srv1496312995=7, srv1905167449=16, srv349450709=22, srv350037620=23, srv1157832106=2, srv1079467104=0, srv1887792527=15, srv616893719=26, srv681518771=27, srv1607970221=8, srv270923889=21, srv1832656932=13, srv958651420=29, srv1981209123=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-27 04:59:08,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-27 04:59:08,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-27 04:59:08,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-27 04:59:08,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-27 04:59:08,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-27 04:59:08,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-27 04:59:08,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-27 04:59:08,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-27 04:59:08,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-27 04:59:08,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-27 04:59:08,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-27 04:59:08,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-27 04:59:08,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-27 04:59:08,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-27 04:59:08,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-27 04:59:08,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-27 04:59:08,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-27 04:59:08,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-27 04:59:08,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-27 04:59:08,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-27 04:59:08,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-27 04:59:08,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-27 04:59:08,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-27 04:59:08,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-27 04:59:08,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-27 04:59:08,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-27 04:59:08,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-27 04:59:08,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-27 04:59:08,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-27 04:59:08,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-27 04:59:08,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-27 04:59:08,775 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1311941235147227, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 04:59:08,776 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912606431663593, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1311941235147227, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-03-27 05:01:08,756 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9391407 different iterations.  Found a solution that moves 1838 regions; Going from a computed imbalance of 0.9912606431663593 to a new imbalance of 4.2899836916600065E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6807407407407408, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 05:01:08,761 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1079467104:90 , srv1097338057:90 , srv1157832106:90 , srv1316765318:90 , srv1336551164:90 , srv1391355168:90 , srv139932835:90 , srv1496312995:90 , srv1607970221:90 , srv1641975088:90 , srv1643977595:90 , srv1648110793:90 , srv1700077985:90 , srv1832656932:90 , srv1841566859:90 , srv1887792527:90 , srv1905167449:90 , srv1981209123:90 , srv2038854307:90 , srv217530511:90 , srv248849142:90 , srv270923889:90 , srv349450709:90 , srv350037620:90 , srv408646113:90 , srv594263926:90 , srv616893719:90 , srv681518771:90 , srv775799096:90 , srv958651420:90 }
2023-03-27 05:01:08,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-27 05:01:08,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1641975088=9, srv1648110793=11, srv775799096=28, srv2038854307=18, srv217530511=19, srv1316765318=3, srv1336551164=4, srv248849142=20, srv1700077985=12, srv1643977595=10, srv594263926=25, srv408646113=24, srv1841566859=14, srv1391355168=5, srv1097338057=1, srv139932835=6, srv1496312995=7, srv1905167449=16, srv349450709=22, srv350037620=23, srv1157832106=2, srv1079467104=0, srv1887792527=15, srv616893719=26, srv681518771=27, srv1607970221=8, srv270923889=21, srv1832656932=13, srv958651420=29, srv1981209123=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-27 05:01:08,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-27 05:01:08,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-27 05:01:08,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-27 05:01:08,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-27 05:01:08,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-27 05:01:08,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-27 05:01:08,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-27 05:01:08,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-27 05:01:08,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-27 05:01:08,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-27 05:01:08,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-27 05:01:08,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-27 05:01:08,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-27 05:01:08,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-27 05:01:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-27 05:01:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-27 05:01:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-27 05:01:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-27 05:01:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-27 05:01:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-27 05:01:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-27 05:01:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-27 05:01:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-27 05:01:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-27 05:01:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-27 05:01:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-27 05:01:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-27 05:01:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-27 05:01:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-27 05:01:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-27 05:01:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-27 05:01:08,776 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 05:01:08,776 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1079467104:90 , srv1097338057:90 , srv1157832106:90 , srv1316765318:90 , srv1336551164:90 , srv1391355168:90 , srv139932835:90 , srv1496312995:90 , srv1607970221:90 , srv1641975088:90 , srv1643977595:90 , srv1648110793:90 , srv1700077985:90 , srv1832656932:90 , srv1841566859:90 , srv1887792527:90 , srv1905167449:90 , srv1981209123:90 , srv2038854307:90 , srv217530511:90 , srv248849142:90 , srv270923889:90 , srv349450709:90 , srv350037620:90 , srv408646113:90 , srv594263926:90 , srv616893719:90 , srv681518771:90 , srv775799096:90 , srv958651420:90 }
2023-03-27 05:01:08,787 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=252 (was 451), ProcessCount=162 (was 167), AvailableMemoryMB=2262 (was 1612) - AvailableMemoryMB LEAK? -